Analyzing Rap Lyrics with Python

Gathering the data

Wikipedia blocks web scrapers, understandably. So I manually downloaded all of Wikipedia's lists of car manufacturers and brands by country. To reduce the amount of HTML I would have to parse, I selected the Mobile View for each page.

I downloaded ten files in total: a general list of car manufacturers plus individual lists for China, France, Germany, Italy, Japan, Spain, Sweden, the United Kingdom, and the United States.

Cleaning the data

Wikipedia's Mobile View pages contain search bars at the top and the usual "See also," "References," and "Read in another language" links. I started by erasing these (manually again), reducing the files to structural markup and the lists themselves.

With the Python Beautiful Soup library, I extracted the brand name tags by selecting all list elements. Then I used NLTK to remove the HTML tags and regular expressions to remove excess whitespace and text (such as notes about the brand, the brand's years of operation, etc.). This is what I came up with:

def clean_wikilist(filename):
    # open saved html file
    html = open(filename).read()

    # collect bulleted items only
    bullets = SoupStrainer("li")

    # make soup out of the bulleted items
    soup = BeautifulSoup(html, 'lxml', parse_only = bullets).prettify()

    # remove html from soup
    raw = nltk.clean_html(soup)

    # remove extra lines
    raw = re.sub(r'\n \n \n \n \n', r'\n', raw)
    raw = re.sub(r'\n \n \n', r'\n', raw)

    # create and clean tokens
    tokens = raw.split('\n')
    tokens = [re.sub(r'^\s+(?=[\S]+)', r'', token) for token in tokens]
    tokens = [token for token in tokens if not re.findall(r'\[[0-9]+\]|\([\S\s]+[\(\)]?|^\s+$|^[\s\[\]\(\)0-9]+$', token)]
    tokens = list(set(tokens))

    return tokens

After running my script on all ten lists, I had a whopping 2599 brand names. So I decided to limit the set to Germany, Japan, the UK, and the US. The pages for Germany, the UK, and the US separate current brands from defunct brands, so for those countries I used current brands only. The Japan page mixes current and defunct brands into one list; to save time, I used all of them.

These four pages have slightly different structures. The clean_wikilist() script worked nicely for Japan, but captured too much information on the others, so I wrote three additional scripts. Here's the one for Germany:

def autos_ge():
        # open saved html file
        html = open('autos-ge.html').read()

        # create soup object
        soup = BeautifulSoup(html)

        # select current major manufacturers
        majors = soup.select('span.mw-headline')
        majors = [w for w in majors if w.parent.parent.previous_sibling.contents[0]['id'] == 'Current_major_manufacturers']
        major_tokens = [nltk.clean_html(str(w)) for w in majors]
        major_tokens = [re.sub(r'\[\s\S\s\]', r'', token) for token in major_tokens]

        # select current minor manufacturers
        minors = soup.select('li')
        minors = [w for w in minors if w.parent.parent.previous_sibling.contents[0]['id'] == 'Current_minor_manufacturers']
        minor_tokens = [nltk.clean_html(str(w)) for w in minors]
        minor_tokens = [re.sub(r'\s\(\S+\)', r'', token) for token in minor_tokens]

        # combine lists
        tokens = list(set(minor_tokens + major_tokens))

        return tokens

One notable difference between this and the original script is the usage of Beautiful Soup. In clean_wikilist(), the desired elements are selected first, with SoupStrainer, and then used to create a list of the matching HTML tags in unicode format. autos_ge(), on the other hand, creates a Beautiful Soup object out of the entire page; the desired elements are selected via DOM traversal.

The number of brand names from this limited dataset? Just 178.

Analyzing the data

The Rap Lyrics Database contains lyrics for all of Billboard Music's rap songs from 1989 through 2009. It's the only searchable database of hip-hop lyrics (exclusively).

The result pages share the same URL, with the search term appended to the end: http://research.blackyouthproject.com/raplyrics/results/?all/1989-2009/. This made it much easier to automate the search and saving process.

def rap_search(auto_list):
    # search for each brand name
    for brand in auto_list:
        url = 'http://research.blackyouthproject.com/raplyrics/results/?all/1989-2009/' + word

        # save the search results page
        results_html = urllib2.urlopen(url).read()

        # save it as a file named after the brand
        results = word + '.html'

        with open(results, 'w') as results_file:
            results_file.write(results_html)

This saved 178 HTML pages, each named after the appropriate brand search term, onto my computer. I also searched for known nicknames of the brands (e.g., "Bimmer/Beemer/Beamer" for BMW and "Chevy" for Chevrolet).

I used Beautiful Soup again to count the number of results on each page:

def count_rap_results():
        # for all html files in current directory
        for filename in os.listdir('.'):
        if filename.endswith('html'):

        # select song titles
        html = open(filename).read()
        soup = BeautifulSoup(html)
        songs = soup.select('.title')

        # count number of song titles
        count = len(songs)

        # write brand names and number of songs into a text file
        with open('count_rap_autos.txt', 'a') as counter_file:
                counter_file.write('%s%15d\n' % (filename[:-5], count))

This got me a pretty messy-looking text file of each brand and the number of songs in which it was mentioned:

Lea-Francis               1

Ewing      0

Efini         0

Scion       0

Tommy Kaira            15

...

It turned out that the Rap Lyrics Database doesn't recognize spaces. So, a search for "Art and Tech" became a search for "Art"--which of course is a popular word that is often used in a non-automobile context. I removed ambiguous names from the list and combined the results from brands and their nicknames. LibreOffice Calc was helpful in fixing the columns and sorting the results.

The final number of usable brands came out to 153.

More complete data

I slashed the set of brand names to less than 7% of its original size and conflated "car manufacturers" with "car brands." Wikipedia also has a lengthy list of automobile marques, which I didn't even touch.

I'd like to go deeper than brands, into the actual names of cars, and match them against an even bigger database of lyrics. RapGenius and the Last.fm API are possible alternatives to the Rap Lyrics Database. RapGenius has an excellent database, but it contains a significant amount of lyrics from non-hip-hop artists as well.

Semantic orientation

I equated "beloved" to "number of songs mentions." This is obviously not always the case, as rappers name-drop plenty of things they dislike. It's true that rappers generally mention cars in a positive manner, but a more accurate experiment would take into account not just how many times the brand was used, but in what way the brand was used--i.e., the semantic orientation of the brand. A sentiment analysis might be the way to go.

Multi-word brands

The Rap Lyrics Database turns up blank if you search for, say "Aston Martin" (with the quotes and the space), even though Aston Martin is mentioned in a few songs. So multi-word brands with spaces in them turned up short. (Mercedes-Benz doesn't have this issue because it has a hyphen, not a space.)

If I were to use the Rap Lyrics Database again, I'd have to search for "Aston" and "Martin" separately and compare the songs on each results page. Otherwise, RapGenius seems to do spaces nicely.

Nicknames and duplicates

I searched for "Mercedes-Benz" as well as just "Mercedes" and "Benz." However, again, because I didn't compare song names, I ended up nixing the counts for the nicknames to minimize the possibility of duplicates. I also missed some nicknames--for instance, I completely neglected "Caddy" and "Lac" (sorry, Cadillac), "Lex," etc.

Misspellings, plurals, etc.

I did search for "Beamer" and "Beemer," but there's also "Bima" and probably countless other misspellings of "Bimmer" and other car brands. I ignored plurals, whether spelled correctly ("Bimmers") or not ("Bimaz").

Disambiguation

Many car brands double as common words or unrelated proper names, e.g., Prince, Radical, Ram, MINI, Oakland. I discarded these instead of determining whether or not they were referring to the car brand.

Last thoughts

A few months ago, when I first got the idea for this project, I thought it would be ridiculously hard. I envisioned building a large corpus of hip-hop lyrics and determining the classification, meaning, and orientation of each word to uncover the truth.

After going about this somewhat backwards, I think my initial impression remains correct. I'm happy I finally took a stab at the project, and I'm excited to continue working on it. This is just the first of a long series of experiments (and blog posts)!

Source files on GitHub