Attention conservation notice: Preparing a text for automated poetry detection is half spell-checking. That part may be kind of dull; I’m writing it down anyway because this blog is how I remember things now.

Cleanup

To do automated text processing, you need a text. Some people, like the indefatigable Sparrow, type the whole text by hand. I am more defatigable than in-, so I needed another way. The Internet Archive has a text version of The Lord of the Rings, left over from the days of freedom before the Enclosure of the Internet. I
suspect that the lawyers have let it survive because the quality of the scan is so poor. I started with that.

I did most of my work on the Unix command line. My constant companion is a program called “aspell“. Feed it a text file, and it returns a list of all the words it didn’t recognize. A scanner makes predictable errors, such as reading a “u” as “ii” or an “h” as “li”. Those were the most common misspellings. They are easy enough to encode in a stream-editor input file. So I cleaned up all of those that I could. Some were harder, like distinguishing between “ore” and “orc”. The computer will never get that straight, so I did those by hand. All ambiguous cases got changed to “orc”, and then I went to the Moria chapters and turned back the two or three actual mentions of “ore”. (I have no idea how hard this task would be with a book I didn’t know so well, but I may soon find out.)

[ETA: The Guardian has a hilarious article saying that I didn’t get the really fun scanner error.]

The next thing to do was restore the hyphenated words at the ends of lines. That’s easy to do with a little Perl script — check lines that end in hyphens; delete the hyphen and newline, and see if the combination passes spell-check.

Now comes the fun part. LotR is full of proper names, archaic words, British
spellings, and invented languages. The spell-checker will never get those
right. So I had to work my way through the aspell output, picking out by hand the
briticisms and other things that were correct but unknown to the software. This took up my spare time for about a week. The result is a long list of words that are
correctly spelled, despite what the spell-checker thinks. So the process was:

1. Run aspell on the text;
2. Use the comm command to compare the output to the list of correct words. Remove all the words in the output that are correct in this context;
3. Figure out what the corrections might be on the remainder.  If they’re not wrong, add them to the “correct” list.
4. Repeat until the remainder is empty.

The list of correct words is an interesting document in its own right. You never know when that might come in handy, so I’ve published it on the Humanities Commons Core.

We now have a text of the book. Text is not all we need, though. We need sounds.

Phonetics

Next, we have to find the poetry. Alliteration is easy to identify, in theory. We’re  going to look for words that begin with the same sound. Or, rather, words that have the same sound at the beginning of the accented syllable. This is done with a “pronouncing dictionary”. I used the free one from Carnegie-Mellon University. (These were invented so voice-mail systems could read you a printed text over the phone, but they can be used for Good, too.) For each word in the English language, it has a line with the spelling of the word its phonetic expansion:

VOLUNTARY  V AA1 L AH0 N T EH2 R IY0
ASSISTANT  AH0 S IH1 S T AH0 N T
POSTMEN	 P OW1 S T M EH N

The list of phonemes are on the welcome page of the dictionary, but they’re easy to figure out.  Vowels have stresses; the numbers are the amount of stress each vowel gets. Zero means unstressed.  One means stressed. Subsidiary stresses have higher numbers.  Straight out of the box, the dictionary has over 130,000 words in it, but of course that’s not enough.  I spent the next couple of weeks making dictionary entries for all the LotR-words I collected in the last step. The supplementary dictionary is also on line at the Humanities Commons.  Surprisingly, the CMU dictionary contains all of the phonemes I needed except one. The supplement includes “KH” for words like “Erech” and “Grishnakh”.

This concludes the boring part. Next, I plunge into the quagmire of Anglo-Saxon scansion.

One doesn’t have to get very deep into studying Middle-earth to notice Anglo-Saxon alliterative verse. It’s right there on the surface, but an American teenager probably won’t get that it’s a structured verse-form. (At least, I didn’t. I thought it was another kind of vers libre.) I don’t know what fraction of Tolkien fans end up fascinated by this mostly-forgotten poetic form but I’m one of them. 

After one gets the hang of its rhythm, the really interesting part starts: alliterative lines start to show up in all kinds of places in Tolkien.  The first one I noticed was when the Riders of Rohan came to the Pelennor Fields:
‘In dark Mindolluin’s sides / they dimly echoed.’

Years later, I was listening to the Prancing Pony Podcast and heard (line breaks and caesurae added):

Then the warriors of Nargothrond went forth, and…
Tall and terrible / on that day looked Turin,
And the heart of the host / was upheld, as
He rode on the right hand / of Orodreth.[1]
Silmarillion, chapter 21, p. 239.

I started wondering how many of those buried alliterative lines there are in The Lord of the Rings. If there are a lot, they’re almost certainly carrying meaning. Could this be one of the techniques JRRT used to give a feel for where the story was in space and time? Maybe we’ll see lots of them in Rohan (even apart from the poems), and almost none in Lothlorien. If this works, it might be actual scholarship.

I raised one objection with myself: isn’t this just an Anglo-Saxon version of the “found poetry” that was all the rage in the 1960s? My English teachers inflicted that on us and I hated it. Without “intentionality” (as real scholars call it) it can’t be poetry, was my main objection.  I think this isn’t the case, for two reasons. First, nothing Tolkien wrote was unintentional. If alliterative lines are there, he left them there for us to find. Second, the poems they find always seemed to be free verse. If the poetry-finders had ever found a sonnet, or even a limerick, I’d have thought much better of them. Here, I’m looking for a strict verse form.

Granted, it’s possible to accidentally create a formal alliterative line. Tom Shippey pointed out that Steve Earle did it in a country song. Earle wrote it as a couplet:

When your Subaru is over and your Honda’s history,
I’ll be burnin’ down the back roads, just my baby and me…

“Sweet Little ‘66”, lines 22-23

but that second line scans perfectly. This isn’t the same thing as found poetry, because Earle was intentionally writing verse, and liked the alliteration. It’s the mirror image of what Cædmon wrote :

Þa middan-geard  / mann-cynnes weard

“Cædmon’s Hymn”, line 7

Which rhymes, but the rhyme is a poetic flourish, not part of the structure.

In keeping with proper Idiosophical practice, I decided to give the job to a computer. [2] Telling what happened is too long for one blog post, so I’m making this into a series.  Part 2 will be the delights of getting a text in shape. Part 3 will be a scream of frustration at the scholarship of Old English poetic meter. Part 4 will be algorithms, and Part 5 will be the results.

Teaser: the first thing the computer found was something I’d been staring at for a generation without seeing: “Bilbo Baggins / of Bag End” .

[1] This is how I learned the correct pronunciation of “Orodreth”.

[2] That was almost a year ago – I could have done it by hand by now.