Remember the hack test?

Well, it was supposed to be a program that assessed the quality of your writing through a series of objective criteria.

But it had a fatal flaw? Reading is, was, will always be a subjective sport. It’s like ice dancing or rhythmic gymnastics. You can hit all the technical elements, but you still need that artistic flair.

So I abandoned the hack test, but someone else had a similar idea and created the textalyser.

The textalyser measures “readability,” not quality, per se. It does this by measuring word and sentence length, repetition and variation. For fun, I textalysed Tricia’s review of What is Left the Daughter and my post about books and booze. No surprise — Tricia’s posts are deemed more readable than mine. The textalyser said her posts were slightly easier than the optimal and mine were more difficult.

Speaking of web sites that assess text, I Write Like tells writers of which authors their text reminds them. I’m not sure what I Write Like’s criteria is (and I couldn’t find any explanation on the site;) but it compared my Harvey Pekar post with Arthur C. Clarke. Then, it said Big Boi’s lyrics to "Shutterbug" were similar to the work of Agatha Christie.

Finally, I listed a bunch of Indians players’ names — nothing else, just names — and the site told me I write like David Foster Wallace.

So I wouldn't take any comparisons too seriously.

While we’re playing games on the net, try the Interactive Proust Questionnaire from Vanity Fair. Apparently, my answers to the questionnaire were very similar to Brian Wilson’s and Hugh Hefner’s. (I want to put that on my business card. Jason Lea — half Beach Boy, half Playboy.)

Unrelated note: On the 50th anniversary of the release of To Kill a Mockingbird, The Rumpus asks if the book is overrated.

One word answer: No.

-Jason Lea, JLea@News-Herald.com

I’ve been gone for awhile.

I’d explain, but it’s a long story, and not even a long, interesting story. It’s a story that meanders without a point, similar to this prologue. So, instead of explaining why I disappeared, I’ll give you some of the elements involved in my disappearance and let you create a more interesting excuse: a pickup truck, my father-in-law, Wickliffe zoning code and two tons of gravel.

(If you guessed “building a deck,” you’re right… and boring. A better answer would have involved a shootout and a dying wish.)

I have a lot of ground to cover and want to be concise. I’d like to revisit one of Tricia’s comments and, also, the Hack Test one final time before giving it a respite. My co-blogger Tricia (who has carried the weight for me during my absence) and I have what we call the McDonalds Theorum.

Every reader, no matter how erudite, needs a way to unplug. I like the occasional comic book. Tricia likes romance novels. My favorite English professor swore by lousy pulp fiction. The most developed palette needs a cheeseburger every now and again. (That's why we call it the McDonalds Theorum.)

There’s nothing wrong with wanting a cheeseburger. However, Tricia took it a step farther when she wrote: I think we should ban usage of the word good when talking about authors and books.

Tricia and I disagree here. Some writing is good. Some writing is bad. You want proof? Read our newspaper — some of the writing is brilliant; some of it isn’t. I’m not knocking my coworkers. I’ve churned out more than my share of workmanlike copy. Deadlines make hacks of us all.

You can enjoy bad writing if you like, but that doesn’t mean there’s no such thing as good writing.

Robert Pirsig phrased it more eloquently in Zen and the Art of Motorcycle Maintenance:

Quality… you know what is, but you don’t know what it is. But that’s self-contradictory. But some things are better than others, that is, they have more quality. But when you try to say what quality is, apart from the things that have it, it all goes poof! There’s nothing to talk about. But if you can’t say what Quality is, how do you know what it is, or that it even exists? If no one knows what it is, then for all practical purposes it doesn’t exist at all. But for all practical purposes it really does exists … Obviously some things are better than others… but what’s the “betterness”?

To say there is no “good” or “bad” is to deny that anything is good. But some things are good. We may not be able to say what makes them good, but we know that good exists.

I have a secondary reason for mentioning Pirsig and Zen. In his book, Pirsig recounts how he tried to find a scientific way to define good rhetoric (like my proposed Hack Test). The end result? Pirsig had a psychological meltdown because there is no absolute scientific way to define good rhetoric.

Pirsig’s plight might seem obvious to others. Occasional commenter Neil noted that counting adverbs is not going to give you a guaranteed way to rank literature. I agree.

But I never intended the Hack Test to be an absolute test like Pirsig tried to devise. I paint in broader strokes. I simply wanted to devise a test that might indicate the quality of the narrative based upon measurable quantities in the text.

So I present the inaugural run of the Hack Test. Because I don’t have the years to devote to counting modifiers and “to be” verbs, I ran a mini-test on the first two pages of three books: Kyra by Carol Gilligan, A Good Man is Hard to Find by Flannery O’Connor and Italo Calvino’s collection of Italian Fairy Tales.

One final disclaimer: You can’t assess plot or character confidently from two pages because almost nothing has happened.

For example, Kyra begins with two characters playing chess. I suspect it’s a symbol for matching wits. That’s an overly familiar plot mechanism; but I wouldn’t call the plot of Kyra cliché. Similarly, the characters or setting do not stand out in two pages, but I the characters being the highpoint of Kyra the first time I read the book.

Consequently, it is unfair to assess plot, character or setting based upon two pages of reading — just language. All of these elements are vital to good reading, but I won’t pretend to assess them on the microtest.

Kyra
How many modifiers does the author use?
I counted 22 (but there could have been more.) Perhaps, more importantly, it felt like there were more. Gilligan did not describe a chair or room or piece of fashion without adding its color. I do not know the two characters’ last names, but I do know the color of their eyes. I’m not saying their last name is more important their eye color, but a lot of emphasis has been placed on pigment in these first two pages.

How many of those modifiers could be removed without changing the intent of the sentence?
I realize already this question is subjective. I thought it was objective. I was wrong. As a journalist, I’m trained to keep my writing tight. I would have excised most of the modifiers—11, in total; but someone else might feel differently. (Hence, any hack test would need to be personalized.)

How many cliches does the author use?
Two.
“standing around the fireplace”
And one character asks another what they are thankful for on Thanksgiving.

How many pathetic fallacies does the author use?
None. Gilligan wrote a line about the “sun igniting yellow leaves;” and while she wasn’t being literal, it’s not a pathetic fallacy.

How many times does the author use a tense of the verb “to be?”
13

How many times does the author use an extraneous phrase that could be removed completely? (Not just a modifier, but an entire phrase.)
I intended for this question to identify wasted verbiage: for example, when people say they are going OVER there. I counted three examples.

One other thing that should be noted about Gilligan’s text: she often used passive voice when it wasn’t necessary. In fact, she did it twice in her second sentence.

A Good Man is Hard to Find
How many modifiers does the author use?
18

How many of those modifiers could be removed without changing the intent of the sentence?
I would have only removed four, maximum.

How many cliches does the author use?
One
“seizing at every chance”

How many pathetic fallacies does the author use?
I counted none.

How many times does the author use a tense of the verb “to be?”
Eight

How many times does the author use an extraneous phrase that could be removed completely? (Not just a modifier, but an entire phrase.)
I counted one.

O’Connor devoted most of her first two pages to dialogue. Consequently, there were less opportunities narrative sloppiness.

Italian Folk Tales
How many modifiers does the author use?
12 (Most of these modifiers came from the character’s name, Dauntless Little John, being repeated.)

How many of those modifiers could be removed without changing the intent of the sentence?
Four.

How many cliches does the author use?
I expected a lot of clichés because fairy tales tend to depend on tropes (daring young men, people in distress, haunted houses et al) and, while the plot has some predictable elements, I only counted two narrative clichés. They involved “lighting the way” and “living happily.”

How many pathetic fallacies does the author use?
None.

How many times does the author use a tense of the verb “to be?”
Twelve.

How many times does the author use an extraneous phrase that could be removed completely? (Not just a modifier, but an entire phrase.)
I counted five. Most of them tacked “up” or “down” unnecessarily to the end of verbs.

Conclusion
The only conclusion I derived from my mini-test is that the actual number of modifiers used does not seem to matter. Gilligan and O’Connor both used several modifiers, but O’Connor’s writing felt much more concise.

The most important statistic was the number of modifiers that could have been removed; and identifying useless modifiers requires the reader to be subjective, which defeats the purpose of the Hack Test.

My hypothesis was that “language can be used to objectively rank the quality of the writing.” My microtest seems to have proven my hypothesis false.

Damn. I hate when that happens.

-Jason Lea, JLea@News-Herald.com

Before we use the Hack Test, I have one more revision. This lengthy suggestion comes from my old college housemate, Joe Paxton.

He’s working on his masters for something or another at Harvard. (No, most OU graduates do not continue to Harvard. He’s always been the type of guy who ruins the curve for all of us.) Consequently, he uses phrases like “multi-dimensional quality space” and “personalized learning algorithm.” He also makes an excellent point that simultaneously reaffirms and ruins my Hack Test. His words:

I love this idea. I think, though, the algorithm would have to be a learning algorithm rather than something static, because of how difficult it is to articulate exactly what makes for good/bad writing. And if might be a good idea to tailor it to the individual as well, since tastes vary.

I imagine that, with an individually-tailored learning algorithm, you could feed a book into a program that implements that algorithm, and rate it along some scale from good to bad. The program would then analyze the text of the book along multiple dimensions -- the dimensions you suggested would be a good start, but you could constantly add dimensions as you think of them.

After you’ve fed in a couple dozen books, the computer would have some idea of what, on average, you think makes for a good book. This would be encoded in a “multi-dimensional quality space” that combines the good/bad rating for a given book with the dimension scores for that book, and averages these multi-dimensional quality scores over all books.

So now you have your personalized multi-dimensional quality space, which tells you how important you personally take each dimension to be. I imagine that we could construct a database that holds the dimension scores for any arbitrary book. By comparing the dimension scores for that book to your personalized multiple-dimensional quality space, you could get a good idea of whether you’d be likely to enjoy the book.

I worry, though, that this method wouldn’t allow you to flag the really great books. The really great books often seem to be great because they break the mold, somehow going beyond most everything you’ve read before. But I do think this kind of personalized learning algorithm approach would enable you to at least filter out the hacks without having to read them. And it might even give you some decent recommendations.

Of course, this personalized learning algorithm approach is currently pretty impractical. But that’s not because the algorithm is particularly complicated. It’s mainly due to the fact that a lot of books have not been digitized yet. And that’s particularly true of fiction books. It seems that Google Books is in the process of changing that. So I could imagine such a system being practical within the next ten years.

And someone will write the program, given enough demand. In fact, I would be surprised if some computer scientists with a passion for literature hasn’t already started writing it.

Three thoughts on Joe’s statement: One, he’s absolutely right about this test not identifying great books. The best books do not follow logic, they defy it. However, I expect it will identify writers with hacky tendencies. (Thus, it is a Hack Test and not a Genius Exam.)

Two: I think Joe underrates how difficult it might be to write a computer algorithm for the hack test (and not just because my knowledge of HTML code ends with knowing how to underline statements). The sticking point would be the cliché clause. I feel the same way about clichés that Potter Stewart feels about pornography. I may not be able to list all the clichés in the English language, but I know them when I see them. Also, clichés have iterations that even the most inclusive algorithm might miss.

For example, if I write about “the straw that breaks the literary critic’s back,” it’s a cliché. It may not be verbatim, but it still counts.

Third, and most importantly: Joe is correct to say that these algorithms would need to be personalized. What is important to me will matter less to other readers. We can list factors (originality, conciseness of writing), but each reader must decide how to weigh these factors.

My Hack Test roots in William Zinsser’s notion of good writing. Don’t use three words when two will suffice. But Zinsser’s philosophy does not fit Cervantes, Dickens or Shakespeare. It would be pointless to apply my hack test to anything before 1940. But it offers one more tool we can use to measure literature.

The next time I post, I’ll be testing the test.

-Jason Lea, JLea@News-Herald.com

P.S. Because Joe mentioned it, an update on the Google case.

I disappeared for a week — holidays, family birthdays, Office Space-style printer destruction — but I’m back and still on the same subject: the hack test.

I turned my proposed algorithm over to that tool of sex offenders and bored homemakers — Facebook.

Most of my Facebook friends are smarter than me (as are most of the people I meet at the grocery store, county fair and city jail) and a few of them had some suggestions for my initial model.

First, Brian Jones noted a flaw in the system:

I don’t think this is possible. Even if the whole test is just a count of grammatical errors, sometimes those are an important part of the writing style (i.e. Emily Dickinson). I also don’t think originality is quite as simple as you’re making it out to be. If you ask me, nothing is ever truly original, so that would have to be measured on some numeric scale, as opposed to a simple “yes” or “no”.

Brian makes one great point. This system is not going to be infallible. It’s poorly suited to judge poetry and not designed for nonfiction. Also, the older an author gets, the less likely my system will effectively rate it. (Reading tastes and language itself change overtime. Chaucer, for example, would break my formula.)

As per originality, there are two schools of thought when it comes to originality: the Voltaire school (“Originality is nothing but judicious plagiarism”) and the Degas camp (“Art is either plagiarism or revolution.”) My original test applied the Degas standard. Something is either original or it isn’t. Brian is arguing for Voltaire. Nothing is original, just slightly less unoriginal.

Brian’s not wrong but his proposed remedy — a numeric scale — breaks the central tenet of the hack test. No subjectivity. Ideally, I would like to create a system that can be used to impartially rank literature. A numeric scale forces us to have an opinion.

However, Brian is correct. There is a middle ground between original and plagiarist; so what if, for now, we adapted the originality scale? (This would amend the almost identical questions pertaining to character, setting and plot.) What if we had a ranking system of three? It would be uncomplicated, but more nuanced than what predated it.

Settings, characters and plot could be (1) overly familiar, we recognize this character or setting etc. from multiple pieces or work, (2) familiar but not a copy, the character or plot may judiciously plagiarize one or two ideas without committing theft, and (3) if not completely original, very close to it. This character, plot or setting would need to be unlike something we had seen before.

I think the categories here are broad enough to permit as little interpretation as possible; but I won’t know until I apply the hack test to something. (I intend to do this soon. One more post revising the hack test. Then, a test run.)

-Jason Lea, JLea@News-Herald.com

Addendum: Tricia and I seem to have uploaded our posts almost simultaneously. I don't mean to ignore her opinion; but I prepared this post before I read it.

What if we created a hack test?

The difficulty with critiquing an author is that reading is subjective. Sometimes, you like an author. You know they’re middling or worse, but you still read them. Either they write about a subject that interests you or have a style that appeals despite its faults. Likewise, sometimes you unfairly hate a writer. Their technique is flawless, their characters and plot original, but the text doesn’t move you.

But what if we created a test that assessed the writing separate from emotion? What if we made an algorithm that spit out an “author score?”

Sure, it wouldn’t be infallible (and partly takes the fun away from reading,) but it could be fascinating.

The trick is it would have to measure quality quantitatively. We would need to glean measurable numbers from the text or our Hack Test would devolve into “I like them, so they’re not a hack.”

Roy Peter Clark and Harold Bloom already gave us two qualifiers to search for — overuse of unnecessary modifiers and dependence upon cliches.

We can read through passages and identify if a writer uses too many adverbs. That’s not subjective. That’s not a matter of opinion. If Author One uses twice as many adverbs as Author Two in the same amount of pages, Author Two has demonstrated more tact. If Author One depends on familiar turns of phrase, he is using a crutch.

We can count the number of pathetic fallacies and see if it exceeds good taste. We can check for dependence on “to be” verbs or overuse of passive voice. The language might be the easiest part of writing to score. Other aspects like characterization or plotting will be trickier.

We all know characterization is important, but how do we measure good characters?

The only quantifiable yardstick I can think of is originality. Tricia and I might argue about whether or not a character is likable or interesting, but we can agree when an author repeats character archetypes.

The same principle could be applied to plot or setting. Whether or not a plot is “good” is subjective. What I call good, you could call trash. Neither of us would be wrong, because they’re opinions. But originality? That’s not an opinion. Somebody has already used an idea before or it’s original. (Someone might be plagiarizing from an unknown source; but, in lieu of omniscience, we’ll have to depend on our own knowledge.)

So this is just a rough draft on the hack test. I expect revisions, but we must start somewhere.

First, some categories that we will rank:
Language
Plot
Characters
Setting

Questions pertaining to language:
How many modifiers does the author use?
How many of those modifiers could be removed without changing the intent of the sentence?
How many cliches does the author use?
How many pathetic fallacies does the author use?
How many times does the author use a tense of the verb “to be?”
How many times does the author use an extraneous phrase that could be removed completely? (Not just a modifier, but an entire phrase.)

Questions pertaining to plot:
Has the author used this or a similar plot in previous works? If yes, how many times?
Has this plot been used in other authors’ work previously?

Questions pertaining to character:
Has the author used similar characters in previous works? If yes, how many times?
Is this character almost identical to other characters previously created by another author?

Questions pertaining to setting:
Has the author used similar settings in previous works? If yes, how many times?
Has an almost identical setting been used in other authors’ work previously?

I’m not worried about a ranking system yet. Let’s get a usable test first.
So what other quantifiable qualities need to be included in the hack test?

-Jason Lea, JLea@News-Herald.com