The Real Jeopardy Story
Blog 2 of 3 in the Due Diligence Often Discovers Discrepancies series.
As I said in my previous Watson blog earlier this week, Watson, a computer built by IBM, faced off against Ken Jennings and Brad Rutter, the two greatest Jeopardy players of all time and trounced them. In a two day match, Watson earned $77,174 to Ken Jennings' $24,000 and Brad Rutter's $21,600. While the results seem to show that the human race's crown as Jeopardy masters has been passed, a deeper analysis of the facts tell a different story. I have no doubt that one day a computer will be Jeopardy champion, but that day isn't today. What you saw wasn't a fair match among opponents but rather something that was closer to an infomercial demonstration where the product produces "too good to be true" results based on a tilted playing field.
Now back to the game...In Jeopardy, you're not allowed to push the buzzer right away. You have to wait until Alex finishes reading the question. At that point, a light goes off and then you can ring in to answer. If you try and anticipate the light and ring in too early, you are locked out for a quarter second, meaning that there is next to no chance to win the buzzer race. This is where Watson's unfair advantage comes in. If during the period Alex is reading the question, Watson comes up with an answer that it thinks is right (based on my observation, that would be an answer that it has scored as having an 80% or more probability), it can ring in just 10 milliseconds after the light goes off - enabling it beat the human contestants, with their mere mortal reflexes, to the buzzer every time. So, even when the human contestants know the answers, Watson gets all of the points. The Jeopardy results didn't accurately reflect Watson's question answering ability, they reflected the combination of its question-answering ability plus its superhuman reflexes.
So how would Watson have fared if it had to rely on just its question-answering ability? To answer that question, we analyzed the results of Game 2 of the two-game series (Ideally we would have analyzed both games, but since we only TIVO'ed Game 2 and the match isn't available online, it'll have to do). In Game 2, the three contestants scored as follows:
- Watson: $41,413
- Ken Jennings: $19,200
- Brad Rutter: $11,200
However, final scores aren't necessarily a good measure of how each player fared. They are highly dependent on how players bet in the Final Jeopardy, who gets Daily Doubles and how much they bet on Daily Doubles. Taking out Final Jeopardy and Daily Doubles, the players scored as follows:
- Watson: $25,200
- Ken Jennings: $14,600
- Brad Rutter: $5,600
In watching the game, it was pretty easy to tell when Ken Jennings wanted to ring in but was beaten to the buzzer by Watson. He held the buzzer chest high and you could see when he pressed the trigger and lost. Since Brad Rutter kept his buzzer below the podium, it wasn't possible to tell when he tried to ring in. But, the data from Ken Jennings is enough to figure out the impact of reflexes. Of Watson's $25,200, $19,200, all but $6,000 worth, was won on questions where Ken Jennings tried to ring in. Had Watson and Ken had equal reflexes, it stands to reason that Ken would have buzzed in first in half those cases. Adjusting for reflexes (including the possibility that Ken would have rung in first and gotten it wrong, hurting him instead of helping him) would add $9,088 to Ken's score and taken off $9,344 from Watson, giving revised scores for those two players of:
- Watson: $15,856
- Ken Jennings: $23,688
Since both Watson and Ken Jennings got the Final Jeopardy question right, instead of losing, Ken would have had a sizable victory over Watson. In conclusion, we’ll end this post with our own game of Jeopardy.
Category: Man vs. Machine
$1,000 Clue: As of February 16, 2011, although not the fastest to the buzzer, these biological beings were still the best at answering Jeopardy questions.
Question: What are Humans?
Look for our next blog and the final on this series where our partner and colleague Jon Klein of The Topline Strategy Group explores what he feels are the three top objections.
This article was contributed by Jon Klein. Jon is the founder and general partner of The Topline Strategy Group, a strategy consulting and market research firm specializing in emerging technologies. Jon brings a unique blend of strategy consulting and hands on operating experience to The Topline Strategy Group and works closely with Semaphore on a variety of engagements.