Hot-Buzzer Topic: Did IBM's Watson Have an Advantage on 'Jeopardy!'?

You could say Jeopardy! contestant Brad Rutter's buttons have been pushed, if not buzzed.

As the dust settles following the closely watched, man-against-machine Jeopardy! contest, a debate has begun over whether the IBM supercomputer Watson had an advantage over Rutter and the other human competitor, Ken Jennings, while buzzing-in answers to Jeopardy! clues.

According to an article on the Kurzweil web site, IBM (IBM) researchers say it was the humans who had the advantage, since they could anticipate the buzzer light going off and could respond even faster than Watson. But during an interview with DailyFinance, Rutter says the IBM folks are acting more like "politicians than scientists" by making such comments. "Technically their comments are true," Rutter says, "but highly misleading."

Unhuman Reflexes

Rutter racked up a record $3.26 million in game winnings over his Jeopardy! career, and is well-known for his speed in buzzing-in answers. But he says human contestants can rarely hit the button as soon as the light comes on, whereas Watson is consistently fast.

"It's hard to be in that zone," Rutter says, noting Jeopardy! players who jump the gun in anticipation of the light often end up being shut out for a brief moment. "Watson doesn't have to worry about being shut because he doesn't anticipate."

Jennings, who won the most games in the show's history, expressed similar sentiments in a New York Daily News Op-Ed piece: "The reflexes of even a very good human player will vary slightly," he wrote, "but not Watson's. If it knows the answer, it makes the perfect buzz. Every single time. And it's hard to win if you can't buzz."

Going into the event, Rutter says he wasn't sure what to fully expect. While watching "sparring" videos Watson had with other Jeopardy! contestants, he could see the supercomputer was winning roughly 70% of the time. While he sensed the biggest advantage for Watson would be its buzzing ability, Rutter also thought he would prove a strong rival.

On Day 1 of the contest, Rutter managed to keep pace with the supercomputer. Watson and Rutter ended the day tied with $5,000 each in winnings.

"We had one more practice round before the game and I felt good," he recalled. "I hit my stride. I got the first clue and got it right but then Watson got hot and locked us out through the first commercial break. It was amazing to see how fast he is when he's confident. But after the commercial break, he got a few wrong and had some hiccups."

No Ebb and Flow in the Game

Although Rutter gained a substantial lead, Watson recovered toward the end of the round to tie. Rutter initially thought Day 1 of the event was similar to a typical Jeopardy! program -- where there is an ebb and flow as the game progresses, with contestants pulling ahead and falling behind.

Day 2, however, changed all that. Rutter says a number of the categories randomly selected that day tended to favor Watson's sweet spot: concrete, fact-based clues rather than abstract concepts and short clues. Watson dominated game play on the second day -- landing a number of Daily Doubles that allowed him to dramatically boost its score.

"Once Watson got in the lead, the Daily Doubles became even more important to find," Rutter says, noting his usual strategy is to rack up some winnings before seeking out the Daily Doubles, in order to try for a bigger bang with his bucks.

"If you have zero dollars and get a Daily Double, that's still zero dollars," he says. Rutter observed that Watson was on the prowl for Daily Doubles even though it had only several hundred dollars. That strategy appeared to ensure the computer's lead while keeping the other two contestants from later boosting their scores.

Luck is Still a Factor

Rutter says he knew Day 3 would be an uphill battle -- and notes that Jennings was fortunate to hit a Daily Double that put him back in the game. But in the end, Rutter placed third, Jennings second and Watson first.

"I felt I put in a pretty good showing," Rutter says. "With Jeopardy!, there is usually a lot of luck involved. A couple little breaks here and there can make a big difference."

And while CBS (CBS) got high television ratings for the event, Rutter doesn't expect computers to replace humans on game shows.

"I think the ratings bump came from the novelty factor. I think over time people would get bored watching computers," Rutter says, adding wryly that "Watson never jumped up or down if he got it right."

View all Articles » Dawn Kawamoto is an award-winning business and technology reporter for DailyFinance. She previously covered the business angle of technology companies at CBS Interactive's CNET News and has also worked as a business reporter for the Los Angeles Daily News, Orange County Register and Santa Barbara News-Press.

Read More

View the original article here

Watson Wasn't Perfect: IBM Explains the 'Jeopardy!' Errors

Even big brains can have a blip. And IBM's supercomputer Watson is no exception, despite its bank of 90 IBM Power 750 servers that can process the equivalent of 1 million books of information a second.

Those blips, obviously, were few and far between -- and they didn't slow down Watson's highly publicized victory this week against two human champions on the TV game show Jeopardy!. The flesh-and-blood competitors were no slouches, either. Brad Rutter had earned $3.26 million playing Jeopardy!, making him the largest dollar winner in the show's history. And Ken Jennings had the program's longest-running winning streak.

You Can't Blame "Human Error"

On Monday, Day 1 of the three-day contest, Watson tied Rutter at $5,000 each in winnings. Along the way, however, the supercomputer hit a few blips when it came to processing clues given by Jeopardy! host Alex Trebek. Knowing that writing off the mistakes to "human error" wouldn't cut it, Big Blue's (IBM) Watson team have explained the supercomputer's miscalculations in a variety of posts on the Internet.

One of those misfires occurred during the "Name that Decade" category. The clue: "The first modern crossword puzzle is published and Oreo cookies are introduced."

Jennings was first to buzz, but gave he wrong answer of "What is the 1920s?". Watson was next to answer, but apparently wasn't listening. The mega-machine repeated Jennings' mistake, prompting a gentle scolding from Trebek. But Watson isn't listening to Trebek, either: The supercomputer has no ears, nor the ability for speech recognition.

Chris Welty works on Watson's algorithms team. According to an Ars Technica post, Welty and his crew thought it wasn't necessary for Watson to crunch through the other contestants' wrong answers.

An Arm and a Leg

Similarly, Watson may have benefited from Jennings' wrong answer in responding to the clue: "It was the anatomical oddity of U.S. Gymnast George Eyser, who won a gold medal on the parallel bars in 1904." Jennings answered Eyser was missing an arm -- and Watson then offered up, "What is a leg?"

Although Watson got the body part right, he was dinged for failing to note it the leg in question was "missing."

In a blog post, David Ferrucci, who heads up the Watson project, noted Watson likely didn't understand the word "oddity." According to Ferrucci: "The computer wouldn't know that a missing leg is odder than anything else."

But Watson's creators note that, over time and by playing more games and by gaining greater exposure to more material, the supercomputer could possibly gain a greater understanding of those concepts, since it's loaded with machine-learning technology. Watson's rivals, in mapping out their strategy to go head-to-head with it, were well aware he seemed to struggle with abstract concepts and short clues.

On Day 2, Watson missed one clue by a country mile -- better make that an entire country. During a Final Jeopardy! segment that included the "U.S. Cities" category, the clue was: "Its largest airport was named for a World War II hero; its second-largest, for a World War II battle."

Watson responded "What is Toronto???," while contestants Jennings and Rutter correctly answered Chicago -- for the city's O'Hare and Midway airports.

In a blog post, Ferrucci pointed to several issues that may have tripped-up Watson:

First, the category names on Jeopardy! are tricky. The answers often do not exactly fit the category. Watson, in his training phase, learned that categories only weakly suggest the kind of answer that is expected, and, therefore, the machine downgrades their significance. The way the language was parsed provided an advantage for the humans and a disadvantage for Watson, as well. "What US city" wasn't in the question. If it had been, Watson would have given US cities much more weight as it searched for the answer. Adding to the confusion for Watson, there are cities named Toronto in the United States and the Toronto in Canada has an American League baseball team. It probably picked up those facts from the written material it has digested. Also, the machine didn't find much evidence to connect either city's airport to World War II. (Chicago was a very close second on Watson's list of possible answers.) So this is just one of those situations that's a snap for a reasonably knowledgeable human but a true brain teaser for the machine.

Learning the Game

Like any artful player, however, Watson developed a sense of when to hold, to fold or to play.

Watson knew the Toronto answer could be big-time bust, so it wagered a mere $947.

According to a blog post by Gerald Tesauro, an IBM researcher, Watson's wagering style largely hinges on two questions: "How likely am I to answer the Daily Double clue correctly?" and "How much will a given bet increase or decrease my winning changes when I get the Daily Double right or wrong?"

As any Jeopardy! fan knows, the Daily Double can make or break a winning streak. A couple of these are hidden on the game board, and players who land one can bet from $5 to their entire holdings on that single clue. A Daily Double not only has the potential for a contestant to double his money but it's also not subject to a rival jumping in with his own answer.

Wonky Wagers

In making a wager, Watson first relies on mathematical models and algorithms, processing the data from its vast database, to determine the likelihood of a correct answer.

Answering the second question is far more involved. Watson uses a Game State Evaluator, a complex model that estimates its chances of winning based on such things as the competitors' scores, the number of remaining Daily Doubles and value of the clues remaining.

That technology also includes an in-category Daily Double confidence level, providing Watson with a view into its odds of winning a game based on a Daily Double bet. And its risk analytics software also weighs the likelihood of winning with a particular bet.

Because of Watson's betting strategy, it often ends up with nontraditional bets that forgo rounded values. Hence, the wonky $947 Toronto bet, versus a $900 or $1,000 bet.

Notes Tesauro: "Such values may make the arithmetic a little more challenging for the humans when computing their bets."

As if that's the only thing humans have to worry about when it comes to Watson. . .

View all Articles » Dawn Kawamoto is an award-winning business and technology reporter for DailyFinance. She previously covered the business angle of technology companies at CBS Interactive's CNET News and has also worked as a business reporter for the Los Angeles Daily News, Orange County Register and Santa Barbara News-Press.

Read More

View the original article here