Network Computing is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IBM's Watson: A Watershed Event For Information Technology--And For Society: Page 2 of 5

Now, with chess, the measure of success is easy: Simply beat the world chess champion in a match of several games. But what is the equivalent benchmark for natural language questions and answers? Well, as it turns out, the long-televised U.S.-based game show "Jeopardy" provides the benchmark. In "Jeopardy," three contestants have to answer a variety of difficult natural language-posed questions and answers.

Let's look at what Watson has to do. IBM states that there are five dimensions that the technology of automatic question answering has to take into account for a "Jeopardy" game: broad/open domain, complex language, high precision, accurate confidence and high speed:

  • Broad/open domain This means questions touch on any subject and information that an ordinary person could have easy access to, but at non-trivial depth. For example, do you know what mitosis and cytokinesis are? Existing systems, such as expert systems, are very narrowly domain specific.
  • Complex language A natural language is ambiguous and contextual. There seems to be an endless number of ways to express the same meaning; if you tried to find a rule for everything, it would seem as if no matter how many rules you have, there would never be enough.
  • High precision A definitive single answer is needed; not a list of possible answers of which one may be correct.
  • Accurate confidence "Jeopardy" penalizes wrong answers, so guessing does not pay off. However, not answering questions does not pay off, either. Getting answers that meet or exceed a confidence threshold is critical.
  • High speed  To match humans, Watson has to reply in 3 seconds or less.

In essence, "Jeopardy" has served two purposes for IBM. One is to provide a source of questions for testing Watson and a measure of its improvement over time by comparing its success to what is necessary to succeed on "Jeopardy." The second is to serve as a "final examination" to determine whether Watson can beat the two best players in the game's history, which is the equivalent of simultaneously beating a pair of world chess champions. The result of this examination is to use "Jeopardy" as a showcase to the average consumer of the power of big data, analytics and workload optimized systems.

Watson is a self-contained hardware system, meaning that all the computing power, storage and network connections are local. There is no connection to the Internet, so Watson cannot search for information that is not already available. Rather than the custom-built Deep Blue components IBM used in the Kasparov matches, for Watson, the company put together a lot of commercially available hardware.

In fact, Watson represents what IBM calls a "workload-optimized" system. That said, the sheer numbers are impressive: 90 IBM Power 750 servers with 2880 POWER7 cores, 500GBytes per second on-chip bandwidth, 10GByte Ethernet network, 15 terabytes of memory and 20 terabytes of clustered disk storage. Overall, the system operates at 80 teraflops, and, obviously, massive parallelism is essential.

Doesn't this mean that IBM is simply throwing hardware at a problem, a la Deep Blue? The answer is no. Hardware is a necessary component of a natural language system but is not entirely sufficient. If sheer speed alone were the issue, computers would have won a long time ago. (Many computers operate technically faster than the human brain. For understanding more about the brain versus the computer from a hardware and software perspective, see Jeff Hawkins' "On Intelligence" and Ray Kurzweil's "The Singularity Is Near"..