Building Watson: It's Not So Elementary My Dear!

Abstract

This case is set inside IBM Research's efforts to build a computer that can successfully take on human challengers playing the game show Jeopardy! It opens with the machine named Watson offering the incorrect answer 'Toronto' to a seemingly simple question during the championship match. Was the answer a reflection of a strategic weakness, or was it actually consistent with design principles established by the development team? The case seeks to expand students' view of the product development process. Traditional software development projects begin with the gathering of requirements and analysis of the problem, and the writing of a detailed specification. The Jeopardy problem is different - it requires a probabilistic approach where there is no closed form solution. Instead statistical patterns in the data are important and there is no obvious mapping to the way queries are expressed. Such problems are increasingly common in data mining, optimization problems, or Internet applications where the goal is to find an acceptably good solution in a short amount of time, when a deterministic approach might be less fruitful or impractical. We aspire for students to recognize that product development can take many forms, and that these are enabled by creativity and the right organizational flexibility and mindset. The case highlights the key role of performance metrics in building a flexible system that could be refined through experimentation and testing, steadily improving performance with the incorporation of new algorithmic ideas and new data sources. The case then delves extensively into the analysis of the 'Toronto' failure and why the answer that Watson produced was a rational product of a sound strategy. This leaves students to judge the generality of the strategy and its applicability to important business problems.

Learning Objective: Illustrate an alternate view of the product development process in software.