Idea Evaluation Mechanisms for Collective Intelligence in Open Innovation Communities: Do Traders Outperform Raters?
Proceedings of 32nd International Conference on Information Systems, 2011
24 Pages Posted: 2 Oct 2011
Date Written: October 1, 2011
Abstract
The increasing popularity of open innovation approaches has led to the rise of various open innovation communities on the Internet which might contain several thousand user-generated ideas. However, a company’s absorptive capacity is limited regarding such an amount of ideas so that there is a strong need for mechanisms supporting the evaluation of these ideas. In this paper, we focus on the evaluation of such mechanisms for collective idea evaluation. Applying a multi-method approach, we compare six different configurations of a prediction market with a multi-criteria rating scale that performed best in previous research. We combine a web-based experiment with 448 participants, data from a participant survey, and an independent expert jury. Based on cognitive load theory, we explain why a multi-criteria rating scale outperforms prediction markets in terms of evaluation accuracy and evaluation satisfaction. This study contributes to theory building in the emerging field of collective intelligence.
Keywords: open innovation, collective intelligence, rating scales, prediction markets, idea evaluation, communities, crowdsourcing, cognitive load theory
JEL Classification: D80
Suggested Citation: Suggested Citation