Idea Evaluation Mechanisms for Collective Intelligence in Open Innovation Communities: Do Traders Outperform Raters?

Proceedings of 32nd International Conference on Information Systems, 2011

24 Pages Posted: 2 Oct 2011

See all articles by Ivo Blohm

Ivo Blohm

University of St. Gallen

Christoph Riedl

Northeastern University - D’Amore-McKim School of Business

J. M. Leimeister

University of St. Gallen; University of Kassel - Information Systems

Helmut Krcmar

Technische Universität München (TUM)

Date Written: October 1, 2011

Abstract

The increasing popularity of open innovation approaches has led to the rise of various open innovation communities on the Internet which might contain several thousand user-generated ideas. However, a company’s absorptive capacity is limited regarding such an amount of ideas so that there is a strong need for mechanisms supporting the evaluation of these ideas. In this paper, we focus on the evaluation of such mechanisms for collective idea evaluation. Applying a multi-method approach, we compare six different configurations of a prediction market with a multi-criteria rating scale that performed best in previous research. We combine a web-based experiment with 448 participants, data from a participant survey, and an independent expert jury. Based on cognitive load theory, we explain why a multi-criteria rating scale outperforms prediction markets in terms of evaluation accuracy and evaluation satisfaction. This study contributes to theory building in the emerging field of collective intelligence.

Keywords: open innovation, collective intelligence, rating scales, prediction markets, idea evaluation, communities, crowdsourcing, cognitive load theory

JEL Classification: D80

Suggested Citation

Blohm, Ivo and Riedl, Christoph and Leimeister, Jan Marco and Krcmar, Helmut, Idea Evaluation Mechanisms for Collective Intelligence in Open Innovation Communities: Do Traders Outperform Raters? (October 1, 2011). Proceedings of 32nd International Conference on Information Systems, 2011, Available at SSRN: https://ssrn.com/abstract=1936590

Ivo Blohm

University of St. Gallen ( email )

Müller-Friedberg-Strass 8
St. Gallen, 9000
Switzerland

Christoph Riedl (Contact Author)

Northeastern University - D’Amore-McKim School of Business ( email )

360 Huntington Ave.
Boston, MA 02115
United States

HOME PAGE: http://www.christophriedl.net

Jan Marco Leimeister

University of St. Gallen ( email )

Varnbuelstr. 14
Saint Gallen, St. Gallen CH-9000
Switzerland

University of Kassel - Information Systems ( email )

Pfannkuchstraße 1
Kassel, 34121
Germany

Helmut Krcmar

Technische Universität München (TUM) ( email )

Arcisstrasse 21
Munich, DE 80333
Germany

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
638
Abstract Views
3,156
Rank
76,367
PlumX Metrics