The Effect of Rating Scales on Decision Quality and User Attitudes in Online Innovation Communities

International Journal of Electronic Commerce (IJEC), 17(3), 7-36, 2013.

36 Pages Posted: 19 Oct 2012 Last revised: 22 Mar 2013

See all articles by Christoph Riedl

Christoph Riedl

Northeastern University - D’Amore-McKim School of Business

Ivo Blohm

University of St. Gallen

J. M. Leimeister

University of St. Gallen; University of Kassel - Information Systems

Helmut Krcmar

Technische Universität München (TUM)

Date Written: October 19, 2012

Abstract

Given the rise of Internet, consumers increasingly engage in co-creating products and services. Whereas most co-creation research deals with various aspects of generating user-generated content, this study addresses designing ratings scales for evaluating such content. In detail, we analyze functional and perceptional aspects of two frequently used rating scales in online innovation communities. Using a multi-method approach, our experimental results show that a multi-criteria scale leads to higher decision quality of users than a single-criteria scale, that idea elaboration (i.e., idea length) negatively moderates this effect such that the single-criteria rating scale outperforms the multi-criteria scale for long ideas, and finally that the multi-criteria scale leads to more favorable user attitudes towards the website. Based on our experimental data, we applied a bootstrap-based Monte Carlo simulation for assuring robustness of our results. We find that around 20 user ratings per idea are sufficient for creating stable idea rankings and that a combination of both rating scales leads to a 63% performance improvement over the single-criteria rating scale and 16% over the multi-criteria rating scale. Our work contributes to co-creation research by offering insights as to how the interaction of the technology being used (i.e., rating scale), and attributes of the rating object affects two central outcome measures: the effectiveness of the rating in terms of decision quality of its users and the perception of the scale by is users as a predictor of future use.

Keywords: Website, online innovation community, rating, scale, decision quality, Monte Carlo, bootstrap, user-generated content, co-creation

Suggested Citation

Riedl, Christoph and Blohm, Ivo and Leimeister, Jan Marco and Krcmar, Helmut, The Effect of Rating Scales on Decision Quality and User Attitudes in Online Innovation Communities (October 19, 2012). International Journal of Electronic Commerce (IJEC), 17(3), 7-36, 2013., Available at SSRN: https://ssrn.com/abstract=2164362

Christoph Riedl (Contact Author)

Northeastern University - D’Amore-McKim School of Business ( email )

360 Huntington Ave.
Boston, MA 02115
United States

HOME PAGE: http://www.christophriedl.net

Ivo Blohm

University of St. Gallen ( email )

Müller-Friedberg-Strass 8
St. Gallen, 9000
Switzerland

Jan Marco Leimeister

University of St. Gallen ( email )

Varnbuelstr. 14
Saint Gallen, St. Gallen CH-9000
Switzerland

University of Kassel - Information Systems ( email )

Pfannkuchstraße 1
Kassel, 34121
Germany

Helmut Krcmar

Technische Universität München (TUM) ( email )

Arcisstrasse 21
Munich, DE 80333
Germany

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
152
Abstract Views
1,327
Rank
347,535
PlumX Metrics