@article{7d3f081b344e48e8abbfb89d3cc327f5,
title = "An evaluation framework for software crowdsourcing",
abstract = "Recently software crowdsourcing has become an emerging area of software engineering. Few papers have presented a systematic analysis on the practices of software crowdsourcing. This paper first presents an evaluation framework to evaluate software crowdsourcing projects with respect to software quality, costs, diversity of solutions, and competition nature in crowdsourcing. Specifically, competitions are evaluated by the min-max relationship from game theory among participants where one party tries to minimize an objective function while the other party tries to maximize the same objective function. The paper then defines a game theory model to analyze the primary factors in these minmax competition rules that affect the nature of participation as well as the software quality. Finally, using the proposed evaluation framework, this paper illustrates two crowdsourcing processes, Harvard-TopCoder and AppStori. The framework demonstrates the sharp contrasts between both crowdsourcing processes as participants will have drastic behaviors in engaging these two projects.",
keywords = "competition rules, crowdsourcing, game theory, software engineering",
author = "Wenjun Wu and Tsai, {Wei Tek} and Wei Li",
note = "Funding Information: Project concepts involve developing new ideas for software development. AppStori is an example that allows software developers to create new ideas and seek crowd funding for financial support. In general, multiple teams will come up with project concepts, yet the funding is limited, thus not all Funding Information: Harvard-TopCoder is a joint project funded by Harvard University to take advantage of the TopCoder platform for biomedical research. AppStori is a community-based, collaborative platform for the development of smart-phone applications. It provides a “preview” window for iPhone application enthusiasts to choose their favorite ideas, support and actively engage in the promising projects. Figs. 1 and 2 show the development process for both projects respectively. Funding Information: methodology to have a small but cohesive team to work on the coding. And the team has weak min-max relationship with other parties in the process, including the AppStori review board, funding contributor crowd, and beta tester crowd. • wmm (project team, review board) The founders of the project team can post a project proposal, specifying the features of the mobile application, the project budget, the potential revenue model in AppStore and milestones. They needs to minimize the issues of their proposal in front of the AppStori review board to make the proposal become an AppStori project online. The AppStori review board screens the submitted proposals by maximizing the weakness in the proposal to ensure the quality of the accepted AppStori projects. • wmm (project team, funding contributor crowd) Every AppStori project is funded by crowd. On the web page of the project, there is a progress indicator showing how much donation the project has received and how much to go by the deadline. The project team must endeavor to push the project toward the milestones defined in the proposal and minimize the probability that the donation crowd would withdraw their contributions. • wmm (project team, beta tester crowd) Any AppStori community member can sign up to become a beta tester in a project. The team must work collaboratively with beta testers from the community to minimize the bugs in the code. After the beta testing, the project will submit the product to the AppStore review board of Apple, which often assigns two reviewers to check the product following the AppStore review guidelines [20]. The quality of the accomplished product and the good communication with the community determine whether the project can eventually reach the funding goal. Funding Information: As a crowdfunding driven software crowdsourcing site, AppStori presents a different way to run crowdsourcing project. There are neither explicit ranking of individual designers and developers nor contest prizes for each project. Instead, each AppStori team estimates the effort for their project including its deadline and budget, and reach out to crowd for funding support. Therefore, the success of the project totally depends upon the dedication made by the project teams and the involved evaluators. For the perspective of the AppStori platform, it must constructively help application teams to refine their work to attract more donors for funding. We can apply the weak min-max model defined in Eqs. (7)–(9) to further elaborate this observation. Funding Information: Acknowledgements This project was sponsored by U.S. National Science Foundation project DUE 0942453 and National Nature Science Foundation of China (Grant No. 61073003), National Basic Research Program of China (2011CB302505), the Open Fund of the State Key Laboratory of Software Development Environment (SKLSDE-2009KF-2-0X and SKLSDE-2012ZX-19), and Fujitsu Laboratory.",
year = "2013",
month = oct,
doi = "10.1007/s11704-013-2320-2",
language = "English (US)",
volume = "7",
pages = "694--709",
journal = "Frontiers of Computer Science",
issn = "2095-2228",
publisher = "Springer Science + Business Media",
number = "5",
}