| |
Social Science Research Network Working Paper Series (13 October 2010)
Abstract
Amazon’s Mechanical Turk is an online labor market where "requesters" post jobs and "workers" choose which jobs to do for pay. The central purpose of this paper is to demonstrate how to use this website for conducting behavioral research. We describe general techniques that apply to a variety of types of research and experiments across disciplines. We begin by discussing some of the advantages of doing experiments on Mechanical Turk, such as easy access to a large, diverse subject pool, low ...
|
| |
In Find Work in Wikipedia. Intelligent User Interfaces (IUI (2007), pp. 32-41
Abstract
Member-maintained communities ask their users to perform tasks the community needs. From Slashdot, to IMDb, to Wikipedia, groups with diverse interests create communitymaintained artifacts of lasting value (CALV) that support the group’s main purpose and provide value to others. Said communities don’t help members find work to do, or do so without regard to individual preferences, such as Slashdot assigning meta-moderation randomly. Yet social science theory suggests that reducing the cost and increasing the personal value of contribution would motivate members ...
|
| |
Abstract
Six incentive experiments were conducted in a nonprofit online panel. In each experiment, the incentive offered for participation was a cash lottery. The control group was not offered any incentive. The cash lottery was offered in two versions: Either the total payout of the lottery was mentioned, or the lottery was split into multiple prizes. Dependent measures included response and retention rates. The results of the six individual experiments were meta-analytically summarized. Cash lotteries relative to no incentives did not reliably ...
|
| |
|
| |
Abstract
Implicit preferences for Whites compared to Blacks can be reduced via exposure to admired Black and disliked White individuals (Dasgupta & Greenwald, 2001). In four studies (total N= 4,628), while attempting to clarify the mechanism, we found that implicit preferences for Whites were weaker in the “positive Blacks” exposure condition compared to a control condition (weighted average d= .08). This effect was substantially smaller than the original demonstration (Dasgupta & Greenwald, 2001; d= .82). Factors beyond exposure to admired Blacks may ...
|
| |
Abstract
A longstanding idea in the literature on human cooperation is that cooperation should be reinforced when conditional cooperators are more likely to interact. In the context of social networks, this idea implies that cooperation should fare better in highly clustered networks such as cliques than in networks with low clustering such as random networks. To test this hypothesis, we conducted a series of web-based experiments, ...
|
| |
Judgment and Decision Making, Vol. 5, No. 5. (2010), pp. 411-419
Abstract
Although Mechanical Turk has recently become popular among social scientists as a source of experimental data, doubts may linger about the quality of data provided by participants recruited from online labor markets. We address these potential concerns by presenting new demographic data about the Mechanical Turk subject population, reviewing the strengths of Mechanical Turk relative to other online and offline methods of recruiting participants, and comparing the magnitude of effects obtained using Mechanical Turk and traditional subject pools. We further discuss ...
|
| |
Abstract
The web provides an unprecedented opportunity to evaluate ideas quickly using controlled experiments, also called randomized experiments, A/B tests (and their generalizations), split tests, Control/Treatment tests, MultiVariable Tests (MVT) and parallel flights. Controlled experiments embody the best scientific design for establishing a causal relationship between changes and their influence on user-observable behavior. We provide a practical guide to conducting online experiments, where end-users can help guide the development of features. Our experience indicates that significant learning and return-on-investment (ROI) are seen ...
|
| |
Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc, Vol. 35, No. 4. (November 2003), pp. 614-620
Abstract
This study evaluated the psychometric equivalency of Web-based research. The Sexual Boredom Scale was presented via the World-Wide Web along with five additional scales used to validate it. A subset of 533 participants that matched a previously published sample (Watt & Ewing, 1996) on age, gender, and race was identified. An 8 x 8 correlation matrix from the matched Internet sample was compared via structural equation modeling with a similar 8 x 8 correlation matrix from the previously published study. The ...
|
| |
American Psychologist, Vol. 59, No. 2. (0 2004), pp. 105-117
Abstract
As the Internet has changed communication, commerce, and the distribution of information, so too it is changing psychological research. Psychologists can observe new or rare phenomena online and can do research on traditional psychological topics more efficiently, enabling them to expand the scale and scope of their research. Yet these opportunities entail risk both to research quality and to human subjects. Internet research is inherently no more risky than traditional observational, survey, or experimental methods. Yet the risks and safeguards against ...
|
| |
In Arizona-Nevada Academy of Science Annual Meeting (April 2005)
|
| |
Abstract
Online experiments have recently become very popular, and—in comparison with traditional lab experiments— they may have several advantages, such as reduced demand characteristics, automation, and generalizability of results to wider populations (Birnbaum, 2004; Reips, 2000, 2002a, 2002b). We replicated Dandurand, Bowen, and Shultz’s (2004) lab-based problem-solving experiment as an Internet experiment. Consistent with previous results, we found that participants who watched demonstrations of successful problem-solving sessions or who read instructions outperformed those who were told only that they solved problems correctly ...
|
| |
Cyberpsychology & behavior : the impact of the Internet, multimedia and virtual reality on behavior and society, Vol. 5, No. 4. (August 2002), pp. 305-313
Abstract
The Internet can be a valuable data collection tool for organizational psychology researchers. It can be less expensive than traditional paper-and-pencil survey methods, and the potential pool of participants is much larger. In addition, it can be used in situations where traditional data collection methods are not feasible, such as research involving sensitive issues such as negative employee attitudes or deviant behaviors at work. In this study, we examined the organizational attitudes of employees from various companies using (a) a snowball ...
|
| |
Cyberpsychology & behavior : the impact of the Internet, multimedia and virtual reality on behavior and society, Vol. 6, No. 1. (February 2003), pp. 73-80, doi:10.1089/109493103321167983
Abstract
The Internet can be an effective medium for the posting, exchange, and collection of information in psychology-related research and data. The relative ease and inexpensiveness of creating and maintaining Web-based applications, associated with the simplicity of use via the graphic-user interface format of form-based surveys, can establish a new research frontier for the social and behavioral sciences. To explore the possible use of Internet tools in psychological research, this study compared Web-based assessment techniques with traditional paper-based methods of different measures ...
|
| |
Abstract
The rapid growth of the Internet provides a wealth of new research opportunities for psychologists. Internet data collection methods, with a focus on self-report questionnaires from self-selected samples, are evaluated and compared with traditional paper-and-pencil methods. Six preconceptions about Internet samples and data quality are evaluated by comparing a new large Internet sample (N = 361,703) with a set of 510 published traditional samples. Internet samples are shown to be relatively diverse with respect to gender, socioeconomic status, geographic region, and ...
|
| |
Behav Res Methods, Vol. 37, No. 2. (May 2005), pp. 287-292
Abstract
The Web Experiment List (http://genpsylab-wexlist.unizh.ch/), a free Web-based service for the recruitment of participants in Internet-based experiments, is presented. The Web Experiment List also serves as a searchable archive for the research community. It lists more than 250 links to and descriptions of current and past Web experiments. Searches can be conducted by area of research, language, type of study, date, and status (active vs. archived). Data from log file analyses reveal an increasing use of the Web Experiment List and ...
|
| |
Behav Res Methods Instrum Comput, Vol. 33, No. 2. (May 2001), pp. 201-211
Abstract
In fall 1995, the worldwide-accessible Web Experimental Psychology Lab (http://www.genpsylab.unizh.ch) opened its doors to Web surfers and Web experimenters. It offers a frequently visited place at which to conduct true experiments over the Internet. Data from 5 years of laboratory running time are presented, along with recommendations for setting up and maintaining a virtual laboratory, including sections on the history of the Web laboratory and of Web experimenting, the laboratory's structure and design, visitor demographics, the Kids' Experimental Psychology Lab, access ...
|
| |
Behavior Research Methods, Instruments, & Computers, Vol. 33, No. 2. (1 May 2001), pp. 226-233
Abstract
There are many methodological differences between Web-based studies, differences that could substantially affect the results. The present study investigated whether sample type, offering payment through a lottery, and requiring participants to enter personal information would affect dropout rates and/or the substantive results in a study of jury decision making in capital cases. Asking participants to enter their e-mail addresses increased dropout rates, and offering payment through a lottery tended to do so as well. Participants offered payment tended to be less ...
|
| |
Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc, Vol. 35, No. 2. (May 2003), pp. 217-226
Abstract
Previous research has made a beginning in addressing the importance of methodological differences in Web-based research. The present paper presents four studies investigating whether sample type, financial incentives, time when personal information is requested, table design, and method of obtaining informed consent influence dropout and sample characteristics (both demographics and measured attitudes). Undergraduates were less likely to drop out than nonstudents, and nonstudents offered a financial incentive were less likely to drop out than those offered no incentive. Complex tables, tables ...
|
| |
Abstract
New information technology (e.g., Internet) allows some personnel selection procedures to be adapted to or developed with this new framework. However, the process of adaptation or development of new procedures produces new questions for research. This paper has three main objectives. First, to examine whether the paper-and-pencil version of a Big Five personality questionnaire can be translated to an Internet-based version without loss of psychometric properties. Second, to explore the perception and reactions of the examinees to the new version of ...
|
| |
|
| |
Abstract
The purpose of this research was to illustrate the broad usefulness of simple video-game-based virtual environments (VEs) for psychological research on real-world behavior. To this end, this research explored several high-level social phenomena in a simple, inexpensive computer-game environment: the reduced likelihood of helping under time pressure and the bystander effect, which is reduced helping in the presence of bystanders. In the first experiment, participants had to find the exit in a virtual labyrinth under either high or low time pressure. ...
|
| |
Abstract
Abstract Social scientists are often interested in understanding how the dynamics of social systems are driven by the behavior of individuals that make up those systems. However, this process is hindered by the difficulty of experimentally studying how individual behavioral tendencies lead to collective social dynamics in large groups of people interacting over time. In this study, we investigate the role of social influence, a process well studied at the individual level, on the puzzling nature of success for cultural products ...
|
| |
|
| |
|
| |
(2001)
Abstract
[Collection of articles, including discussion of issues in online survey research, online experiments, and perception research. Discusses procedures involved in developing an experimental Web laboratory] Internet Science is a new and exciting interdisciplinary field. Its purpose is the conduct of empirical studies which examine the Internet as both an instrument for, and an object of, scientific investigation. This book is the first comprehensive collection of contributions to Internet Science appearing in English, written by highly respected experts from seven different countries. Dimensions ...
|
| |
Abstract
Although Internet-based experiments are gaining in popularity, most studies rely on directly evaluating participants' responses rather than response times. In the present article, we present two experiments that demonstrate the feasibility of collecting response latency data over the World-Wide Web using WebExp-a software package designed to run psychological experiments over the Internet. Experiment 1 uses WebExp to collect measurements for known time intervals (generated using keyboard repetition). The resulting measurements are found to be accurate across platforms and load conditions. In ...
|
| |
Abstract
Internet-based psychological experimenting is presented as a method that needs careful consideration of a number of issues—from potential data corruption to revealing confidential information about participants. Ten issues are grouped into five areas of actions to be taken when developing an Internet experiment (dos) and five errors to be avoided (don’ts). Dos include: (a) utilizing dropout as a dependent variable, (b) the use of dropout to detect motivational confounding, (c) placement of questions for personal information, (d) using a collection of ...
|
| |
Being There Together (November 2010), pp. 205-231
Abstract
This chapter gives an overview of how multi-user virtual environments are increasingly being used for social research, beginning with a typology and review of these. One advantage is that all interactions can be recorded, and another that experiments can be done that would not be possible in face-to-face interactions. The chapter also details different methods that are being used, including participant observation or ethnographies, large-scale logging of populations, analyzing small group interaction. Some examples of small group interaction analysis are provided ...
|
| |
|
| |
Abstract
Data from Web-delivered experiments conducted in browsers by remote users of PsychExperiments, a public on-line psychology laboratory, reveal experiment effects that mirror lab-based findings, even for experiments that require nearly millisecond accuracy of displays and responses. Textbook results are obtained not just for within-subjects effects, but for between-subjects effects as well. These results suggest that existing technology is adequate to permit Web delivery of many cognitive and social psychological experiments and that the added noise created by having participants in different ...
|
| |
Abstract
In fall 1995, the worldwide-accessible Web Experimental Psychology Lab (http://www.genpsylab. unizh.ch) opened its doors to Web surfers and Web experimenters. It offers a frequently visited place at which to conduct true experiments over the Internet. Data from 5 years of laboratory running time are presented, along with recommendations for setting up and maintaining a virtual laboratory, including sections on the history of the Web laboratory and of Web experimenting, the laboratory’s structure and design, visitor demographics, the Kids’ Experimental Psychology Lab, ...
|
| |
Abstract
Self-control problems have recently received considerable attention from economic theorists. We conducted two studies involving behavioral interventions expected to affect performance, providing some of the first experimental data in this area. In the first we investigate whether evenly spaced interim deadlines lead to higher completion rates for a lengthy task, where procrastination could be a factor. We found that these interim deadlines in fact led to lower completion rates in our set-up; we also found no evidence in the aggregate data ...
|
| |
Social Science Research Network Working Paper Series (16 April 2010)
Abstract
Online labor markets have great potential as platforms for conducting experiments. They provide immediate access to a large and diverse subject pool, and allow researchers to control the experimental context. Online experiments, we show, can be just as valid - both internally and externally - as laboratory and field experiments, while often requiring far less money and time to design and conduct. To demonstrate their value, we use an online labor market to replicate three classic experiments. The first finds quantitative ...
|
| |
Abstract
Two contemporary promises of participatory democracy are addressed in the present article; (a) democratic decisions reached in small group deliberation between lay citizens and (b) the possibility to exploit information and communication technology in democratic dialogue and decision making. Initially, a quasi-experimental approach was used to explore the potential and impact of face-to-face deliberation between citizens. In this design, a random sample of adult citizens was first surveyed and invited to take part in the experiment. The original face-to-face experiment was ...
|
| |
Abstract
Combining evolutionary models with behavioral experiments can generate powerful insights into the evolution of human behavior. The emergence of online labor markets such as Amazon Mechanical Turk (AMT) allows theorists to conduct behavioral experiments very quickly and cheaply. The process occurs entirely over the computer, and the experience is quite similar to performing a set of computer simulations. Thus AMT opens the world of experimentation to evolutionary theorists. In this paper, I review previous work combining theory and experiments, and I ...
|
| |
Abstract
When running online experiments, getting numbers is easy; getting numbers you can trust is hard. ...
|
| |
Abstract
Computer-simulated virtual environments (VEs) offer promise for assessing people's spatial abilities in large real-world environments. This paper introduces the WALKABOUT, a VE-based test that simulates navigation through large spaces. Participants' ability to form an accurate mental representation of a familiar large-scale environment correlated nearly as highly with their performance on the WALKABOUT as it did with their self-reported sense of direction. Performance on two subscales of the test further indicated that the ability to account for changing egocentric relationships as a ...
|
| |
Abstract
It is increasingly easy and, therefore, increasingly common to conduct experiments and questionnaire studies in online environments. However, the online environment is not a data collection medium that is familiar to many researchers or to many research methods instructors. Because of this, researchers have received little information about how to address ethical issues when conducting online research. Researchers need practical suggestions on how to translate federal and professional ethics codes into this new data collection medium. This article assists United States ...
|
| |
Abstract
Three studies examined the notion that computer-mediated communication (CMC) can be characterised by high levels of self-disclosure. In Study One, significantly higher levels of spontaneous self-disclosure were found in computer-mediated compared to face-to-face discussions. Study Two examined the role of visual anonymity in encouraging self-disclosure during CMC. Visually anonymous participants disclosed significantly more information about themselves than non-visually anonymous participants. In Study Three, private and public self-awareness were independently manipulated, using video-conferencing cameras and accountability cues, to create a 2 × 2 design ...
|
| |
Experimental psychology, Vol. 49, No. 4. (2002), pp. 243-256
Abstract
This article summarizes expertise gleaned from the first years of Internet-based experimental research and presents recommendations on: (1) ideal circumstances for conducting a study on the Internet; (2) what precautions have to be undertaken in Web experimental design; (3) which techniques have proven useful in Web experimenting; (4) which frequent errors and misconceptions need to be avoided; and (5) what should be reported. Procedures and solutions for typical challenges in Web experimenting are discussed. Topics covered include randomization, recruitment of samples, ...
|
| |
Abstract
In recent years the Internet has become a useful tool for carrying out research. The authors provide information specific to relationship research regarding qualitative and quantitative research methods, the advantages and disadvantages of online research, and practical concerns researchers may have regarding data quality and validity, sampling issues, and ethical considerations. They conclude with an empirical example of how the Internet can be used to carry out longitudinal, daily diary, self-report, and relationship research. ...
|
| |
Abstract
Web experiments generate insights and promote innovation. ...
|
| |
Abstract
Stanley Milgram's 1960s experimental findings that people would administer apparently lethal electric shocks to a stranger at the behest of an authority figure remain critical for understanding obedience. Yet, due to the ethical controversy that his experiments ignited, it is nowadays impossible to carry out direct experimental studies in this area. In the study reported in this paper, we have used a similar paradigm to the one used by Milgram within an immersive virtual environment. Our objective has not been the ...
|
| |
Abstract
Online virtual worlds, electronic environments where people can work and interact in a somewhat realistic manner, have great potential as sites for research in the social, behavioral, and economic sciences, as well as in human-centered computer science. This article uses Second Life and World of Warcraft as two very different examples of current virtual worlds that foreshadow future developments, introducing a number of research methodologies that scientists are now exploring, including formal experimentation, observational ethnography, and quantitative analysis of economic markets ...
|
