Following up on the use of Mechanical Turk in research, it turns out there’s more flaws to this approach than just bot-induced corruption, as NewScientist notes:
People seem to be answering research survey questions randomly on Amazon’s crowdsourcing website. The findings could mean that many academic studies are wrong.
Michael Chmielewski at Southern Methodist University in Texas and Sarah Kucker at Oklahoma State University recently revisited data they had collected on Amazon’s Mechanical Turk (MTurk) platform, a virtual labour marketplace where people are paid to perform short tasks. MTurk is often used to gather survey responses for social science research.
Since 2015, Chmielewski and Kucker had used MTurk to collect data on how a child’s language skills developed depending on their parents’ personalities. When New Scientist published an article in 2018 claiming automated bots were targeting the site and ruining academic studies, the pair revisited their data and found inconsistencies. But rather than bots ruining their data, it seems humans racing through possible survey answers and not reading the questions were causing the problems.
By performing a statistical analysis on their results, the team found that the responses just weren’t right. “The conclusions were just massively wrong,” says Chmielewski. “Well-established links between neuroticism and depression weren’t there. We were seeing links in the wrong directions. Things that should have been negatively related were now positively related.”
One of the most difficult elements of science is collection of data, and it appears Mechanical Turk was merely an illusion of a source of data. A worker elaborates:
Kristy Milland who does work offered on MTurk wasn’t surprised by the results. “MTurk is a labour platform, not a participant pool,” she says. “We signed up to make money, not to help science.”
Self-reporting and other sources that might be classified as secondary must always be treated with skepticism by researchers, even when that is the only available data. As Chmielewski & Kucker demonstrate, it’s possible, in many cases, to examine the data for consistency and plausibility, even though that might seem to be placing an otherwise unwarranted expectation on the data. In the future, more researchers should do the same.
