Posted by: Gabriele Paolacci | February 27, 2012

Screening AMT workers using Qualtrics

Duplicate respondents across related experiments are a substantial problem for conducting programmatic research on AMT (Chandler, Mueller, & Paolacci). In the tutorial linked below, we provide a straightforward alternative that allows researchers who use Qualtrics to exclude workers who participated in a previous study. This approach allows researchers to exclude workers who have completed any HIT before, without having to use AMT’s Command Line Tools.

We conducted a test HIT with 100 AMT workers who were asked to follow a link to Qualtrics that was posted using the script in the tutorial. In terms of browser, there were Firefox, Chrome, Internet Explorer, and Opera users. In terms of operating system, there were Windows users, Mac users, and Linux users, with various versions. Nobody reported seeing any error message and all proceeded to the survey. All of the Worker IDs recorded on Qualtrics matched the results of the HIT, therefore the code seems to be stable and reliable. However, the tutorial is subject to improvements. Should you find any bug or have suggestions, please contact the authors or better comment on this post. The tutorial was updated on May 2, 2012. Further tutorials will follow, and you will find an updated list in the Resources section.

Selectively Recruiting Participants from Amazon Mechanical Turk Using Qualtrics
Eyal Peer (Carnegie Mellon University), epeer@andrew.cmu.edu
Gabriele Paolacci (Erasmus University)
Jesse Chandler (Princeton University)
Pam Mueller (Princeton University)

You can also download a sample .qsf file that you can import on your Qualtrics account.

(Suggested citation: Pe’er, Eyal, Paolacci, Gabriele, Chandler, Jesse and Mueller, Pam, Selectively Recruiting Participants from Amazon Mechanical Turk Using Qualtrics (May 2, 2012). Available at SSRN: http://ssrn.com/abstract=2100631 or http://dx.doi.org/10.2139/ssrn.2100631)

About these ads

Responses

  1. Thank you for this tutorial. I have built a HIT using the command line tools and am linking the HIT to a Qualtrics survey. Given I used command line to build the HIT, do you know what might need to change to use this process? In short, is there a different way to paste in the script and pass things to Qualtrics?
    Thanks.

  2. The HTML script was specifically written for the web interface. I am not sure how one could implement the same principles using CLT. It’s something worth looking at…

  3. Hi, I very much appreciate your tutorial as well. I am encountering one problem I can’t seem to figure out though — I followed the instructions as they were laid out; however, when I get to the preview and finish mode the text does not change to “the link will appear here only if you accept this HIT.” It remains “URL not shown because there is an error with Javascript on your computer. To Perform this HIT, you must have Javascript and cookies enabled on your browser.” Is there a way to get around this error message? I am using the web interface, so I don’t think it is a command line tool error (and Javascript and cookies are enabled on my browser.)

    Thank you so much for your help!
    Jessica

  4. Hi Jessica,
    Are you sure you’re using the latest version of the tutorial (May 2)? It should refer to this script as the one to paste on mturk: http://dl.dropbox.com/u/3863544/files%20to%20keep/appending%20ID%20script.txt
    It seems to work well… Feel free to email me if needed!
    Gabriele

    • Hi Gabrielle,

      Thank you for your prompt reply. I did try the May 2 version now, and I think it is working well (just got the first hit response!!)

      Thank you for all of your help!!
      Jessica

    • Hi Gabrielle,
      I am having the exact problem that Jessica was having. Even when I use the script from May 2. Is there anything I can do?

  5. We have a list of over 4,000 Workers we need to exclude. Qualtrics returns an error that says this is too much for the Embedded Data function–which seems a reasonable limit, given the field is embedded as a value for every respondent. Any ideas how to use your strategy to look the data up from the list without storing it all in embedded data? If not, it seems that we will have to build a PHP or Rails app to do this–are there any prefab solutions you recommend?

    • Hi Neil, in the second note at the end of our document we added that we believe the limit to be about 2000 ids. In any case, you can expand that limit by creating more than one “excluded” embedded data (e.g., “excluded1″, “excluded2″, etc.) and split the workers ids into each of them. Then, you will simply need to change the survey flow so that the workerid is checked against “excluded1″, or “excluded2″ etc. We’ve used this method before and it should work. Thanks, Eyal.

  6. Thanks so much for these instructions! I think I’ve followed them all correctly, but I can’t get it to work.

    At the design layout stage I input the script with the modified Qualtrics url. I get the message “URL not shown because there is an error with Javascript on your computer. To perform this HIT, you must have Javascript and cookies enabled on your browser”. So far so good.

    However, when I hit Preview, I get a red bullet at the top of the next page saying “Dhtml template must contain a question,” along with a message that says “Unable to update your project”.

    In the HIT preview window I get a Qualtrics message saying “Unfortunately this survey has been closed. Thank you for your time.” But the survey is active.

    I can’t get past this stage. Any advice?

    • Hagop, it seems like the errors are not related to the script: the “Dhtml template must contain a question” error means you did not include a question in your HIT template. Try to re-create it starting from their basic “survey link” template. The “survey closed” error has something to do with how you set up the survey on qualtrics. Check the survey options there, especially the ones under “survey termination” to see why this is happening.

  7. Thanks for your response, Eyal. I did as you said and started a completely new project using the “survey link” template. I cut and pasted the code from the dropbox link above as is, in its entirety, without replacing the existing Qualtrics address (just to see if it would work as is). I still get the same messages from Amazon. MTurk refuses to even save the code. I tried on Chrome and Explorer. Don’t know what’s going on.

    It’s a shame because I really appreciate the work that’s been put into this, and think it would be invaluable. I know I must be doing something wrong but can’t figure out what it is.

    Am I supposed to do something else other than changing the existing Qualtrics URL in the code to my own survey link?

    • No, you’re not supposed to change anything else. Can you send me the full HIT code to my email – epeer@andrew.cmu.edu – and I’ll try to de-bug it?
      Eyal.

  8. This was a big help! As an FYI, I was having the same problem as Jessica and Erin when I used the pdf document (even the May 2 doc) but when I used the link you provide in the comments here, it worked without any problems. Thanks so much.

  9. This is awesome! Thank you so much for all the effort that went into creating and sharing this post.

    I have a question about handling rejected workers. Since the script requires workers to accept the HIT before they see the link, rejected “repeat” workers (via the Exclude branch) still have to click Submit HIT, right? And therefore they will also get paid because in the system it would seem as if they’ve completed it?
    I haven’t done many HITs as a worker myself but I seem to recall that if you return an already accepted HIT it counts against you. If I am wrong about this perhaps the message upon exclusion should also say “Please return this HIT.” or something like that?

    Thank you!

    • Returned HITs do not count as rejected ones. Workers have the right to return any HIT if they decide to do so within the allotted time. I simply make sure that the time for the HIT is enough and not too short and that works fine. You can customize the exclusion message to include that if you like, but after running numerous studies with this approach, I did not get any complaints of this sort. Best of luck with your research, Eyal.

      • Oh that’s great! Thank you! It worked as a charm and no complaints so far :).

  10. I’d like to echo the thanks for this page. We learned a lot from it. However, we are most concerned about preventing duplicate responses in real time, rather than preventing responses from respondents to different studies. So we weren’t thrilled with the prospect of needing to do manual updates of large blocks of embedded data on a regular basis. We also wanted to avoid a third-system solution such as a custom PHP/MySQL middleman site; handling AMT and Qualtrics is complicated enough, we don’t need another point of failure. We had a long call with Qualtrics support, and they identified a real-time dupe-prevention strategy that we are planning to pilot early in 2013.

    The strategy involves using Qualtrics’ panel trigger feature to add mTurk WorkerIDs to a panel in a preliminary survey, then shunting to a second survey that uses an authenticator to check the panel for the WorkerID. The panel won’t prevent duplicate rows, but it doesn’t appear to matter–our tests indicate the authenticator feature will allow only one response per WorkerID, even if it’s in the panel multiple times. We’ll let you know if it works and publish thorough instructions with a link to them here.

    • That sounds very interesting. Do share with us whatever you managed to implement. Thanks, Eyal.

  11. To all:
    I am looking to build a qualification using the command tools so that the workers taking my survey is specific to only one state through the use of thier zipcode.
    I am so new to the use of A Turk that I am looking for someone to hire to help me produce the survey with the specific qualifications.

    Please let me know if you are at all interested!!!!

  12. Hi,I’m having the same problem as Jessica and Erin when I used the pdf document, the link provided on the page, etc. with the javascript not working. I’ve tried replacing the http:// part of the survey and when I realized my link was including an https:// I replaced only the link portion of the survey location instead of including the http:// part. Do you have any advice on what else I can try? Sorry to be a pest as you’re all kind enough to provide this to the academic community for free but I’m stumped and am not sure how to proceed.

    Thanks,
    Jackie

  13. I love what you guys are usually up too. Such clever work
    and reporting! Keep up the great works guys I’ve you guys to my blogroll.

  14. Hi Eyal,

    I really like this post and trying to implement this for my research. However, I had the same problem with Hagop which is “Dhtml template must contain a question.” I copy paste the instruction to MTurk and only changed the URL. It seems that Mturk wants me to provide a space for respondents to fill an answer (e.g. code). Please help!! Thanks!

  15. Hi Tyson,

    That is correct — our script does not substitute the usual instructions that people put on mturk to redirect workers to their study, including the box where participants are supposed to write the code you give them at the end of your study. You can use the “Survey link” template that MTurk provides on this page: https://requester.mturk.com/create/projects/new and substitute “survey link …” with our html code. Good luck!

    Gabriele


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: