Reply to comment

The contention that all studies have noise is reasonable, but the question still remains whether or not the noise is worse in web studies. In a web study, a person could easily re-do it a dozen times without the experimenter being any the wiser. I don't think this is possible in person. Even more importantly, if the study is on something controversial, one person who is emotionally or financially motivated to support one particular result could easily create a bot to repeatedly give the answer he wants, completely destroying the value of the study.

At the very least, these studies need to have the visual character identification used by many sites to keep out bots. The "have you taken this test before" sort of question is not very useful for this. To become truly accepted by the establishment, I think it necessary to show that carefully designed web studies give the same results as more established methods and the uncertainty factor associated with the result demonstrated.