Click here to support the Daily Grail for as little as $US1 per month on Patreon

Not Feeling the Future: New Bem Replication Fails to Find Evidence of Psi

Another possible hit this week for Professor Daryl Bem’s controversial “Feeling the Future” experiments, which found positive evidence for precognition, with the publication of a large-scale replication study which found no psi effect. The full paper, “Correcting the Past: Failures to Replicate Psi“, is freely available for download, and I recommend downloading it and having a good read. No doubt many will suffer (as I did) through the more technical descriptions of statistical analysis, but amongst that there is fascinating, respectful discussion of the Bem experiments and this latest replication attempt. In addition to the experiment results, the paper also features a meta-analysis of all replications attempted so far, which found again no significant evidence for precognition effects.

For those that can’t make it all the way through the paper, skeptic Steven Novella already has a large write-up of the new paper at Neurologica, in which he addresses many of the key points, as well as giving a broad overview of the entire Bem controversy.

Bem’s studies have not fared well in replication. Earlier this year Ritchie, Wiseman, and French published three independent replications of Bem’s 9th study, all negative. Of further interest is that the journal that originally published Bem’s article had declined to publish Ritchie et al’s paper claiming that they don’t publish replications. This decision (and editorial policy) was widely criticizes, as it reflects an undervaluing of replications.

It’s good to see that the journal has relented and agreed to publish a replication. Galak, LeBoeuf, Nelson, and Simmons should be commended, not only on their rigorous replication but their excellent article, which hits all the key points of this entire episode.

The researchers replicated experiments 8 and 9 of Bem (they chose these protocols because they were the most objective). They conducted 7 precise replications involving a total of 3,289 subjects (Bem’s studies involved 950 subjects). Six of the seven studies, when analyzed independently, were negative, while the last was slightly statistically significant. However, when the data are taken together, they are dead negative. The authors concluded that their experiments found no evidence for psi.

I leave it to more qualified minds than mine to authoritatively assess the merits of the paper. For what it’s worth, however, here’s my thoughts (caveats abounding):

I would imagine Bem would criticize this new replication on a key point, one which Novella glosses over in his assessment (in calling it a “rigorous” and “precise” replication) – that four of the seven experiments were done online, not in the lab. Additionally, one of the online experiments (#7) had roughly 2400 respondents, so across all 7 tests the amount of ‘lab’ results is only about 12%. This online aspect brings in a number of points of failure, from inattentiveness and distraction, right through to unintentional (by knowing about and thus being prepared for the ‘surprise’ test at the end) or intentional sabotage – it’s worth noting that the availability of the online test was passed around on skeptical forums such as the JREF and Rational Skepticism. Bem himself has criticized a previous paper from Galak et al. on this very point, saying when you do the test online, “you lose total control over it”. Interestingly, two of the three lab-based experiments done by the researchers had significantly lower p-values (p=0.04 and p=0.10) than the other tests.

Additionally, the researchers acknowledge explicitly that after reading Bem’s replication notes, they noticed that “there were at least three differences between our experiments (which followed the procedure described in Bem’s published paper) and the full procedure actually employed by Bem.” This included using a different word set to Bem for some tests. Steven Novella notes in his blog summary the importance of precise replications, saying that “a precise replication should have no degrees of freedom.” I find it hard to imagine how he reconciles this view with his support of the paper (describing it multiple times as a “precise” replication), and therefore that it “provides further evidence against psi as a real phenomenon, and specifically against the claims of Daryl Bem”.

Having said that, the paper itself does an admirable job of explaining these limitations/problems, and providing alternate analysis excluding some of these factors which appears to still show support for their conclusions. However, I had the distinct feeling that some of those exclusions were rather arbitrary (for instance, how to exclude possible sabotage), and in the end I’m not sure that they overcome the larger problem of (a) the massive online component and (b) the lack of precise replication, in terms of making this latest study acceptable as a true replication of Bem’s experiments. Nevertheless, fascinating reading and well worth your time.

You might also like…

  1. Amazes me how these
    Amazes me how these researches are totally ignorant of the fact that ESP abilities and such are often episodic and heavily dependant on the ambient environment.
    Tell you what – everybody watch the new upcoming season of “Long Island Medium.” It is very impressive, and most importantly it reveals that the ability has nothing to do with metaphysical mumbo jumbo and dark rites. For some people the ability is as natural as eating and loving, and as Theresa Caputo often says, everybody has the ability. It is just dormant in most people most of the time. The revolution coming upon us is that it will be awakened in many more people in the near future, and then every time we look at people like Romney and Obama we will damn well know what a pack of liars they are and not even let them into the house.

  2. Hello!,
    I have yet to read


    I have yet to read both the original paper and the replication, but as to this articles’ comments, I can readily say a few things:

    -On online experiments vs lab experiments:
    There exists a growing literature, specially due to studies ran through Amazon’s Mechanical Turk, on replication of lab effects online. Many effects do hold on online replications, regardless of the suggested limitations. Which means that, though it is important to run studies both in the lab and online, argumenting simply on the basis of it being an online study is not sufficient to discredit its conclusion anymore than you should discredit the conclusions of any single study. Also see the comment below.

    -On exact replications vs conceptual replications:
    You should read Klaus Fiedler’s opinion on this subject. In a very short note I would say that an exact replication is an exact waste of time. If an effect is real and robust, all that is required is a conceptual replication that controls for other confounding variables. Indeed, an exact replication may simply perpetuate the effect of variables that are irrelevant to the theory but cause the effect (e.g. the exact set of words used in the study vary in some dimension that primes the target; therefore generating an effect that can be interpreted as precognition but is in effect simple priming).

    Have a nice day!,

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Mobile menu - fractal