Frank van Kolfschooten / Psychologist's defense challenged

Science 30 May 2014:
Vol. 344 no. 6187 pp. 957-958
DOI: 10.1126/science.344.6187.957

Psychologist's defense challenged
Frank van Kolfschooten*

With a once distinguished career, a €5 million grant, and a new professorship all hanging in the balance, social psychologist Jens Förster has vigorously fought back against recent charges that some of his results are so statistically improbable that they could only have resulted from data manipulation. But a key element of his defense has also exposed Förster, who recently resigned from the University of Amsterdam (UvA), to a new line of attack.

E-mails reviewed by Science call into question his assertion that some of the studies at the heart of the controversy were not conducted recently at UvA but were done much earlier in Germany over a decadelong period ending in 2008. In explaining certain puzzling aspects of the study subjects, and his inability to provide the original data files on them, Förster had said the studies took place long ago at another institution. The e-mails between Förster and a UvA colleague discuss how to conduct various aspects of some of those studies—yet the e-mails were sent in May 2009, undercutting Förster's timeline.

This perplexing saga began in 2012 when another psychologist expressed concern to UvA about unusually large effects that seemed improbably consistent, reported in 40 studies described in three papers. After participants had been “primed” by subtle stimuli, such as hearing poems, their scores on a cognitive ability test rose significantly. The complaint sparked a UvA investigation that delved into the statistics of Förster's publications and found some “virtually impossible” results in papers published in 2009, 2011, and 2012, but did not conclude misconduct. A second inquiry, by the Netherlands' National Board for Research Integrity, did conclude that data had been manipulated in the 2012 paper published in Social Psychological and Personality Science, however. UvA has recommended that that paper be retracted, but Förster has not acceded.

In addition to questioning Förster's results, the person who filed the complaint had expressed skepticism about the recorded information on the study subjects, noting, for example, that the sex ratio of the participants was very different from that among students at UvA. The psychologist also found it suspicious that the 2012 paper and two others recorded not a single study dropout, out of more than 2200 participants across 40 experiments, and not a single participant who had guessed the purpose of the study—even though many of them were presumably psychology students at UvA.

In a 11 May posting on his website, Förster reiterated a previous denial of data manipulation and stressed that his challenger had made a key mistaken assumption—the studies documented in the 2011 and 2012 papers, Förster says, were conducted from 1999 to 2008 in Germany, mostly at Jacobs University Bremen, with the help of more than 150 co-workers. Förster further hinted that one of those people might have manufactured the exceptional results. “I can also not exclude the possibility that the data has been manipulated by someone involved in the data collection or data processing. … During the time of investigation I tried to figure out who could have done something inappropriate. However, I had to accept that there is no chance to trace this back; after all, the studies were run more than 7 years ago and I am not even entirely sure when, and I worked with too many people,” he wrote.

In an earlier posting on 29 April, Förster wrote that he had thrown away the original questionnaires for the challenged studies when he moved to a much smaller office in Amsterdam. (In that posting, he did not note where the studies were done.) In his 11 May statement, Förster noted that he provided both investigations with processed data files, one of which contained questionnaire answers in German, which he says proves the experiments were performed in Germany and not in Amsterdam as had been assumed. Those files are time-stamped February 2013, however, according to Förster's challenger and a second source, who has access to the data files but does not want to be named.

Förster's accuser says the integrity committee of UvA should have asked for data files with a time stamp before September 2012, when the complaint was filed. He also says the UvA erred by not confiscating Förster's computer, as often happens in such investigations. (The whistleblower's identity is known to Förster, the university, and the national investigating body, but he agreed to talk to Science only if not named.)

The real challenge to Förster's timeline may lie in e-mails between him and Pieter Verhoeven, his research assistant at UvA from September 2008 to June 2009, who made the correspondence available to Förster's accuser. In it, the two discuss how to conduct what are evidently the same experiments Förster's blog declares were completed much earlier in Bremen. For instance, among the stimuli used are three unintelligible audio recordings, which the 2011 paper says were described to the subjects as “Moldavian” poems. In an 18 May 2009 e-mail, Verhoeven comes up with the idea to describe the poem that way, rather than as Malaysian, because the reader of the poem has a German accent.

In another e-mail, sent to Verhoeven on 13 May 2009, Förster reports having found “little boxes” at a home appliance store. “I'm optimistic that this will work,” Förster writes. The 2012 and 2011 papers describe having study subjects touch an object “that consisted of four square plastic boxes.”

Verhoeven has confirmed to Science that these e-mails discussing details of studies seemingly described in the 2011 and 2012 paper are genuine. “Reading back our correspondence 5 years later, I can only conclude we were still working on the exact design of the experiments in May 2009,” says Verhoeven, now an interior designer in Amsterdam, who says he has no animosity toward Förster. He provided the e-mails only after Förster's accuser approached him.

Förster's accuser has also informed UvA that six more of his papers contain statistically improbable data. UvA, however, tells Science that it will take no action because there has been no filing of a “new, formal, and substantiated” complaint.

In lieu of any new investigation, colleagues are judging Förster themselves, often posting their views online. For example, psychologist Uri Simonsohn of the University of Pennsylvania, whose statistical analysis of the data published by psychologist Dirk Smeesters of Erasmus University Rotterdam exposed his scientific misconduct, joined with Leif Nelson of University of California, Berkeley, to analyze the 2012 paper. On 8 May on the blog datacolada.org, they describe conducting 100,000 data simulations and other statistical analyses to examine how likely the results are. “[W]e have a conceptual replication of ‘these data are not real,’” they concluded.

Förster has not responded to Science's multiple attempts to reach him. As the psychology community mulls the bewildering tale, he awaits word from Ruhr University Bochum, which has postponed plans for him to start a professorship endowed with €5 million in funding from the German Alexander von Humboldt Foundation. The university and the foundation now say they may not address the matter until October.

* Frank van Kolfschooten is a freelance writer in Amsterdam and the author of two books on scientific misconduct.