An interview with a next generation methodological freedom fighter

Meet Anne Scheel, PhD candidate at LMU Munich. She stood up and asked the author of the opinion piece on “methodological terrorism” for a statement after her keynote at the DGPS conference. Since tough questions in front of big audiences by young women are still a rare thing to encounter at conferences (and elsewhere), we were curious to know how this went down for her. And of course, we also took the opportunity to discuss the content of the piece in question.

(For those who have missed this recent 🔥storm🔥 in the context of the replication crisis: The above piece is a rather explicit criticism of researchers active in the field of replication (keywords “methodological terrorism”, “destructo-critics”, “self-appointed data police”, “ad hominem smear tactics”). In addition to reading the above piece, we recommend this detailed timeline of why our field is where we are now and why we are not terrorists.)

Cogtales: You (as a young female researcher) stood up and asked a very tough question. We are curious what motivated you to do that?

Susan Fiske’s commentary is a very broad attack against discussions of research on social media. She admits that “not all self-appointed critics behave unethically”, but she refuses to “name names” or pinpoint what kind of criticism/discussion is ok and what isn’t. This is what upset me: Because the piece is so vague, a naive reader might get the impression that academic social media is a lawless Wild West of unjustified personal attacks blasting left and right towards anyone who looks the wrong way. Which it obviously isn’t.
I have personally benefitted immensely from academic social media and I think it is the driving force behind the current movement to push psychology towards greater transparency, reproducibility, and robustness. I want to motivate more colleagues to chime in on these online discussions. But Fiske’s commentary might do the opposite: It communicates a severe distrust towards social media; and it possibly deters people who previously had been on the fence about getting more involved.

I wanted to talk to her about her commentary for two reasons: First, because I found it unfair and insulting and thought that specifying what/whom she was targeting was crucial for a constructive debate. Second, because I want psychology as a field to move forward, educate itself, adopt better practices. That won’t happen – or at least it won’t happen as quickly – if we draw a line in the sand and toss insults at each other. I am convinced that (unmoderated!) discussions on social media are helpful and even vital for that process, and I genuinely want to learn about the beliefs and worries of people who don’t share this view.

Ok, but I think you also wanted to know where I took the courage?
Just before Fiske’s keynote I was standing with a couple of people from the Network of Open Science Initiatives (NOSI [German only unfortunately]), discussing if/how we should try to confront her with her APS Observer commentary. In that moment I decided that I’d try anything to get that question through. We would never get a better chance to make this an open discussion – it was timely, it was in person, it was in front of many people who might not even have heard about her commentary. And I have absolutely nothing to lose: I am just a little PhD student, and as a developmental psychologist, I’m not even in her field.
Yes, I was nervous, so nervous in fact that I had serious worries about being able to hold the microphone. But that was just stage fright, not a fear of negative consequences for myself. I feel that some more senior but untenured people would have had to put more on the line than I did.

Cogtales: That’s interesting that you as a PhD student feel like you have less to lose – many would argue that you are especially vulnerable in this position, where you are quite dependent on your supervisor and you will soon be in the position to search for a position. Can you say a bit more about that?

You are right in that I feel much more vulnerable within my own working group. I often am reluctant to fight more openly for or against certain practices when I feel that my colleagues and especially supervisors are sceptical. This is something that really upsets me: I am basically betraying my own ideals by being a bystander to things I don’t agree with. Implicit norms can be a hard nut to crack when you are very worried about offending people and especially when you’re at the bottom end of the hierarchy. But I’m trying to get better!

But in this case I was faced with someone maximally far away from me. I’m sure that people in Susan Fiske’s position theoretically have enough power to influence an untenured researcher’s career. But I am so junior that I have no career that could be affected yet. She certainly didn’t know my name before and I doubt that she knows it now – I’m just not important to her in any way. So being a nobody allows me to fly under the radar.

Cogtales: You are also quite active on twitter. We feel that, much like for asking questions in front of conference audiences, young female researchers are rather underrepresented in scientific exchange on twitter or via blogs. Would you agree?

Absolutely! I just checked: only 32% of my Twitter followers are women, and I’m guilty myself – only 37% of the people I follow are women. But to be honest, it feels to me that the actual engagement of women in online discussions is even lower than one third.

Cogtales: And do you have an idea why this is the case?

I have no scientific evidence to back up the following, but from my personal experiences, I think these are the main problems: a) Women tend to be much less confident about their knowledge/reasoning, b) at the same level of confidence as men, women tend to give the consequences of being wrong more weight than the consequences of being right when they consider speaking up. In other words, women feel that they have more to lose.

The men I spend time with (for some reason most of my friends are male) are often much more keen to debate and don’t worry half as much about negative consequences. You speak your mind, you get flak, you shoot back, you have another drink. It’s more of a game.

Online discussions can appear more rude because tone is harder to convey in written language, commentators are less inclined to take your feelings into account, and sometimes because the character limit simply leaves no space for politeness. If you fear negative consequences of speaking up, that might be even more pronounced on social media.

I can relate to that, I’m also very averse to being called out on stupid mistakes. That’s why I try to re-read and fact check pretty much everything I post. But some may perceive the bar as too high to really participate. The pace of online discussions is fast, at some point you just have to shoot from the hip. I think the trick is to just tell yourself again and again that a) nobody is going to kill you and b) 80% of those other morons don’t have more of a clue than you do, just a bigger mouth. In other words – imposter syndrome therapy!

Cogtales: Do you ever feel that being a young female researcher on social media puts you in a different position from your colleagues (and more so than in more “classical” forms of communication like presenting a poster or publishing a paper?)

I don’t feel that I get treated differently on social media if that is what you are asking. That is, I don’t feel that being female makes much of a difference. The thing I am more aware of is status – many of my interactions on Twitter are with more senior researchers. This is what I love about Twitter: There is no hierarchy, and most people are amazingly open and accessible. They don’t let me feel that I am more junior than they are, but I often worry about not being experienced or knowledgeable enough to make meaningful contributions to a discussion.

Cogtales: So now that we’re already on topic – a key point of the opinion piece was a criticism of the use of new media (twitter, facebook, blogs) to evaluate published findings. One argument was the lack of peer review – what would you say to that?

In my opinion the lack of peer review on social media is precisely what makes it so great!

First, peer review is slow. Getting a commentary published can take weeks, months, even more than a year! In contrast, on Twitter and Facebook (e.g., PsychMAP or Psychological Methods Discussion Group), interesting papers frequently get analysed within hours after they were published – or even before publication, when they are uploaded to a preprint server (or linked to as supplementary material in a press release).

Apart from being slow, traditional peer review is still an intransparent process (although some journals make an effort to improve the situation). It is not objective and not reliable and can therefore not be valid (e.g., Peters & Ceci, 1982; Hodgson, 1997). Peer review in its current form allows a few people in power to control the landscape of scientific publication.

Don’t get me wrong – many editors and reviewers out there are doing an amazing job and work their asses off while getting shamefully little recognition for it. But the system is vulnerable to misconduct – Questionable Review Practices, if you like. Publication bias is the number one threat to a robust literature. Sloppy research practices are one thing, but if all 19 null results to every sexy finding were out there, we would be in much less trouble today.

Opinions that go against the mainstream are often particularly hard to publish. But we need these opinions. And every article needs to be discussed. Three peer reviewers at one point in time cannot replace the hive mind. That’s why I would love to see more structured forums for open pre- and post-publication peer review. PubPeer is one platform for that, but unfortunately it seems to have a negative connotation to some – “if a paper has a PubPeer comment, it can’t be trusted”. The truth is that every single paper is flawed, and we know it. We just haven’t gotten used to embracing an open discussion about our research as a fundamentally benign, constructive, and necessary process.

As Brian Nosek puts it, we need to “disentangle publication from evaluation”. If publication is the default (e.g. via preprint servers like PsyArXiv), our reputations don’t hinge on it any more and hopefully that will make it easier to tolerate criticism/discussion.

Cogtales: This question of new media, and a lot of other points that were being made, seem to us like a huge generation or cultural gap in approaches to science. The two parties in this dispute are not really arguing on the same level, but rather come from completely different places (this comment illustrates that quite well for me). So we wonder, when you were asking the question at the conference, how did that feel? Were you satisfied with the answers you got and the ensuing discussion?

First off, as I mentioned before, I felt very lucky to get the chance to ask this question in person. Talking to someone directly always makes it easier for me to understand where that person is coming from and I also think we’re more likely to take things to heart that are said to our face.

Susan Fiske could have easily dismissed my question as it was not about her talk and there wasn’t much time, but she chose to respond. She just said that she objected to unmoderated online discussions and that she knew of a significant number of people who felt threatened by personal attacks. She then offered to discuss this issue further afterwards, so Felix Schönbrodt, Malte Elson, Ruben Arslan, Julia Rohrer and I talked to her for another 15 minutes or so.

I’m not satisfied with her answers in the sense that neither of us seemed to have much of an impact on the other’s opinion. And she actually defended her “methodological terrorism” analogy – that bit I found shocking. But it was a civil interaction. I am grateful for her giving us the chance to tell her about our perspective – that didn’t go without saying.

Cogtales: Another key point was evolving around “name-calling” and personal accusations. What do you think about that?

I have witnessed personal attacks in online discussions. Who hasn’t? But relative to the total number of interactions I would be very surprised if they were more common online than in conventional scientific media. If you believe that peer review is always fair, balanced, and polite, have a look at Dale Barr’s terrific Shit My Reviewers Say. In our discussion, Susan Fiske also mentioned having witnessed smear campaigns attacking tenure candidates with anonymous letters to the tenure committee. Of course things like that are terrible and we should try to reduce their prevalence. But that is not a social media problem!

The discussion about tone started before Fiske’s commentary came out. Tal Yarkoni has written a terrific blog post about it and I find myself agreeing with him on almost every point.

In scientific discussions, we need to distinguish between research and researcher. If you want to talk about research, your arguments should be as impersonal as possible. But, importantly, if they are not, that fact alone does not excuse the addressee from responding to them if they have substance. I prefer to be polite and assume the best intentions in my opponent because a) I think that’s what basic decency demands and b) the other person will be more likely to listen. But I also find that people can be overly sensitive. This is science! Talk about the facts and respond to arguments, don’t take it so personal to begin with. I might have a pretty radical position here, but I believe that publishing research obligates you to discuss it with anyone who publicly questions it.

Personal attacks are never ok, but putting on a thicker skin would also help us to stick to the actual discussion points (I have to keep reminding myself of that too).

Cogtales: Do you have any other thoughts on the ongoing debate you’d like to share with us?

Just one thing that is on my mind a lot lately: In my answer to the last question, I said that we shouldn’t conflate research and researcher when what we want to talk about is research. But there are times when we actually want to talk about the researcher.

Some have called the replication crisis a crisis of trust and I agree with that notion – too much is looking suspicious with False-Positive Psychology goggles on. I think many people suspect that QRPs were rampant in at least some fields in the last decades, but it is very hard to pinpoint that because basically none of these studies were pre-registered and data and scripts are hardly ever made available. What’s left is a bad taste in your mouth and a growing anger in your gut.

I think anger is justified here. P-hacking, HARKing, and running underpowered studies is unethical when it is funded by taxpayers’ money, when the time of study participants is wasted (and sometimes their health is put to risk), and when it increases the chance that public funds will be wasted in attempts to build on this flawed research in the future. This behaviour should have consequences. Yet in the current system, accountability seems to be near zero. There is no mechanism in place that holds researchers accountable for practices that lead to a corrupted literature but cannot clearly be classified as fraud. I believe that that fact increases the likelihood of personal attacks in discussions that should only be about research. In a way, some may really behave like a “self-appointed data police” as Fiske calls it.

When we talk about how to improve research practices, I’m all for focussing on the future and not shame researchers for their past behaviour. But there is a discussion to be had about what happened in the past. Of course it is hard or impossible to prosecute something that is so vague that we call it “questionable” rather than “wrong” or “fraudulent” – and of course there is a huge grey area. But at least I would like to see a discussion about how we can establish greater accountability in the future.

Cogtales: Thanks for the interview, Anne!

Thank you for the interesting questions and for giving me this opportunity! I sincerely apologise for my long answers 😉 I really admire your blog, keep up the great work!


3 thoughts on “An interview with a next generation methodological freedom fighter

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s