What happens when you stand up to the big wigs? A follow-up interview with Anne Scheel

Two years ago, Team CogTales (Sho and Christina) interviewed Anne Scheel. We were impressed how she stood up to ask a tough question at Germany’s largest Psychology conference (the DGPs Kongress) after a keynote presentation. Two years later, Christina and Anne actually met up at the next installment of the very same conference, and a lot has changed in the short time span. So it seems like a perfect moment to catch up and take stock.

CogTales: Hi Anne, thank you for talking to us again. And of course we are curious to hear whether you think your actions had any consequences for you and / or your career.

Anne: Unfortunately we didn’t set this up as an RCT so I have no idea how control-group Anne is doing, but I’ll happily overinterpret the correlational data and say that it’s had quite an impact on my career — in a snowball-/Matthew-effect kind of way: First I gained a lot of Twitter followers and the attention of some more influential people, who then gave me a platform (like you!), which led to more people knowing me, and so on. Now I regularly get invited to give workshops and talks, I’ve been a guest on two podcasts… if you had told me that two years ago, I would have recommended you to go easy on the mushrooms. I still think it’s crazy.

In fairness, some of this has (very likely) been the result of starting a blog with three friends, as we’ll talk about below — the causal relations are hard to disentangle!

One of the few things that can be traced back to my DGPs moment with certainty is that several people have bought me a drink for it (you guys are amazing).

CogTales: You presented and co-chaired a symposium at this year’s DGPs conference as well, congratulations. How did your session on “Ramping up Rigor: Current Practices, Problems, Potential Solutions” go? Do you think the field has changed in the past years?

Anne: We had a bit of a thankless slot on the first morning of the conference but still got a nicely filled room and some really interesting discussions. I found this year’s DGPs fascinating — so many open science/reproducibility sessions! There were several different ones every day, some by people I hadn’t known before, including a large symposium and keynote on theory and a policy-focussed panel with representatives from several psychological societies.

I think it’s safe to say that the movement has grown in width and depth. My experience is probably not representative and I’m likely to overestimate adoption rates, but it feels like e.g. preregistration has become so common that in most psychology departments you might meet someone who has done it. At the same time, the discussions about improved research practices have become more specialised and cover more ground; see for example the increasing whispers about the “theory crisis”. To me that’s progress. We started discovering our problems at the far and most concrete end — the numbers that are churned out by our research process — and since we lifted the rug, we’ve been tracing the problem back to its roots. It’s a long way, and the questions that arise become more and more philosophical, which unfortunately doesn’t make them easier to answer. But in any case, it feels like the need for reform is turning into an accepted narrative. I find that very exciting.

 

CogTales: We also noted that you are now part of a blog team, of course we hope the positive experience with CogTales helped in that decision. How do you experience blogging?

Anne: Funnily enough, The 100% CI is also a consequence of DGPs 2016. After Susan Fiske’s keynote, Malte Elson, Julia Rohrer, and Ruben Arslan were among the people who stuck around to talk to her, and after the conference the four of us stayed in touch and eventually decided to start a blog together.

It’s been a great experience! I can highly recommend blogging in a team — it’s much easier to provide regular(ish) content and reach a wider audience, and putting your thoughts out into the world feels less scary because you can review each other’s posts before publishing. And it’s just way more fun. Not that I’d need to tell you guys that!

We’ve gotten lots of positive responses and I learned a lot from the others and from discussions with our readers. That said, I have to admit that I’m a terrible blogger; I take ages to write anything. At the moment it’s so bad that I can’t really take any credit for the blog at all, it’s the other three who keep it going. Luckily they’re not just funny and smart but also some of the kindest people I know and bear with me…

 

CogTales: Do you think more early career researchers should blog?

Anne: It depends! For the individual (blogger) it’s a great opportunity to structure your thoughts, practice writing, and of course make a name for yourself and build your network. But I personally have a relatively high bar for what I think should be published (cf. taking ages to write a post…). People have limited time and there are so many blogs out there, so if you want impact, I think you need to find a niche and/or aim for (very) high quality. And that can then add a lot of self-inflicted pressure you otherwise wouldn’t have.

But if your main goal is to practice writing and your blog is intended as more of a public diary or writing tool, these (potential) drawbacks don’t apply of course. And I can see a lot of value in using it that way! Blog posts are usually a lot less formal than scientific papers, and that can be a great help when you struggle to put your thoughts on a blank page.

 

CogTales: Two years ago we talked about the skepticism social media assessments of published papers face. Has it become more accepted to blog about a paper? Is blogging and tweeting now more of a typical aspect of publishing and communicating science or do we still have a long way to go?

Anne: That is a really interesting question. On the one hand I think that publicly criticising published papers has become a lot more common in the last two years. Take the “pizza papers” affair, the avalanche of faulty papers by Brian Wansink that was uncovered by Tim van der Zee, Nick Brown, Jordan Anaya, and James Heathers: This investigation has been a constant source of news in my corner of the Twittersphere for the last couple of years, and helped popularise error detection and “forensic statistics” e.g. using tools like statcheck, GRIM/GRIMMER, and SPRITE. “We found inconsistencies in a paper” is a sentence I find a lot less surprising (or alarming) now than I did a few years ago.

On the other hand, criticising individual authors still seems to make people very uncomfortable — so uncomfortable that it reliably triggers pushback. I still get the strong sense that some people are only fine with criticism if it is directed against a behaviour or a group of people that is ill-defined enough to prevent individual group members from being identified. I have no numbers on this, but this “effect” feels exactly as strong to me as it did two years ago, and I find that really interesting.

Regarding the second part of your question, blogging and tweeting about research feels like a very “established” way of communicating to me, and I regularly see blog posts cited in published articles. But I’m never quite sure how much that reflects an actual change in the field and how much is just a result of the bubble I’m in — it’s absolutely possible that less Twitter-focussed colleagues would disagree with me on this.

 

CogTales: So it seems like we’ve already come a long way, but there’s still a lot to do. Tell us – what’s the most pressing issues we need to resolve in the way we do Science in the years to come?

Anne: At the moment I’m very concerned about standards for the new research practices that have been promoted in the last years, for example preregistration and data sharing. The concept “preregistration” is currently very poorly defined — it can be anything from a few lines of text stating a vague hypothesis to an extremely detailed, watertight research protocol including the analysis code. If the label “preregistered” applies to anything between these extremes, it doesn’t tell us anything and will inevitably lose its value in the long run (a recent study by Veldkamp et al. shows that even relatively strictly-reviewed preregistrations leave a lot of room for researcher degrees of freedom). And it leads to hard feelings: You now see cases popping up where a new finding from a “preregistered” study is published that some sceptics don’t find believable and proceed to criticise the preregistration as too vague. That then leads to surprised/frustrated responses from the authors, who feel unfairly treated because they already “went the extra mile” and still get flak.

If we want to get as many researchers as possible into this boat and also make sure that the new steps they are required to take really make their research more reproducible, the instructions have to be clear and fool-proof — without suffocating research lines that don’t fit neatly into the bog-standard hypothetico-deductive mould of experimental psychology research.

Similar issues apply to data sharing: Making your data publicly accessible has become a lot more common, but currently the used file formats, repositories, and data documentation are a huge mess, which is a problem for automated meta-analyses and long-term usability. Another aspect of data sharing that worries me is what happens when existing datasets do get reused: How can we make sure that secondary analyses don’t turn into the same overfitting spree that we just started to get a grip on for primary analyses? A group of people led by Sara Weston have done great work to develop a framework for preregistering analyses of pre-existing data, and I agree with them that it is possible. But what if individual researchers are less concerned with making use of such safeguards?

I see a risk of “collective overfishing” of existing datasets in the long run and wonder if we should replace open data with lightly-gated data: Imagine a central data-sharing platform where you have to request access to any dataset you want to retrieve. Authors could decide how strongly they want to restrict access to their data, e.g. fully private or access only for selected individuals/institutions. In the most “open” setting, anyone’s request for access would be approved immediately, but this instance of granted access would always be tracked and time-stamped. This kind of regulation could take a lot of strain off the the current system, which relies (too) heavily on interpersonal trust for my liking (nullius in verba!). As added benefits, a central platform would make it easier to establish standardised data structures and meta-data, and could even include built-in tools to facilitate and regulate cross-validation (e.g. split requested datasets into training data and holdout and restrict access to the holdout for a certain time period).

Long story short: I think we’ve seen many great new ideas and solutions to our problems sprouting in the last years, and now we’re entering a phase of pruning and channelling those ideas into effective, actionable, sustainable tracks — in other words: new standards. Developing standards that achieve their goals with minimal negative side effects is an extremely hard, but extremely important task.

Advertisements

Screen, Baby  – Let’s Look at Evidence, not Trends

Joint post by Nawal Abboub, PhD, & Sho Tsuji, PhD

French version here

Who hasn’t heard that “Children under two years of age should absolutely not be exposed to screen media, no matter what!” – maybe accompanied by the reasoning that “Screens will hinder the development of children’s intelligence.”

Why does the topic of the effect of screens on young children, especially with regard to their brain development, evoke so much controversy and fear? And should we actually think of all types of screen media as equal? What can scientific research teach us? Continue reading Screen, Baby  – Let’s Look at Evidence, not Trends

Through the eyes of an undergraduate student: I was part of ManyBabies, an international collaboration project

Guest post by Meghan Mastroberardino, third year undergraduate student in Psychology at Concordia University

So, you think you might want to have a career in psychology? In North America, most people who end up calling themselves psychologists began as undergraduate students in a Bachelor of Psychology program and have then completed a PhD. I am a third year undergraduate student in Psychology at Concordia University. One of the best ways to get a better picture of grad school is to volunteer in a research lab and take part in research projects. I have found that it has been challenging journey but it’s when I joined the Concordia Infant Research Lab and met my supervisor in my second year, Dr. Krista Byers-Heinlein, that I felt that maybe psychology really was meant for me. I pushed myself and took on as much responsibility as I could in the lab and for the past year, she and I have worked closely together on a large-scale, pre-registered study called ManyBabies1. Continue reading Through the eyes of an undergraduate student: I was part of ManyBabies, an international collaboration project

Rejection report: Part 1

Sometimes, things just fall into place:  The evening before the most recent Academic Crisis Line on dealing with rejection and frustration, I got a pre-holiday manuscript rejection. As pointed out by the crisis liners, rejection in academia happens to everyone on a rather regular basis. So what we should really be concentrating on is to deal with it in the most self-preserving and productive ways possible. One thing that can really help is to talk through it, and to connect with others in similar situations.

So I thought it would be an interesting experiment to share my unfiltered thoughts while I deal with this rejected paper here on this blog.

Continue reading Rejection report: Part 1

Why we don’t need to be afraid that preregistration prevents creativity

One argument that often comes up when I talk to skeptics of preregistration is that it stands in the way of creative and exciting research. I couldn’t disagree more. Preregistration and registered reports are among the very best developments that have come out of Psychology’s replication crisis. Both guide a way towards better research. But since the sentiment that preregistration and creativity are not compatible is so prevalent and seems so genuine (as opposed to being an excuse to engage in questionable research practices), I do want to expand on some main reasons why preregistration does not dampen creativity in research. Continue reading Why we don’t need to be afraid that preregistration prevents creativity

Scientists need more Europe, not less

This month, we are celebrating the anniversary of Germany’s reunification. This event is very important to me, as it reminds me every year, together with the anniversary of the fall of the Berlin wall in November, how lucky I am to be able to live the international scientist nomad life. Although this lifestyle with short term contracts has a lot of downsides, it is also a unique opportunity to work with various amazing people and follow your curiosity and ambition where it takes you. Continue reading Scientists need more Europe, not less

The Straightforward Academic (or: Your advisor also poops)

This entry is part of a summer series over at the wonderful My Scholarly Goop, featuring true tales of early career researchers’ scholarly paths. Read my contribution here, and head over to the series to get a new essay each Friday, the whole summer long.

Not pursuing a PhD was, frankly, never part of my thought process. My parents are both researchers (albeit, in Biophysics and Physical Chemistry), and I grew up spending Sunday afternoons in the lab proudly reproducing the Briggs-Rauscher reaction in my own little lab coat while my parents were working. I’m a bit like the medical doctor who became a doctor because her parents are also MDs. You could say I tried to be a bit different and decided to become an opthamologist instead of a dermatologist, but that’s really about how crazy I went. Continue reading The Straightforward Academic (or: Your advisor also poops)

The only real mistake is the one from which we learn nothing

Naomi Havron is currently a postdoc at the LSCP in Paris, France. She is investigating syntactic adaptation and syntactic category learning in children and infants.

I spent four years writing my thesis, four years of ups and downs, p values smaller than .05, but also some t values smaller than .05. At times, I felt confident and optimistic, at times less so – this was somewhat correlated with the p values, but not significantly so.

Then it came time to start gathering everything I did into an article thesis. In the Hebrew University, at least in my year, not all of these articles had to be published articles, and you could also include manuscripts you did not plan to submit for publication. I thought I would write up everything, even my null results, because whoever reads my dissertation (well, at least all those 1st year PhD students and my grandfather) could probably learn just as much from my failed attempts than from my success stories. Continue reading The only real mistake is the one from which we learn nothing

Shouting from the ivory tower: SciComm for beginners

This week, I am tweeting not from my own account, but from the science communication “rocur” (rotating curator) account realsci_DE, the German version of realscientists. This is the latest instance of a few experiences in talking about science and my work as a scientist to a non-expert audience, and I’ve learned a great deal from and about this in the past. So I thought I’d share a few insights that I had along the way (fun fact: there are no natural born communicators, it all needs practice).

First tell me how this helps scientists!

Continue reading Shouting from the ivory tower: SciComm for beginners

We Stand in Solidarity with Scientists Across the Globe

Two weeks ago, I came back to the US after my holiday trip to Germany and France. This moment when you stand in front of the frosty immigration officer who makes you press all your fingers onto the dirty glass of the fingerprinting scanner, takes a webcam photo of your travel-exhausted face, and then scrutinizes your papers. This moment alone always makes me feel like an illegitimate intruder. But of course, it always goes well for me. This last time the immigration officer, still with her poker face on, noticed that my visa would run out in 3 months. Yes, I said hurriedly, I need to reapply. Ever wondered why they make you do this, the officer asked. I looked at her, slightly alarmed. To make a shitload of money out of you, she said, looked up, and smiled a friendly smile. I also smiled, relieved. We both laughed, and that is how I re-entered the United States of America.

Continue reading We Stand in Solidarity with Scientists Across the Globe