What happens when you stand up to the big wigs? A follow-up interview with Anne Scheel

Two years ago, Team CogTales (Sho and Christina) interviewed Anne Scheel. We were impressed how she stood up to ask a tough question at Germany’s largest Psychology conference (the DGPs Kongress) after a keynote presentation. Two years later, Christina and Anne actually met up at the next installment of the very same conference, and a lot has changed in the short time span. So it seems like a perfect moment to catch up and take stock.

CogTales: Hi Anne, thank you for talking to us again. And of course we are curious to hear whether you think your actions had any consequences for you and / or your career.

Anne: Unfortunately we didn’t set this up as an RCT so I have no idea how control-group Anne is doing, but I’ll happily overinterpret the correlational data and say that it’s had quite an impact on my career — in a snowball-/Matthew-effect kind of way: First I gained a lot of Twitter followers and the attention of some more influential people, who then gave me a platform (like you!), which led to more people knowing me, and so on. Now I regularly get invited to give workshops and talks, I’ve been a guest on two podcasts… if you had told me that two years ago, I would have recommended you to go easy on the mushrooms. I still think it’s crazy.

In fairness, some of this has (very likely) been the result of starting a blog with three friends, as we’ll talk about below — the causal relations are hard to disentangle!

One of the few things that can be traced back to my DGPs moment with certainty is that several people have bought me a drink for it (you guys are amazing).

CogTales: You presented and co-chaired a symposium at this year’s DGPs conference as well, congratulations. How did your session on “Ramping up Rigor: Current Practices, Problems, Potential Solutions” go? Do you think the field has changed in the past years?

Anne: We had a bit of a thankless slot on the first morning of the conference but still got a nicely filled room and some really interesting discussions. I found this year’s DGPs fascinating — so many open science/reproducibility sessions! There were several different ones every day, some by people I hadn’t known before, including a large symposium and keynote on theory and a policy-focussed panel with representatives from several psychological societies.

I think it’s safe to say that the movement has grown in width and depth. My experience is probably not representative and I’m likely to overestimate adoption rates, but it feels like e.g. preregistration has become so common that in most psychology departments you might meet someone who has done it. At the same time, the discussions about improved research practices have become more specialised and cover more ground; see for example the increasing whispers about the “theory crisis”. To me that’s progress. We started discovering our problems at the far and most concrete end — the numbers that are churned out by our research process — and since we lifted the rug, we’ve been tracing the problem back to its roots. It’s a long way, and the questions that arise become more and more philosophical, which unfortunately doesn’t make them easier to answer. But in any case, it feels like the need for reform is turning into an accepted narrative. I find that very exciting.


CogTales: We also noted that you are now part of a blog team, of course we hope the positive experience with CogTales helped in that decision. How do you experience blogging?

Anne: Funnily enough, The 100% CI is also a consequence of DGPs 2016. After Susan Fiske’s keynote, Malte Elson, Julia Rohrer, and Ruben Arslan were among the people who stuck around to talk to her, and after the conference the four of us stayed in touch and eventually decided to start a blog together.

It’s been a great experience! I can highly recommend blogging in a team — it’s much easier to provide regular(ish) content and reach a wider audience, and putting your thoughts out into the world feels less scary because you can review each other’s posts before publishing. And it’s just way more fun. Not that I’d need to tell you guys that!

We’ve gotten lots of positive responses and I learned a lot from the others and from discussions with our readers. That said, I have to admit that I’m a terrible blogger; I take ages to write anything. At the moment it’s so bad that I can’t really take any credit for the blog at all, it’s the other three who keep it going. Luckily they’re not just funny and smart but also some of the kindest people I know and bear with me…


CogTales: Do you think more early career researchers should blog?

Anne: It depends! For the individual (blogger) it’s a great opportunity to structure your thoughts, practice writing, and of course make a name for yourself and build your network. But I personally have a relatively high bar for what I think should be published (cf. taking ages to write a post…). People have limited time and there are so many blogs out there, so if you want impact, I think you need to find a niche and/or aim for (very) high quality. And that can then add a lot of self-inflicted pressure you otherwise wouldn’t have.

But if your main goal is to practice writing and your blog is intended as more of a public diary or writing tool, these (potential) drawbacks don’t apply of course. And I can see a lot of value in using it that way! Blog posts are usually a lot less formal than scientific papers, and that can be a great help when you struggle to put your thoughts on a blank page.


CogTales: Two years ago we talked about the skepticism social media assessments of published papers face. Has it become more accepted to blog about a paper? Is blogging and tweeting now more of a typical aspect of publishing and communicating science or do we still have a long way to go?

Anne: That is a really interesting question. On the one hand I think that publicly criticising published papers has become a lot more common in the last two years. Take the “pizza papers” affair, the avalanche of faulty papers by Brian Wansink that was uncovered by Tim van der Zee, Nick Brown, Jordan Anaya, and James Heathers: This investigation has been a constant source of news in my corner of the Twittersphere for the last couple of years, and helped popularise error detection and “forensic statistics” e.g. using tools like statcheck, GRIM/GRIMMER, and SPRITE. “We found inconsistencies in a paper” is a sentence I find a lot less surprising (or alarming) now than I did a few years ago.

On the other hand, criticising individual authors still seems to make people very uncomfortable — so uncomfortable that it reliably triggers pushback. I still get the strong sense that some people are only fine with criticism if it is directed against a behaviour or a group of people that is ill-defined enough to prevent individual group members from being identified. I have no numbers on this, but this “effect” feels exactly as strong to me as it did two years ago, and I find that really interesting.

Regarding the second part of your question, blogging and tweeting about research feels like a very “established” way of communicating to me, and I regularly see blog posts cited in published articles. But I’m never quite sure how much that reflects an actual change in the field and how much is just a result of the bubble I’m in — it’s absolutely possible that less Twitter-focussed colleagues would disagree with me on this.


CogTales: So it seems like we’ve already come a long way, but there’s still a lot to do. Tell us – what’s the most pressing issues we need to resolve in the way we do Science in the years to come?

Anne: At the moment I’m very concerned about standards for the new research practices that have been promoted in the last years, for example preregistration and data sharing. The concept “preregistration” is currently very poorly defined — it can be anything from a few lines of text stating a vague hypothesis to an extremely detailed, watertight research protocol including the analysis code. If the label “preregistered” applies to anything between these extremes, it doesn’t tell us anything and will inevitably lose its value in the long run (a recent study by Veldkamp et al. shows that even relatively strictly-reviewed preregistrations leave a lot of room for researcher degrees of freedom). And it leads to hard feelings: You now see cases popping up where a new finding from a “preregistered” study is published that some sceptics don’t find believable and proceed to criticise the preregistration as too vague. That then leads to surprised/frustrated responses from the authors, who feel unfairly treated because they already “went the extra mile” and still get flak.

If we want to get as many researchers as possible into this boat and also make sure that the new steps they are required to take really make their research more reproducible, the instructions have to be clear and fool-proof — without suffocating research lines that don’t fit neatly into the bog-standard hypothetico-deductive mould of experimental psychology research.

Similar issues apply to data sharing: Making your data publicly accessible has become a lot more common, but currently the used file formats, repositories, and data documentation are a huge mess, which is a problem for automated meta-analyses and long-term usability. Another aspect of data sharing that worries me is what happens when existing datasets do get reused: How can we make sure that secondary analyses don’t turn into the same overfitting spree that we just started to get a grip on for primary analyses? A group of people led by Sara Weston have done great work to develop a framework for preregistering analyses of pre-existing data, and I agree with them that it is possible. But what if individual researchers are less concerned with making use of such safeguards?

I see a risk of “collective overfishing” of existing datasets in the long run and wonder if we should replace open data with lightly-gated data: Imagine a central data-sharing platform where you have to request access to any dataset you want to retrieve. Authors could decide how strongly they want to restrict access to their data, e.g. fully private or access only for selected individuals/institutions. In the most “open” setting, anyone’s request for access would be approved immediately, but this instance of granted access would always be tracked and time-stamped. This kind of regulation could take a lot of strain off the the current system, which relies (too) heavily on interpersonal trust for my liking (nullius in verba!). As added benefits, a central platform would make it easier to establish standardised data structures and meta-data, and could even include built-in tools to facilitate and regulate cross-validation (e.g. split requested datasets into training data and holdout and restrict access to the holdout for a certain time period).

Long story short: I think we’ve seen many great new ideas and solutions to our problems sprouting in the last years, and now we’re entering a phase of pruning and channelling those ideas into effective, actionable, sustainable tracks — in other words: new standards. Developing standards that achieve their goals with minimal negative side effects is an extremely hard, but extremely important task.


Getting through rough patches where you don’t have motivation for anything.

Franziska Hartung, PhD

This is part of (academic) life. The session was short, so I won’t bother to summarize. Just watch the video:

Some important points I had discussed in a previous session also apply here:

Time and energy management: My take on work-life-balance

View original post

How to use Twitter for networking in academia

Within the span of two months, I’ve been asked to give essentially the same talk three times. The topic: how to network on Twitter (and other social media). How did this happen? Well, first a group of Parisian post docs organized a day-long workshop and apparently my tweeting is good enough to warrant inviting me back to my former home. Because I was invited, I took some care to prepare, and I think I did a decent job – decent enough, at least, to get some audience members to tweet about it and putting into practice what I just told them. Continue reading How to use Twitter for networking in academia

Building a network of women and nonbinary cognitive modelers

Here’s a (maybe not so well-kept) secret: I’ve got a PhD in modeling! No, not the posing kind, I constructed computational models of babies’ minds and behavior to better understand their early language acquisition. I learned a lot about cognition, babies, and data in that time. Next to that and two programming languages (Python and R) I also learned a bit about the modeling world. A key insight came to me after repeatedly trying to network with senior men and that being taken … the very wrong way. I must admit, I don’t know how much not being taken seriously as a modeler by some (no, not all) fellow modelers contributed to the fact that I took a step away from this field and am now an infant and a meta-science researcher most of the time. I am often thinking about what I’d recommend fellow women aspiring to a modeling career. So, at last, here’s the insight: build a support network of women modelers.* For those who watched a recent instalment of Academic Crisis Line, this might not be terribly earth shattering, but you have to realize that this is something that holds for your corner of science. I met a node in this support network soon thereafter, Olivia Guest, with whom I could talk forever about all those “fun” encounters. At some point, the idea to make a list of all fantastic, but probably vastly underappreciated women and nonbinary folks in modeling emerged, as she writes in her blog. There was some back and forth, questions about time investment, criteria, subcategories, so we effectively never got started, but such lists are super useful. For example, I suggested replacement speakers when asked to give a talk recently, and this list would have made my life much easier. So I am glad that Olivia turned to Twitter and simply asked others to make a list. The resulting thread is a goldmine. Continue reading Building a network of women and nonbinary cognitive modelers

How to deal with gender bias in academia?

Franziska Hartung, PhD

My honest first answer before this session was ‘I don’t know’. Similar as Atsuko put it in this live session, I am not living in my home country for many years and it is hard to tell what is about me being a women and what is a cultural thing I don’t understand. But there is this almost painful awareness of having to justify my existence all the time. This constant feeling of having to proof that I deserve to be here.

The most important thing I learned in this session was that this will never stop. And accepting that this is something which will always be part of my professional life also brings some kind of relief with it. I am not alone and I am just as much part of the solution as everybody else.

I am very grateful for the insightful and genuine discussion with my colleagues…

View original post 118 more words

How to review a manuscript for a journal

Franziska Hartung, PhD

Good reviews are supportive, constructive, thoughtful and fair. They identify both strengths and weaknesses alike and offer concrete suggestions for improvement. Good reviewers acknowledge their own biases and knowledge limitations and justify their conclusions.

Bad reviews are superficial, petty, and arrogant. Bad reviewers are very opinionated but typically don’t justify their biases. Their reports focus on weaknesses only but don’t offer solutions or other form of helpful feedback.

In today’s session, I walked you through the review process and told you how I write review reports:

Here you can find a template for the review report.

Additional ressources:

https://authorservices.wiley.com/Reviewers/journal-reviewers/how-to-perform-a-peer-review/step-by-step-guide-to-reviewing-a-manuscript.html offers a detailed step by step guide.

https://editorresources.taylorandfrancisgroup.com/reviewers-guidelines-and-best-practice/ offer additional advice and concrete examples of how to express criticism diplomatically.

http://www.sciencemag.org/careers/2016/09/how-review-paper features a lot of personal strategies and experiences which are often different from what I do.

Where I stole the summary from (almost word by word): https://facultystaff.richmond.edu/~rterry/NECTFL/How_to_Review_a_Journal_Article_NECTFL.pdf

View original post

When authorship sails away – Stories of the intricacies of academic accreditation

Papers are the currency in academia, they seem to determine our whole career. So, naturally, we try to publish as much as we can, while at the same time trying to produce good science. But sometimes authorship can become tricky, with hard decisions and disappointment. We share author-hard-ship stories here that cover a range of experiences, from being undeservedly excluded over the impression of getting too much credit to our own case that we consider ambiguous to this day.* All stories illustrate one key advice: Talk about authorship as early as possible in a project. This includes defining who is responsible for what, and discussing who is the lead of this project.**
Continue reading When authorship sails away – Stories of the intricacies of academic accreditation

Looking back on 2017

Dear CogTales reader, this post is about and made possible by you! You made this year the best yet in this little blog’s history. In today’s post, we want to take a moment and review which five posts you were most interested in. But first, we want to thank you. We’re hoping that you continue to come back and maybe even tell a friend or two about us. You can even contribute, if you have a story you would like to share, either with your name or anonymously, just get in touch.

So now let’s take a look back at the year as it is ending and review the top five posts according to our visitor statistics in 2017. Continue reading Looking back on 2017

Shared office space – a delicate ecosystem

Just a few months ago, I moved into the very first office that has only my name on it. During my whole scientific career, I shared offices of various sizes with between 1 and 7+ people. My office history ranges from the windowless undergrad thesis internship room where 6 students working on related projects and shared science and cookies over the room with a view and 2 colleagues as a PhD to the office with a server and between 0 and 3 others I occupied as post doc. During this time, I experienced many ways of sharing space: From the uncomplicated folks that tolerate your occasional cursing at the monitor to the weirdly expansion obsessed colleague who insisted that I would have more room if I just moved closer to the wall. Now, I’d like to think I know a bit about how to navigate shared office spaces, and I want to impart some bits of my wisdom and open them for discussion. Don’t hesitate to share your office mate horror stories in the comments and/or add useful tips how to improve office life!

Continue reading Shared office space – a delicate ecosystem

Scientists need more Europe, not less

This month, we are celebrating the anniversary of Germany’s reunification. This event is very important to me, as it reminds me every year, together with the anniversary of the fall of the Berlin wall in November, how lucky I am to be able to live the international scientist nomad life. Although this lifestyle with short term contracts has a lot of downsides, it is also a unique opportunity to work with various amazing people and follow your curiosity and ambition where it takes you. Continue reading Scientists need more Europe, not less