What happens when you stand up to the big wigs? A follow-up interview with Anne Scheel

Two years ago, Team CogTales (Sho and Christina) interviewed Anne Scheel. We were impressed how she stood up to ask a tough question at Germany’s largest Psychology conference (the DGPs Kongress) after a keynote presentation. Two years later, Christina and Anne actually met up at the next installment of the very same conference, and a lot has changed in the short time span. So it seems like a perfect moment to catch up and take stock.

CogTales: Hi Anne, thank you for talking to us again. And of course we are curious to hear whether you think your actions had any consequences for you and / or your career.

Anne: Unfortunately we didn’t set this up as an RCT so I have no idea how control-group Anne is doing, but I’ll happily overinterpret the correlational data and say that it’s had quite an impact on my career — in a snowball-/Matthew-effect kind of way: First I gained a lot of Twitter followers and the attention of some more influential people, who then gave me a platform (like you!), which led to more people knowing me, and so on. Now I regularly get invited to give workshops and talks, I’ve been a guest on two podcasts… if you had told me that two years ago, I would have recommended you to go easy on the mushrooms. I still think it’s crazy.

In fairness, some of this has (very likely) been the result of starting a blog with three friends, as we’ll talk about below — the causal relations are hard to disentangle!

One of the few things that can be traced back to my DGPs moment with certainty is that several people have bought me a drink for it (you guys are amazing).

CogTales: You presented and co-chaired a symposium at this year’s DGPs conference as well, congratulations. How did your session on “Ramping up Rigor: Current Practices, Problems, Potential Solutions” go? Do you think the field has changed in the past years?

Anne: We had a bit of a thankless slot on the first morning of the conference but still got a nicely filled room and some really interesting discussions. I found this year’s DGPs fascinating — so many open science/reproducibility sessions! There were several different ones every day, some by people I hadn’t known before, including a large symposium and keynote on theory and a policy-focussed panel with representatives from several psychological societies.

I think it’s safe to say that the movement has grown in width and depth. My experience is probably not representative and I’m likely to overestimate adoption rates, but it feels like e.g. preregistration has become so common that in most psychology departments you might meet someone who has done it. At the same time, the discussions about improved research practices have become more specialised and cover more ground; see for example the increasing whispers about the “theory crisis”. To me that’s progress. We started discovering our problems at the far and most concrete end — the numbers that are churned out by our research process — and since we lifted the rug, we’ve been tracing the problem back to its roots. It’s a long way, and the questions that arise become more and more philosophical, which unfortunately doesn’t make them easier to answer. But in any case, it feels like the need for reform is turning into an accepted narrative. I find that very exciting.

 

CogTales: We also noted that you are now part of a blog team, of course we hope the positive experience with CogTales helped in that decision. How do you experience blogging?

Anne: Funnily enough, The 100% CI is also a consequence of DGPs 2016. After Susan Fiske’s keynote, Malte Elson, Julia Rohrer, and Ruben Arslan were among the people who stuck around to talk to her, and after the conference the four of us stayed in touch and eventually decided to start a blog together.

It’s been a great experience! I can highly recommend blogging in a team — it’s much easier to provide regular(ish) content and reach a wider audience, and putting your thoughts out into the world feels less scary because you can review each other’s posts before publishing. And it’s just way more fun. Not that I’d need to tell you guys that!

We’ve gotten lots of positive responses and I learned a lot from the others and from discussions with our readers. That said, I have to admit that I’m a terrible blogger; I take ages to write anything. At the moment it’s so bad that I can’t really take any credit for the blog at all, it’s the other three who keep it going. Luckily they’re not just funny and smart but also some of the kindest people I know and bear with me…

 

CogTales: Do you think more early career researchers should blog?

Anne: It depends! For the individual (blogger) it’s a great opportunity to structure your thoughts, practice writing, and of course make a name for yourself and build your network. But I personally have a relatively high bar for what I think should be published (cf. taking ages to write a post…). People have limited time and there are so many blogs out there, so if you want impact, I think you need to find a niche and/or aim for (very) high quality. And that can then add a lot of self-inflicted pressure you otherwise wouldn’t have.

But if your main goal is to practice writing and your blog is intended as more of a public diary or writing tool, these (potential) drawbacks don’t apply of course. And I can see a lot of value in using it that way! Blog posts are usually a lot less formal than scientific papers, and that can be a great help when you struggle to put your thoughts on a blank page.

 

CogTales: Two years ago we talked about the skepticism social media assessments of published papers face. Has it become more accepted to blog about a paper? Is blogging and tweeting now more of a typical aspect of publishing and communicating science or do we still have a long way to go?

Anne: That is a really interesting question. On the one hand I think that publicly criticising published papers has become a lot more common in the last two years. Take the “pizza papers” affair, the avalanche of faulty papers by Brian Wansink that was uncovered by Tim van der Zee, Nick Brown, Jordan Anaya, and James Heathers: This investigation has been a constant source of news in my corner of the Twittersphere for the last couple of years, and helped popularise error detection and “forensic statistics” e.g. using tools like statcheck, GRIM/GRIMMER, and SPRITE. “We found inconsistencies in a paper” is a sentence I find a lot less surprising (or alarming) now than I did a few years ago.

On the other hand, criticising individual authors still seems to make people very uncomfortable — so uncomfortable that it reliably triggers pushback. I still get the strong sense that some people are only fine with criticism if it is directed against a behaviour or a group of people that is ill-defined enough to prevent individual group members from being identified. I have no numbers on this, but this “effect” feels exactly as strong to me as it did two years ago, and I find that really interesting.

Regarding the second part of your question, blogging and tweeting about research feels like a very “established” way of communicating to me, and I regularly see blog posts cited in published articles. But I’m never quite sure how much that reflects an actual change in the field and how much is just a result of the bubble I’m in — it’s absolutely possible that less Twitter-focussed colleagues would disagree with me on this.

 

CogTales: So it seems like we’ve already come a long way, but there’s still a lot to do. Tell us – what’s the most pressing issues we need to resolve in the way we do Science in the years to come?

Anne: At the moment I’m very concerned about standards for the new research practices that have been promoted in the last years, for example preregistration and data sharing. The concept “preregistration” is currently very poorly defined — it can be anything from a few lines of text stating a vague hypothesis to an extremely detailed, watertight research protocol including the analysis code. If the label “preregistered” applies to anything between these extremes, it doesn’t tell us anything and will inevitably lose its value in the long run (a recent study by Veldkamp et al. shows that even relatively strictly-reviewed preregistrations leave a lot of room for researcher degrees of freedom). And it leads to hard feelings: You now see cases popping up where a new finding from a “preregistered” study is published that some sceptics don’t find believable and proceed to criticise the preregistration as too vague. That then leads to surprised/frustrated responses from the authors, who feel unfairly treated because they already “went the extra mile” and still get flak.

If we want to get as many researchers as possible into this boat and also make sure that the new steps they are required to take really make their research more reproducible, the instructions have to be clear and fool-proof — without suffocating research lines that don’t fit neatly into the bog-standard hypothetico-deductive mould of experimental psychology research.

Similar issues apply to data sharing: Making your data publicly accessible has become a lot more common, but currently the used file formats, repositories, and data documentation are a huge mess, which is a problem for automated meta-analyses and long-term usability. Another aspect of data sharing that worries me is what happens when existing datasets do get reused: How can we make sure that secondary analyses don’t turn into the same overfitting spree that we just started to get a grip on for primary analyses? A group of people led by Sara Weston have done great work to develop a framework for preregistering analyses of pre-existing data, and I agree with them that it is possible. But what if individual researchers are less concerned with making use of such safeguards?

I see a risk of “collective overfishing” of existing datasets in the long run and wonder if we should replace open data with lightly-gated data: Imagine a central data-sharing platform where you have to request access to any dataset you want to retrieve. Authors could decide how strongly they want to restrict access to their data, e.g. fully private or access only for selected individuals/institutions. In the most “open” setting, anyone’s request for access would be approved immediately, but this instance of granted access would always be tracked and time-stamped. This kind of regulation could take a lot of strain off the the current system, which relies (too) heavily on interpersonal trust for my liking (nullius in verba!). As added benefits, a central platform would make it easier to establish standardised data structures and meta-data, and could even include built-in tools to facilitate and regulate cross-validation (e.g. split requested datasets into training data and holdout and restrict access to the holdout for a certain time period).

Long story short: I think we’ve seen many great new ideas and solutions to our problems sprouting in the last years, and now we’re entering a phase of pruning and channelling those ideas into effective, actionable, sustainable tracks — in other words: new standards. Developing standards that achieve their goals with minimal negative side effects is an extremely hard, but extremely important task.

Advertisements

Screen, Baby  – Let’s Look at Evidence, not Trends

Joint post by Nawal Abboub, PhD, & Sho Tsuji, PhD

French version here

Who hasn’t heard that “Children under two years of age should absolutely not be exposed to screen media, no matter what!” – maybe accompanied by the reasoning that “Screens will hinder the development of children’s intelligence.”

Why does the topic of the effect of screens on young children, especially with regard to their brain development, evoke so much controversy and fear? And should we actually think of all types of screen media as equal? What can scientific research teach us? Continue reading Screen, Baby  – Let’s Look at Evidence, not Trends

Data Visualization – the Why and How

Sho: I recently gave a talk on data visualization at the International Conference on Infant Studies (you can find my slides, along with the other wonderful talks on power, preregistration, and ethical data peeking here on the OSF). I also played the German Cats and Dogs Scientist in the barbarplots campaign on better data visualization (on the same topic, Article 1 and Article 2 on why bar and line plots hide differences in underlying distributions). In fact, being part of the barbarplots team was my entry point into thinking more about the importance of visualizing your data in a maximally informative and honest way. Informative means finding a good balance between simplifying/summarizing and showing the underlying data structure. Honest means not (accidentally) hiding important aspects of your data. Mahiko is my office mate and – this is something I discovered while preparing the above talk – an enthusiastic data visualizer. That’s why I asked him to put together our (mostly, his) favorite data viz resources.

Continue reading Data Visualization – the Why and How

Two Simple Recipes for Women in Science Events

Naomi Havron is a postdoc at the Laboratoire de Sciences Cognitives et Psycholinguistique at Ecole Normale Supérieure.

She was recently interviewed for the departmental newsletter on her experiences as a woman and mother in science, and here’s a statement we found particularly on point:

Being a woman in science became difficult once I was a mom. Before, I believed that any inequality could be surmounted by working my ass off. But once I was a mom, I had to leave work early for my children. And I was judged differently from men. If it was me that was leaving early, that was judged as non-professional and proof I wasn’t invested enough in research. If it was a male colleague that was leaving for his children, he was complimented as being a devoted parent. (paraphrased and translated, see original French version here).

Naomi has been involved in organizing two amazing women-in-science events at our department recently, and we’re very excited she’s sharing her experiences and materials here with us.

Continue reading Two Simple Recipes for Women in Science Events

How to land a Postdoc position.

If you decide to do a postdoc, do everything you can to do it right from the get-go! Watch what Sho (Cogtales) and Franziska (Ph_Dial) have to say ❤

Franziska Hartung, PhD

Doing a postdoc can be a fantastic experience. In the last session of ACL, I talked with Sho Tsuji from Ecole Normale Supérieure de Paris who – just as me – is a very happy postdoc.

The most important thing is to find  lab in which you can grow and have a PI that will be a great mentor not only for now, but for the rest of your career. Do a careful screening of whom you want to work with and try to get to know them and people who worked with them (or still do!). Be open-minded and use your network to find out about labs, job search specifics or grant opportunities in individual countries, and personal recommendations.

You will get the most out of your postdoc if you know what you want to get out of it. Make this guide you to what kind of project or lab…

View original post 325 more words

Networking Part 2: Initiating conversation in person

In our last post, Christina talked about academic networking on social media, specifically Twitter. There’s a reason that was her post, not mine: Even though I’ve been following most of her advice and this has improved my Twitter experience, I still feel awkward and out of place on Twitter, and I can’t get myself to create an account under my own name (instead, I’m tweeting as @cogtalestweet).

So today, I’m talking about my cup of tea: Live, in person networking. Specifically, the focus is on how to initiate conversation.

Continue reading Networking Part 2: Initiating conversation in person

How to use Twitter for networking in academia

Within the span of two months, I’ve been asked to give essentially the same talk three times. The topic: how to network on Twitter (and other social media). How did this happen? Well, first a group of Parisian post docs organized a day-long workshop and apparently my tweeting is good enough to warrant inviting me back to my former home. Because I was invited, I took some care to prepare, and I think I did a decent job – decent enough, at least, to get some audience members to tweet about it and putting into practice what I just told them. Continue reading How to use Twitter for networking in academia

Building a network of women and nonbinary cognitive modelers

Here’s a (maybe not so well-kept) secret: I’ve got a PhD in modeling! No, not the posing kind, I constructed computational models of babies’ minds and behavior to better understand their early language acquisition. I learned a lot about cognition, babies, and data in that time. Next to that and two programming languages (Python and R) I also learned a bit about the modeling world. A key insight came to me after repeatedly trying to network with senior men and that being taken … the very wrong way. I must admit, I don’t know how much not being taken seriously as a modeler by some (no, not all) fellow modelers contributed to the fact that I took a step away from this field and am now an infant and a meta-science researcher most of the time. I am often thinking about what I’d recommend fellow women aspiring to a modeling career. So, at last, here’s the insight: build a support network of women modelers.* For those who watched a recent instalment of Academic Crisis Line, this might not be terribly earth shattering, but you have to realize that this is something that holds for your corner of science. I met a node in this support network soon thereafter, Olivia Guest, with whom I could talk forever about all those “fun” encounters. At some point, the idea to make a list of all fantastic, but probably vastly underappreciated women and nonbinary folks in modeling emerged, as she writes in her blog. There was some back and forth, questions about time investment, criteria, subcategories, so we effectively never got started, but such lists are super useful. For example, I suggested replacement speakers when asked to give a talk recently, and this list would have made my life much easier. So I am glad that Olivia turned to Twitter and simply asked others to make a list. The resulting thread is a goldmine. Continue reading Building a network of women and nonbinary cognitive modelers

How to deal with gender bias in academia?

Franziska Hartung, PhD

My honest first answer before this session was ‘I don’t know’. Similar as Atsuko put it in this live session, I am not living in my home country for many years and it is hard to tell what is about me being a women and what is a cultural thing I don’t understand. But there is this almost painful awareness of having to justify my existence all the time. This constant feeling of having to proof that I deserve to be here.

The most important thing I learned in this session was that this will never stop. And accepting that this is something which will always be part of my professional life also brings some kind of relief with it. I am not alone and I am just as much part of the solution as everybody else.

I am very grateful for the insightful and genuine discussion with my colleagues…

View original post 118 more words