Caroline Poppa presented fascinating insights into her research on "The scope of publication bias in two large German probabilistic panels", in which research proposals and research outcomes are compared in terms of authorship, hypotheses, research questions, and much more. #MetaRep2024

Colleagues from Linnaeus University had 3 related posters. It was great to get a chance to speak to Thomas Nordström on "Reproducibility, Open Science, FAIR and Risk Of Bias in Educational Systematic Reviews", see the group's recommendations for "A Beginners Guide to Open and Reproducible Systematic Reviews in Psychology" based on some best-practices they found in their work, and find out about "The Value of Bibliometrics in Meta-Scientific Literature Reviews". #MetaRep2024

Look-back to the poster session #MetaRep2024. It was great to hear about some exciting #MetaScience projects in #education.

I enjoyed speaking to Lucija Batinovic about "Community Augmented Meta-Analysis of Evidence in Learning and Didactics" and reading up on Naska Goagoses' research into the "Frequency of Pre-Registration for Systematic Reviews in Educational Technology". #EdTech #OpenScience

I'm afraid I missed most of Herbert Bless' talk on "Theoretical aspects of comparing results across studies" as I ended up discussing #OpenScience with a group of similarly opinionated French-speaking colleagues for too long in the break, but from the discussion, it sounded like this paper on "The crucial role of linearity when comparing effects across studies" is worth checking out: https://www.nature.com/articles/s44159-023-00220-w #MetaRep2024 #stats

@lakens I decided to sit at the front for the keynote because most slides so far at #MetaRep2024 have had small-size fonts that I couldn’t read from the back rows. But nobody told me that @lakens’s slides look like this…

The poster session #MetaRep2024 was super interesting. I'll be posting some personal highlights here, but with some delay because the keynote by @lakens is about to start and I want to find out " Which research is worth doing well?"

UPDATE: Daniel's slides can be found here: https://osf.io/ejqa2/.

In line with the previous talk, Anne Gast's team identified a causal effect of recruitment (commercial provider > university recruitment)!

To conclude this #MetaRep2024 symposium, Marie-Ann Sengewald ends with some recommendations on how to realise conceptual replications that control for systematic bias and enable causal interpretations.

I was not familiar with the Causal Replication Framework (Steiner et al. 2019): https://econtent.hogrefe.com/doi/full/10.1027/2151-2604/a000385

Next up in this #MetaRep2024 symposium on Causal Interpretations of Effect Heterogeneity in #Replication Research, Anne Gast on "Understanding Differences in the Evaluative Conditioning Effect across Online and Lab-Settings". Anne stresses the need to consider unintended differences across replications. Examples include participants of different ages typically using different devices to complete online surveys and violations of randomisation between studies due to missing data or non-compliance.

Mathias Twardawski is talking about "Conceptual Replications to Understand Context Sensitivity". In designing a study to examine context sensitivity, Mathias warns that "variation is easy, consistency is hard."

Very interesting conclusion that recruitment method (students and interested people vs. paid participants on Prolific) was the only significant driver of heterogeneity in the study outcomes (though I was personally not surprised). #MetaScience #MetaRep2024 #AcademicChatter

Decided to start the day #MetaRep2024 with Taimoor Khan's talk on "Reproducibility of AI/ML methods in computational social science". Taimoor is presenting the GESIS Methods Hub project, which sounds great. Work is in progress and external launch is planed for mid-2025: https://www.gesis.org/en/services/processing-and-analyzing-data/analyzing-digital-behavioral-data/gesis-methods-hub

On my way to the #MetaRep2024 conference venue this morning, I found a sourdough bakery in Munich that will set you back 1,30€ for a single white breadroll, but it was totally worth it because I am now in an excellent mood and ready for a jam-packed day of insights into #MetaScience, #OpenScience, #reproducibility, #replication and more. Which talks should I attend today? https://meta-rep.de/conference24/programme/

Last presentation of the day for me: "Data Simulation for Everyone: An Interactive Tool for Data Simulation using DAGs" by Finn Luebber. The shiny app is still under development, but feedback is welcome. #MetaRep2024 #MetaScience #Shiny

Have you used/are you using data collected by students in your research? Marlene Altenmüller reports on “Problematic student data collection behaviors” with a survey of students and supervisors on the (suspected) prevalence of various questionable research practices (QRPs) and research misconduct. Another paper for the reading list: https://doi.org/10.32872/spb.9411
#AcademicChatter #MetaRep2024 #ResearchIntegrity

High on cake and earl grey after the break #MetaRep2024 and enjoying Julia Schnepf’s talk on “Fundamental misalignments of current research in #psychology”. Julia shared her experience of obtaining null-results in her #PhD and getting rejection after rejection while attempting to publish her pre-registered studied. Julia finally managed to graduate, but was clearly a victim of the “null-results penalty”. Another exciting paper for my #MetaScience reading list: https://www.sciencedirect.com/science/article/pii/S0732118X24000382.

Last talk before the coffee break: “Dimensions of Redoing - mapping and standardizing Replication Efforts” by Maximilian Frank. We discussed how the terminology of redoing activities (#reproduction, #replication) varies across different disciplines and regions. #MetaRep2024

Rachel Heyard is now presenting the results of a scoping review of #reproducibility metrics: they found 50 (!) different metrics (but one comes from a paper that was then retracted). Should be available as a #preprint in the next few weeks! #MetaScience #MetaRep2024

EDIT: I had dinner with Rachel and it was 50 application papers (of which one has since been retracted) not metrics. Best to wait for the #preprint to come out to find out how many metrics exactly!

“All theorems are true. All models are wrong. And all data are inaccurate. What are we to do?” - Engaging opening quote for Maximilian Mandi’s talk on “Researcher degrees of freedom and minP-adjustment”. #MetaScience #MetaRep2024

There are three parallel strands at #MetaRep2024 and I’m struggling to choose among too many fascinating-sounding abstracts. I’ve just heard “A multiple testing approach for improving the replicability of reported findings” by Juliane Wilcke which focuses on how p-values are reported. I’m not sure I’m convinced (also still digesting lunch!) but looking forward to reading their opinion paper.

Lunch at #MetaRep2024 was delicious and the break long enough to enjoy a brisk walk in the English Garden with new #MetaScience friends! Now, excited to listen to Ambra Perugini speaking about “The importance of reporting and interpreting critical effect sizes” and introducing a new #RStats package: https://github.com/psicostat/criticalESvalue. They are concluding with the TBT motto: “Think Before Testing”. #Stats #statistics