Good list of readings

How big are effect sizes in psychology (~r= .21):

• One Hundred Years of Social Psychology Quantitatively Described (Review of General Psychology, 2003)

Seminal papers on the topic:

• SCIENCE AND ETHICS IN CONDUCTING, ANALYZING,AND REPORTING PSYCHOLOGICAL RESEARCH By Robert Rosenthal (PsycSci, 1994)
• Why Most Published Research Findings Are False (Ioannidis, 2005, PLOSOne)
• Do studies of statistical power have an effect on the power of studies? (Sedlmeier & Gigerenzer, 1989, Psychological bulletin).
• Pitfalls in Human Research Ten Pivotal Points (Book, 1972)

Public media:

• A great 'syllabus' on problems - Everything is fucked: The syllabus
• You've been poisoned. The cure is published, but you have time to read only one paper. You choose which based on: (posted on Twitter, summarizes the dilemma)
• Ten Famous Psychology Findings That It’s Been Difficult To Replicate (BPS digest, 2016)
• We Gave Four Good Pollsters the Same Raw Data. They Had Four Different Results. (NYT, 2016)
• The Inevitable Evolution of Bad Science (The Atlantic, 2016)
• The problem with p-values (Aeon, 2016)
• In cancer science, many "discoveries" don't hold up (Reuters, 2012)
• Daryl Bem Proved ESP Is Real Which means science is broken (Slate, MAY 17, 2017)
• Psychology in crisis as seminal studies are unable to be replicated (Sydney morning herald, 2017)
• The Replication Crisis in Science (The Wire, 2017)
• Why Psychologists’ Food Fight Matters (Slate, 2018)

Great summaries:

• Replicability Review of 2016
• Academic Research in the 21st Century: Maintaining Scientific Integrity in a Climate of Perverse Incentives and Hypercompetition
• False-Positive Citations (Perspectives, 2017)
• The State of Social and Personality Science: Rotten to the Core, Not so Bad, Getting Better, or Getting Worse (JPSP, 2017)

• Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect. Bem - Journal of personality and social psychology, 2011
  • Why the Journal of Personality and Social Psychology Should Retract Article DOI: 10.1037/a0021524 “Feeling the Future: Experimental evidence for anomalous retroactive influences on cognition and affect” by Daryl J. Bem (R index, 2018)
• Feeling the future: A meta-analysis of 90 experiments on the anomalous anticipation of random future events (F1000Res, 2015)
  • My email correspondence with Daryl J. Bem about the data for his 2011 article “Feeling the future”

• The Rules of the Game Called Psychological Science (PPS, 2012)
• Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations (EJE, 2016)
• The 7 biggest problems facing science, according to 270 scientists (Vox, 2016)
• Psychology’s brilliant, beautiful, scientific messiness.
• Scientific Utopia: I. Opening Scientific Communication
• Life after P-Hacking Simmons, Nelson, & Simonsohn @SPSP
• High Impact = High Statistical Standards? Not Necessarily So (PLOSOne)
• A Top 10 List of Psychology’s Big Questions, and the Answers
• Deep impact: unintended consequences of journal rank
• The Practice of Reproducible Research Case Studies and Lessons from the Data-Intensive Sciences (eBook)
• Reproducibility in Psychological Science: When Do Psychological Phenomena Exist? (Frontiers, 2017)
• Estimating the evidential value of significant results in psychological science (PLOSOne, 2017)

• The famous debate - Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect (Bem 2011, JPSP) and the response - Why psychologists must change the way they analyze their data: The case of psi
• Just Post It The Lesson From Two Cases of Fabricated Data Detected by Statistics Alone (Psycsci, 2013)
• How to Cook up Your Own Social Priming Article (Rolf Zwaan's blog post)
• Psychology Disclosure project
• The Chrysalis Effect How Ugly Initial Results Metamorphosize Into Beautiful Articles (JOM, 2014)
• Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention (Trends in Cognitive Psychology, March 2014)
• People analyze dataset differently reaching different conclusions.
• Science Isn’t Broken It’s just a hell of a lot harder than we give it credit for (538 science)
• An investigation of the false discovery rate and the misinterpretation of p-values (Royal Society open science)
• The Prevalence of Statistical Reporting Errors in Psychology (1985-2013)
• Meta-analyses are no substitute for registered replications: a skeptical perspective on religious priming (Frontiers)
• 2 economists tried to replicate a bunch of published economic studies, and the results were disturbing (Business Insider, 2015)
• Questionable research practives revisited (SPPS, 2015)
• Research funding goes to rich clubs / Gender contributes to personal research funding success in The Netherlands (PNAS, 2015)
• Underreporting in Psychology Experiments Evidence From a Study Registry (SPPS, 2016)
• Benefits of Open and High-Powered Research Outweigh Costs (JPSP, 2016)
  • and this tool - True Scientific Discoveries: What Research Approach is Most Cost-Effective?
• Statistical & Methodological Consulting - center for open science
• Reproducibility: A tragedy of errors (Nature, 2016)
• Can you trust research? (in Hebrew, Haaretz, 2016)
• Are You Planning a 10-Study Article? You May Want to Read This First (replicability index, 2016)
• A Bayesian Perspective on the Reproducibility Project: Psychology (PLOSone, 2016)
• Many surveys, about one in five, may contain fraudulent data (Science, 2016)
• Encourage Playing with Data and Discourage Questionable Reporting Practices (Psychometrika, 2016)
• Why A Journalist Scammed The Media Into Spreading Bad Chocolate Science (NPR) and Can chocolate make you smarter? (And thinner? And healthier?) (Guardian)
• The Economics of Reproducibility in Preclinical Research (PLOS Biology, 2015)
• Distributions of p-values smaller than .05 in psychology: what is going on? (Peerj, 2016)
• Cancer Research Is Broken There’s a replication crisis in biomedicine—and no one even knows how deep it runs (Slate, 2016)
• The Crisis of Confidence in Research Findings in Psychology: Is Lack of Replication the Real Problem? Or is it Something Else? (Archives of Scientific Psychology, 2016)
• Editorial: Evidence on Questionable Research Practices: The Good, the Bad, and the Ugly (Journal of Business and Psychology, 2016)
• Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature (preprint)
• Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias (PLOSOne, 2008)
• How replicable is psychology? A comparison of four methods of estimating replicability on the basis of test statistics in original studies (Brunner and Schimmack, 2016)
• Why Science Is Not Necessarily Self-Correcting (Ioannidis, PPS, 2016)
• The natural selection of bad science (Smaldino & McElreath, Royal society open, 2016) - reviewed in So, Maybe Academics Are Unconsciously Set Up for Clickbait Research (NYMag, 2016)
• Why most of psychology is statistically unfalsifiable (Morey & Lakens, submitted)
• Current Incentives for Scientists Lead to Underpowered Studies with Erroneous Conclusions (PLOS Biology, 2016)
• Are Most Published Results in Psychology False? An Empirical Study (Replication index, 2017)
• Preprint of Too good to be false: Nonsignificant results revisited (preprint)
• Replication studies offer much more than technical details They demonstrate the practice of science at its best. (Editorial, Nature, 2017)
• Status of our Science - Study 2 about the failed OSC replications (pre-print, 2017)
• The N-Pact Factor: Evaluating the Quality of Empirical Journals with Respect to Sample Size and Statistical Power (PLOSOne, 2014) (Data)
• Statistical Reporting Errors and Collaboration on Statistical Analyses in Psychological Science (PLOSOne, 2014)
• Four Bad Habits of Modern Psychologists (BS, 2018)
• Prestigious Science Journals Struggle to Reach Even Average Reliability (Brembs, Front. Hum. Neurosci., 2018)

Economics:

• IS REPLICABILITY IN ECONOMICS BETTER THAN IN PSYCHOLOGY? (Brian's ideas)
• Evaluating replicability of laboratory experiments in economics (Science, 2016)
• THE POWER OF BIAS IN ECONOMICS RESEARCH
• To Replicate or Not To Replicate? Exploring Reproducibility in Economics through the Lens of a Model and a Pilot Study (The Economic Journal, 2017)
• The Economics Data Revolution Has Growing Pains Too many studies use small sample sizes that give false positives (Bloomberg, 2017)
• Reproducibility of research: Issues and proposed remedies (PNAS, 2018) SPECIAL ISSUE

Other fields:

• Questionable science and reproducibility in electrical brain stimulation research (PLOSone, 2017)
• Is there a credibility crisis in strategic management research? Evidence on the reproducibility of study findings (Strategic Organization, 2017)
• Replication Failures Highlight Biases in Ecology and Evolution Science

• Calculating and Reporting Effect Sizes to Facilitate Cumulative Science: A Practical Primer for t-tests and ANOVAs (Frontiers in Psychology, 2013)
• Increasing statistical power in psychological research without increasing sample size (blog post from the Open Science collaboration, Sean Mackinnon)
• Revised standards for statistical evidence (PNAS, 2013)
• Bosco, F. A., Aguinis, H., Singh, K., Field, J. G., & Pierce, C. A. (2014, October 13) Correlational Effect Size Benchmarks. Journal of Applied Psychology. Advance online publication.
• Using Nonnaive Participants Can Reduce Effect Sizes (PsycSci, 2015)
• I Fooled Millions Into Thinking Chocolate Helps Weight Loss. Here's How. (IO9 ; May, 2015)
• Using prediction markets to estimate the reproducibility of scientific research (PNAS, 2015)
• Three Guys Talking About Scales (Pysc your mind, 2015) - yes/no versus Likert
• Give p a chance: significance testing is misunderstood (the conversation, 2015)
• Confidence intervals? More like confusion intervals (psychometric society, 2015)
• What journalists get wrong about social science, according to 20 scientists (Vox, 2016)
• Meta-analyses are no substitute for registered replications: a skeptical perspective on religious priming (Front. Psychol., 2015)
• Statistically controlling for confounding constructs is harder than you think Jacob WestfallTal Yarkoni
• How to Make Psychology Studies More Reliable (The Atlantic, 2016)
• NO-WAY INTERACTIONS (Data Colada)
• Zhu, L. L., Aquino, K., & Vadera, A. K. (2016). What Makes Professors Appear Credible: The Effect of Demographic Characteristics and Ideological Beliefs. (JAP)
• Researchers’ Intuitions About Power in Psychological Research (PsycSci, 2016)
• Does expertise matter in replication? An examination of the reproducibility project: Psychology (JESP, 2016)
• Researchers’ Intuitions About Power in Psychological Research (PsycSci, 2016)
• Measurement error and the replication crisis (Science, 2017)
• On the reproducibility of psychological science (Journal of the American Statistical Association, 2016) - it's actually more like 90% didn't replicate
• It’s Time to Broaden the Replicability Conversation: Thoughts for and From Clinical Psychological Science (PPS, 2017)
• ONLY 18 OF 120 POLITICAL SCIENCE JOURNALS HAVE A REPLICATION POLICY

• Four Bad Habits of Modern Psychologists (Behavioral Sciences 2017)
• Quantifying Support for the Null Hypothesis in Psychology: An Empirical Investigation (preprint, 2018)
• The British amateur who debunked the mathematics of happiness

• A Critical Review and Best-Practice Recommendations for Control Variable Usage (Personnel Psychology · January 2016)
• Methodological Urban Legends: The Misuse of Statistical Control Variables (Spector & Brannick, 2010, ORM)
• Statistical control in correlational studies: 10 essential recommendations for organizational researchers (Becker et al., 2016, JOM)

• Peer review: a flawed process at the heart of science and journals (J R Soc Med , 2016)
• How anonymous peer review fails to do its job and damages science (OSC, 2014)
• A Bayesian analysis of peer reviewing (concludes peer review no better than chance, Significance, 2016)

• Fallibility in science: Responding to errors in the work of oneself and others (preprint, Peerj, 2017)

• Data availability, reusability, and analytic reproducibility: Evaluating the impact of a mandatory open data policy at the journal Cognition (preprint, 2018)
• Populating the Data Ark: An attempt to retrieve, preserve, and liberate data from the most highly-cited psychology and psychiatry articles (PLOSOne, 2018)

• tyarkoni OHBM 2016 open science hack room talk
• The p value and the base rate fallacy
• Keynote EuroSciPy 2017: How to Fix a Scientific Culture

• Introducing the p-hacker app: Train your expert p-hacking skills
• 688,112 Statistical Results: Content Mining Psychology Articles for Statistical Test Results (dataset preprint)
• Learn to p-hack like the pros! (a great presentation)
• When does a significant p-value indicate a true effect? Understanding the Positive Predictive Value (PPV) of a p-value
• Simpson's Paradox
• The Heuristic Value of p in Inductive Statistical Inference (Frontiers, 2017)
• Syllabus design for research methods courses (SIPS, on the OSF)

How to:

• Center for Open Science ambassador program on the OSF (lots of resources and presentations here)
• Remedies to the reproducibility crisis - article collection
• Maximizing the Reproducibility of Your Research (OSF, 2017)
• Guidelines for Science:Evidence and Checklists (working paper, look for the final version when it's out)
• Five ways to fix statistics
• A practical guide for transparency in psychological science

Initiatives:

• A manifesto for reproducible science (Nature Human Behavior, 2017)
• A Manifesto for Open Science in Giftedness Research (preprint)
• Commitment to Research Transparency and Open Science
• Check yourself again - Michael Inzlicht
• Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking (Frontiers, 2016)
• Constraints on Generality (COG): A Proposed Addition to All Empirical Papers (preprint)
• The GRIM Test A Simple Technique Detects Numerous Anomalies in the Reporting of Results in Psychology (SPPS, 2017) - example1 / example2
• Collaborative Replications and Education Project (CREP)
• Data reporting at The Leadership Quarterly
• Nineteen Things Editors of Experimental Psychology Journals can do to Increase the Replicability of the Research they Publish
• Analytic Review as a Solution to the Misreporting of Statistical Results in Psychological Science (PPS, 2014)
• Verify Original Results through Reanalysis before Replicating (reply to BBS, 2018)
• Psychological Science needs a standard practice of reporting the reliability of cognitive behavioural measurements (preprint, 2018)

Solutions / platforms:

• Curate Science
• Dataverse
• Bullied Into Bad Science

Articles:

• The pipeline project: Pre-publication independent replications of a single laboratory's research pipeline (JESP, 2016)
• A pre-registration primer (OSF presentation)
• The ‘Call for Participants’ platform connects researchers with participants so as to improve the efficiency and accuracy of research trials.
• How open science helps researchers succeed (eLife, 2016)
• Predicting Experimental Results: Who Knows What? (Della and Pope, in review, 2016)
• The Value of Direct Replication (PPS, 2014)
• Benefits of Open and High-Powered Research Outweigh Costs (JPSP, 2017)
• Too True to be Bad: When Sets of Studies with Significant and Non-Significant Findings Are Probably True (Lakens & Etz, 2017)
  • something similar - The ironic effect of significant results on the credibility of multiple-study articles (article / blog post)
• Reproducible practices are the future for early career researchers (Bishopblog)
• Our path to better science in less time using open data science tools (nature ecology & evolution, 2017)
• Transparent science: A more credible, reproducible, and publishable way to do science (preprint, 2018)
• Implications of the Credibility Revolution for Productivity, Creativity, and Progress (Simine Vazire, PPS, 2018)
• Beyond Bar and Line Graphs: Time for a New Data Presentation Paradigm (PLOSOne, 2018)
• Open Science is Liberating and Can Foster Creativity (PPS, 2018)

Sharing data/code:

• Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results (PLOSOne, 2011)

Pre-registration:

• Seven Selfish Reasons for Preregistration (Observer, 2016)
• Improving on pre-registration

Video lectures:

• Improving the Reproducibility of Our Research Practices (APS, 2016, on Youtube)
• Professor John Antonakis discusses the 5 diseases of academic publishing

Meta analyses:

• Continuously Cumulating Meta-Analysis and Replicability (PPS, 2014)

Crowdsourcing/many-labs:

• Crowdsourcing Science PhD Course slides / syllabus by

Martin Schweinsberg

Teaching/education:

• Collaborative Replications and Education Project (CREP)
• The Replication Crisis in Psychology NOVA
• Open Science Teaching and Training Resources

• Making Replication Mainstream (BBS, 2918)
• Comment on “Estimating the reproducibility of psychological science” (Science, 2016) (covered in Slate)
• P-Curve: A Key to the File-Drawer JEP-G , example for implementation - “Power Posing: P-Curving the Evidence” (PsycSCi, 2017) (data in OSF)
• Replication studies: Bad copy
• Interactive Timeline: “Replication in Psychology: A History Perspective” (Advances in the history of psychology)
• 1,500 scientists lift the lid on reproducibility Survey sheds light on the ‘crisis’ rocking research (Nature, 2016)
• Post publication peer review 2017 - Will Gervais on his own Science article
• The Replication Paradox: Combining Studies can Decrease Accuracy of Effect Size Estimates (Review of General Psychology,2015)
• How to make replication the norm (Nature, 2018)

Satire:

• Replication Mirror

• P Values: Misunderstood and Misused (PLOSOne, 2016)
• p=.20, what now? Adventures of the Good Ship DataPoint (Rolf Zwaan, 2015)
• On the hazards of significance testing. Part 1: the screening problem
• On the hazards of significance testing. Part 2: the false discovery rate, or how not to make a fool of yourself with P values
• Spoiled Science How a seemingly innocent blog post led to serious doubts about Cornell’s famous food laboratory (Chronicles, 2017)
• I’m a non-methodologist, does it matter if my definition is slightly wrong? PsychBrief

• Realistic data peeking isn’t as bad as you* thought – it’s worse
• Data peeking without p-hacking
• Excuses for Data Peeking

• Subgroup analyses
• mediation - Unwarranted inferences from statistical mediation tests – An analysis of articles published in 2015 (JESP, 2018)
• Common misconceptions about data analysis and statistics (Naunyn-Schmiedeberg's Archives of Pharmacology, 2014), nice flowchart of p-hacking, generally great intro paper.
• Journal Data Sharing Policies and Statistical Reporting Inconsistencies in Psychology

• Organizational sciences’ obsession with “that’s interesting!”, Consequences and an alternative
• Ruminations on How We Became a Mystery House and How We Might Get Out (Barley, JOM, 2016)
• I-O Psychology’s Lack of Research Integrity (SIOP, 2016)
• HARKing's Threat to Organizational Research: Evidence From Primary and Meta-Analytic Sources (Personnel Psychology, 2015)
• HARKing: How Badly Can Cherry-Picking and Question Trolling Produce Bias in Published Results? (JBP, 2017)

• Understanding and misunderstanding randomized controlled trials (Social Science & Medicine, 2017)

• Health Services as Credence Goods: A Field Experiment

• Susan Fiske - APS Observer, in pressMob Rule or Wisdom of Crowds?
• Why scientific criticism sometimes needs to hurt (Daniel Lakens)
• Meet The New Bad People (Medium, 2017)

• Principles of the Self-Journal of Science: bringing ethics and freedom to scientific publishing (on an interesting platform).

Press releases issues:

• Exaggerations and Caveats in Press Releases and Health-Related Science News (PLOSone, 2016)

• Calling Bullshit In the Age of Big Data course
• Improving your statistical inferences
• Reproducible Research
• Statistical Inference

• http://steamtraen.blogspot.com/2017/03/strange-patterns-in-some-results-from.html
• http://www.chronicle.com/article/Spoiled-Science/239529
• http://steamtraen.blogspot.com/2017/03/some-instances-of-apparent-duplicate.html
• http://www.timvanderzee.com/the-wansink-dossier-an-overview/
• http://steamtraen.blogspot.com/2017/03/more-apparent-duplication-from-food-and.html

• Some Psychological Effects Replicate Even Under Potentially Adverse Conditions (preprint)

• Psychology’s Replication Crisis and the Grant Culture: Righting the Ship (PPS, 2017)

• Cultural Change Over Time: Why Replicability Should Not Be the Gold Standard in Psychological Science (PPS, 2017)