Issues with research readings
How big are effect sizes in psychology (~r= .21):
- One Hundred Years of Social Psychology Quantitatively Described (Review of General Psychology, 2003)
Seminal papers on the topic:
- SCIENCE AND ETHICS IN CONDUCTING, ANALYZING,AND REPORTING PSYCHOLOGICAL RESEARCH By Robert Rosenthal (PsycSci, 1994)
- Why Most Published Research Findings Are False (Ioannidis, 2005, PLOSOne)
- Do studies of statistical power have an effect on the power of studies? (Sedlmeier & Gigerenzer, 1989, Psychological bulletin).
- Pitfalls in Human Research Ten Pivotal Points (Book, 1972)
Public media:
- You've been poisoned. The cure is published, but you have time to read only one paper. You choose which based on: (posted on Twitter, summarizes the dilemma)
- Ten Famous Psychology Findings That It’s Been Difficult To Replicate (BPS digest, 2016)
- The Inevitable Evolution of Bad Science (The Atlantic, 2016)
- The problem with p-values (Aeon, 2016)
- In cancer science, many "discoveries" don't hold up (Reuters, 2012)
- Daryl Bem Proved ESP Is Real Which means science is broken (Slate, MAY 17, 2017)
- Psychology in crisis as seminal studies are unable to be replicated (Sydney morning herald, 2017)
- The Replication Crisis in Science (The Wire, 2017)
- Why Psychologists’ Food Fight Matters (Slate, 2018)
Great summaries:
- False-Positive Citations (Perspectives, 2017)
Controversial papers that highlight problems
General readings
- Life after P-Hacking Simmons, Nelson, & Simonsohn @SPSP
- The famous debate - Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect (Bem 2011, JPSP) and the response - Why psychologists must change the way they analyze their data: The case of psi
- How to Cook up Your Own Social Priming Article (Rolf Zwaan's blog post)
- Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention (Trends in Cognitive Psychology, March 2014)
- An investigation of the false discovery rate and the misinterpretation of p-values (Royal Society open science)
- Questionable research practives revisited (SPPS, 2015)
-
- Reproducibility: A tragedy of errors (Nature, 2016)
- Can you trust research? (in Hebrew, Haaretz, 2016)
- Are You Planning a 10-Study Article? You May Want to Read This First (replicability index, 2016)
- Many surveys, about one in five, may contain fraudulent data (Science, 2016)
- Encourage Playing with Data and Discourage Questionable Reporting Practices (Psychometrika, 2016)
- The Economics of Reproducibility in Preclinical Research (PLOS Biology, 2015)
- The Crisis of Confidence in Research Findings in Psychology: Is Lack of Replication the Real Problem? Or is it Something Else? (Archives of Scientific Psychology, 2016)
- Editorial: Evidence on Questionable Research Practices: The Good, the Bad, and the Ugly (Journal of Business and Psychology, 2016)
- Why Science Is Not Necessarily Self-Correcting (Ioannidis, PPS, 2016)
- The natural selection of bad science (Smaldino & McElreath, Royal society open, 2016) - reviewed in So, Maybe Academics Are Unconsciously Set Up for Clickbait Research (NYMag, 2016)
- Why most of psychology is statistically unfalsifiable (Morey & Lakens, submitted)
- Are Most Published Results in Psychology False? An Empirical Study (Replication index, 2017)
- Status of our Science - Study 2 about the failed OSC replications (pre-print, 2017)
- Four Bad Habits of Modern Psychologists (BS, 2018)
- Prestigious Science Journals Struggle to Reach Even Average Reliability (Brembs, Front. Hum. Neurosci., 2018)
Economics:
- IS REPLICABILITY IN ECONOMICS BETTER THAN IN PSYCHOLOGY? (Brian's ideas)
- Evaluating replicability of laboratory experiments in economics (Science, 2016)
- Reproducibility of research: Issues and proposed remedies (PNAS, 2018) SPECIAL ISSUE
Other fields:
- Is there a credibility crisis in strategic management research? Evidence on the reproducibility of study findings (Strategic Organization, 2017)
- Calculating and Reporting Effect Sizes to Facilitate Cumulative Science: A Practical Primer for t-tests and ANOVAs (Frontiers in Psychology, 2013)
- Increasing statistical power in psychological research without increasing sample size (blog post from the Open Science collaboration, Sean Mackinnon)
- Revised standards for statistical evidence (PNAS, 2013)
- Bosco, F. A., Aguinis, H., Singh, K., Field, J. G., & Pierce, C. A. (2014, October 13) Correlational Effect Size Benchmarks. Journal of Applied Psychology. Advance online publication.
- Using Nonnaive Participants Can Reduce Effect Sizes (PsycSci, 2015)
- I Fooled Millions Into Thinking Chocolate Helps Weight Loss. Here's How. (IO9 ; May, 2015)
- Three Guys Talking About Scales (Pysc your mind, 2015) - yes/no versus Likert
- Give p a chance: significance testing is misunderstood (the conversation, 2015)
- Confidence intervals? More like confusion intervals (psychometric society, 2015)
- Statistically controlling for confounding constructs is harder than you think Jacob WestfallTal Yarkoni
- How to Make Psychology Studies More Reliable (The Atlantic, 2016)
- NO-WAY INTERACTIONS (Data Colada)
- Zhu, L. L., Aquino, K., & Vadera, A. K. (2016). What Makes Professors Appear Credible: The Effect of Demographic Characteristics and Ideological Beliefs. (JAP)
- Researchers’ Intuitions About Power in Psychological Research (PsycSci, 2016)
- Researchers’ Intuitions About Power in Psychological Research (PsycSci, 2016)
- Measurement error and the replication crisis (Science, 2017)
- On the reproducibility of psychological science (Journal of the American Statistical Association, 2016) - it's actually more like 90% didn't replicate
psychologists don't know stats
- Four Bad Habits of Modern Psychologists (Behavioral Sciences 2017)
Control variables
- A Critical Review and Best-Practice Recommendations for Control Variable Usage (Personnel Psychology · January 2016)
- Statistical control in correlational studies: 10 essential recommendations for organizational researchers (Becker et al., 2016, JOM)
Peer review
- Peer review: a flawed process at the heart of science and journals (J R Soc Med , 2016)
- A Bayesian analysis of peer reviewing (concludes peer review no better than chance, Significance, 2016)
Dealing with mistakes
- Fallibility in science: Responding to errors in the work of oneself and others (preprint, Peerj, 2017)
Data availability
Summary presentations
Tools/demos
- Learn to p-hack like the pros! (a great presentation)
- The Heuristic Value of p in Inductive Statistical Inference (Frontiers, 2017)
- Syllabus design for research methods courses (SIPS, on the OSF)
Solutions
How to:
- Center for Open Science ambassador program on the OSF (lots of resources and presentations here)
- Remedies to the reproducibility crisis - article collection
- Maximizing the Reproducibility of Your Research (OSF, 2017)
- Guidelines for Science:Evidence and Checklists (working paper, look for the final version when it's out)
Initiatives:
- A manifesto for reproducible science (Nature Human Behavior, 2017)
- Verify Original Results through Reanalysis before Replicating (reply to BBS, 2018)
Solutions / platforms:
Articles:
- A pre-registration primer (OSF presentation)
- How open science helps researchers succeed (eLife, 2016)
- Predicting Experimental Results: Who Knows What? (Della and Pope, in review, 2016)
- The Value of Direct Replication (PPS, 2014)
- Too True to be Bad: When Sets of Studies with Significant and Non-Significant Findings Are Probably True (Lakens & Etz, 2017)
- Our path to better science in less time using open data science tools (nature ecology & evolution, 2017)
- Implications of the Credibility Revolution for Productivity, Creativity, and Progress (Simine Vazire, PPS, 2018)
Sharing data/code:
Pre-registration:
- Seven Selfish Reasons for Preregistration (Observer, 2016)
Video lectures:
- Improving the Reproducibility of Our Research Practices (APS, 2016, on Youtube)
Meta analyses:
Crowdsourcing/many-labs:
Teaching/education:
Questionable research practices
Replications
- Making Replication Mainstream (BBS, 2918)
- Comment on “Estimating the reproducibility of psychological science” (Science, 2016) (covered in Slate)
- P-Curve: A Key to the File-Drawer JEP-G , example for implementation - “Power Posing: P-Curving the Evidence” (PsycSCi, 2017) (data in OSF)
- Interactive Timeline: “Replication in Psychology: A History Perspective” (Advances in the history of psychology)
- The Replication Paradox: Combining Studies can Decrease Accuracy of Effect Size Estimates (Review of General Psychology,2015)
- How to make replication the norm (Nature, 2018)
Satire:
Problems with P values
- P Values: Misunderstood and Misused (PLOSOne, 2016)
- p=.20, what now? Adventures of the Good Ship DataPoint (Rolf Zwaan, 2015)
- On the hazards of significance testing. Part 1: the screening problem
- On the hazards of significance testing. Part 2: the false discovery rate, or how not to make a fool of yourself with P values
Problems with peeking
Statistical analyses
- mediation - Unwarranted inferences from statistical mediation tests – An analysis of articles published in 2015 (JESP, 2018)
- Common misconceptions about data analysis and statistics (Naunyn-Schmiedeberg's Archives of Pharmacology, 2014), nice flowchart of p-hacking, generally great intro paper.
Issues with management research
- Ruminations on How We Became a Mystery House and How We Might Get Out (Barley, JOM, 2016)
- I-O Psychology’s Lack of Research Integrity (SIOP, 2016)
- HARKing's Threat to Organizational Research: Evidence From Primary and Meta-Analytic Sources (Personnel Psychology, 2015)
randomized controlled trials
- Understanding and misunderstanding randomized controlled trials (Social Science & Medicine, 2017)
Problems with analysis and interpretation
Concerns and negative reactions to the change in psychological science
- Why scientific criticism sometimes needs to hurt (Daniel Lakens)
- Meet The New Bad People (Medium, 2017)
Other
- Principles of the Self-Journal of Science: bringing ethics and freedom to scientific publishing (on an interesting platform).
Press releases issues: