Show pageOld revisionsBacklinksBack to top This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong. ====== Issues with research readings ====== [[https://docs.google.com/document/d/14lBD0aZDPij2Z6AOpAharOAtmt6ZBI0EuF3_tu8m66I/mobilebasic|Good list of readings]] How big are effect sizes in psychology (~r= .21): * [[http://jenni.uchicago.edu/Spencer_Conference/Representative%20Papers/Richard%20et%20al,%202003.pdf|One Hundred Years of Social Psychology Quantitatively Described]] (Review of General Psychology, 2003) Seminal papers on the topic: * [[http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.508.8161&rep=rep1&type=pdf|SCIENCE AND ETHICS IN CONDUCTING, ANALYZING,AND REPORTING PSYCHOLOGICAL RESEARCH]] By Robert Rosenthal (PsycSci, 1994) * [[http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124|Why Most Published Research Findings Are False]] (Ioannidis, 2005, PLOSOne) * [[http://datacolada.org/wp-content/uploads/2013/10/3416-Sedlmeier-Gigerenzer-Psych-Bull-1989-Do-studies-of-statistical-power-have-an-effect-on-the-power-of-studies.pdf|Do studies of statistical power have an effect on the power of studies?]] (Sedlmeier & Gigerenzer, 1989, Psychological bulletin). * [[http://www.sciencedirect.com/science/book/9780080209340|Pitfalls in Human Research Ten Pivotal Points]] (Book, 1972) Public media: * [[https://hardsci.wordpress.com/2016/08/11/everything-is-fucked-the-syllabus/|A great 'syllabus' on problems - Everything is fucked: The syllabus]] * [[https://twitter.com/thattai/status/775667881161785344|You've been poisoned. The cure is published, but you have time to read only one paper. You choose which based on:]] (posted on Twitter, summarizes the dilemma) * [[https://digest.bps.org.uk/2016/09/16/ten-famous-psychology-findings-that-its-been-difficult-to-replicate/|Ten Famous Psychology Findings That It’s Been Difficult To Replicate]] (BPS digest, 2016) * [[http://www.nytimes.com/interactive/2016/09/20/upshot/the-error-the-polling-world-rarely-talks-about.html?_r=0|We Gave Four Good Pollsters the Same Raw Data. They Had Four Different Results.]] (NYT, 2016) * [[http://www.theatlantic.com/science/archive/2016/09/the-inevitable-evolution-of-bad-science/500609/|The Inevitable Evolution of Bad Science]] (The Atlantic, 2016) * [[https://aeon.co/essays/it-s-time-for-science-to-abandon-the-term-statistically-significant|The problem with p-values]] (Aeon, 2016) * [[http://www.reuters.com/article/us-science-cancer-idUSBRE82R12P20120328|In cancer science, many "discoveries" don't hold up]] (Reuters, 2012) * [[http://redux.slate.com/cover-stories/2017/05/daryl-bem-proved-esp-is-real-showed-science-is-broken.html|Daryl Bem Proved ESP Is Real Which means science is broken]] (Slate, MAY 17, 2017) * [[http://www.smh.com.au/national/health/psychology-in-crisis-as-seminal-studies-are-unable-to-be-replicated-20170530-gwg9xh.html|Psychology in crisis as seminal studies are unable to be replicated]] (Sydney morning herald, 2017) * [[https://thewire.in/208014/replication-crisis-science/|The Replication Crisis in Science]] (The Wire, 2017) * [[http://www.slate.com/articles/health_and_science/science/2014/07/replication_controversy_in_psychology_bullying_file_drawer_effect_blog_posts.html|Why Psychologists’ Food Fight Matters]] (Slate, 2018) Great summaries: * [[https://replicationindex.wordpress.com/2016/12/31/replicability-review-of-2016/|Replicability Review of 2016]] * [[http://online.liebertpub.com/doi/pdf/10.1089/ees.2016.0223|Academic Research in the 21st Century: Maintaining Scientific Integrity in a Climate of Perverse Incentives and Hypercompetition]] * [[https://poseidon01.ssrn.com/delivery.php?ID=088024100123020118115082018077028065121004001038027088066090116107020109065119069098119033023106033000111097106002126000090091106034037051088094088011012065015075033061024026029118099116080002125023090086105071108030104084013026082110094024097029024&EXT=pdf|False-Positive Citations]] (Perspectives, 2017) * [[https://www.academia.edu/32720873/The_State_of_Social_and_Personality_Science_Rotten_to_the_Core_Not_so_Bad_Getting_Better_or_Getting_Worse|The State of Social and Personality Science: Rotten to the Core, Not so Bad, Getting Better, or Getting Worse]] (JPSP, 2017) ===== Controversial papers that highlight problems ===== * [[http://caps.ucsf.edu/wordpress/wp-content/uploads/2011/02/bem2011.pdf|Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect. Bem - Journal of personality and social psychology, 2011]] * W[[https://replicationindex.wordpress.com/2018/01/05/why-the-journal-of-personality-and-social-psychology-should-retract-article-doi-10-1037-a0021524-feeling-the-future-experimental-evidence-for-anomalous-retroactive-influences-on-cognition-a/|hy the Journal of Personality and Social Psychology Should Retract Article DOI: 10.1037/a0021524 “Feeling the Future: Experimental evidence for anomalous retroactive influences on cognition and affect” by Daryl J. Bem]] (R index, 2018) * [[http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4706048/|Feeling the future: A meta-analysis of 90 experiments on the anomalous anticipation of random future events]] (F1000Res, 2015) * [[https://replicationindex.wordpress.com/2018/01/20/my-email-correspondence-with-daryl-j-bem-about-the-data-for-his-2011-article-feeling-the-future/|My email correspondence with Daryl J. Bem about the data for his 2011 article “Feeling the future”]] ===== General readings ===== * [[http://pps.sagepub.com/content/7/6/543.full|The Rules of the Game Called Psychological Science]] (PPS, 2012) * [[http://link.springer.com/article/10.1007/s10654-016-0149-3|Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations]] (EJE, 2016) * [[http://www.vox.com/2016/7/14/12016710/science-challeges-research-funding-peer-review-process|The 7 biggest problems facing science, according to 270 scientists]] (Vox, 2016) * [[http://blogs.scientificamerican.com/psysociety/2013/08/13/psychology-is-a-science/|Psychology’s brilliant, beautiful, scientific messiness.]] * [[http://www.tandfonline.com/doi/full/10.1080/1047840X.2012.692215|Scientific Utopia: I. Opening Scientific Communication]] * [[http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2205186|Life after P-Hacking]] Simmons, Nelson, & Simonsohn @SPSP * [[http://www.plosone.org/article/info:doi/10.1371/journal.pone.0056180|High Impact = High Statistical Standards? Not Necessarily So]] (PLOSOne) * [[http://www.psychologytoday.com/blog/fulfillment-any-age/201304/top-10-list-psychology-s-big-questions-and-the-answers|A Top 10 List of Psychology’s Big Questions, and the Answers]] * [[http://www.frontiersin.org/Human_Neuroscience/10.3389/fnhum.2013.00291/full#B55|Deep impact: unintended consequences of journal rank]] * [[https://www.practicereproducibleresearch.org/|The Practice of Reproducible Research Case Studies and Lessons from the Data-Intensive Sciences]] (eBook) * [[http://journal.frontiersin.org/article/10.3389/fpsyg.2017.00879/full|Reproducibility in Psychological Science: When Do Psychological Phenomena Exist]]? (Frontiers, 2017) * [[http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0182651|Estimating the evidential value of significant results in psychological science]] (PLOSOne, 2017) * The famous debate - [[http://caps.ucsf.edu/wordpress/wp-content/uploads/2011/02/bem2011.pdf|Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect]] (Bem 2011, JPSP) and the response - [[http://commonsenseatheism.com/wp-content/uploads/2010/11/Wagenmakers-Why-Psychologists-Must-Change-the-Way-They-Analyze-Their-Data.pdf|Why psychologists must change the way they analyze their data: The case of psi]] * [[http://pss.sagepub.com/content/early/2013/08/23/0956797613480366.full|Just Post It The Lesson From Two Cases of Fabricated Data Detected by Statistics Alone]] (Psycsci, 2013) * [[http://rolfzwaan.blogspot.com/2013/09/how-to-cook-up-your-own-social-priming.html|How to Cook up Your Own Social Priming Article]] (Rolf Zwaan's blog post) * [[http://psychdisclosure.org/about.html|Psychology Disclosure project]] * [[http://jom.sagepub.com/content/early/2014/03/18/0149206314527133.abstract|The Chrysalis Effect How Ugly Initial Results Metamorphosize Into Beautiful Articles]] (JOM, 2014) * [[http://www.sciencedirect.com/science/article/pii/S1364661314000540|Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention]] (Trends in Cognitive Psychology, March 2014) * [[http://www.spspblog.org/recruiting-analysts-and-co-authors-for-a-crowdsourcing-project/|People analyze dataset differently reaching different conclusions]]. * [[http://fivethirtyeight.com/features/science-isnt-broken/|Science Isn’t Broken It’s just a hell of a lot harder than we give it credit for]] (538 science) * [[http://rsos.royalsocietypublishing.org/content/1/3/140216|An investigation of the false discovery rate and the misinterpretation of p-values]] (Royal Society open science) * [[https://mbnuijten.files.wordpress.com/2013/01/nuijtenetal_2015_reportingerrorspsychology_acceptedmanuscriptbrm.pdf|The Prevalence of Statistical Reporting Errors in Psychology (1985-2013)]] * [[http://journal.frontiersin.org/article/10.3389/fpsyg.2015.01365/abstract|Meta-analyses are no substitute for registered replications: a skeptical perspective on religious priming]] (Frontiers) * [[http://www.businessinsider.com/federal-reserve-paper-on-the-replicability-of-economic-studies-2015-10|2 economists tried to replicate a bunch of published economic studies, and the results were disturbing]] (Business Insider, 2015) * [[https://www.researchgate.net/profile/Norbert_Schwarz2/publication/282854178_Questionable_research_practices_revisited/links/561fdc3e08aed8dd19404099.pdf?origin=publication_list&ev=contentfeed_xdl&_iepl%5BviewId%5D=V0utfA09UDWr2u8xYatM81lL&_iepl%5BsingleItemViewId%5D=60KmpHVDkgqnEHUP0xfzGc1v&_iepl%5BactivityId%5D=631189794852864&_iepl%5BactivityType%5D=person_add_file_publication&_iepl%5BactivityTimestamp%5D=1444928574&_iepl%5BhomeFeedVariantCode%5D=d_EU&_iepl%5Bcontexts%5D%5B0%5D=homeFeed&_iepl%5BinteractionType%5D=publicationDownload|Questionable research practives revisited]] (SPPS, 2015) * [[http://www.pnas.org/content/early/2015/11/12/1520118112.short|Research funding goes to rich clubs]] / [[http://www.pnas.org/content/112/40/12349.abstract|Gender contributes to personal research funding success in The Netherlands]] (PNAS, 2015) * [[http://spp.sagepub.com/content/7/1/8.abstract?rss=1|Underreporting in Psychology Experiments Evidence From a Study Registry]] (SPPS, 2016) * [[http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2616384|Benefits of Open and High-Powered Research Outweigh Costs]] (JPSP, 2016) * and this tool - [[http://shinyapps.org/showapp.php?app=http%3A%2F%2F87.106.45.173%3A3838%2Ffelix%2FN-per-discovery&by=Etienne+LeBel&title=N+per+discovery&shorttitle=N+per+discovery#.VrEg9AHWnxQ.twitter|True Scientific Discoveries: What Research Approach is Most Cost-Effective?]] * [[https://cos.io/stats_consulting/|Statistical & Methodological Consulting - center for open science]] * [[http://www.nature.com/news/reproducibility-a-tragedy-of-errors-1.19264|Reproducibility: A tragedy of errors]] (Nature, 2016) * [[http://www.haaretz.co.il/magazine/.premium-1.2847131|Can you trust research?]] (in Hebrew, Haaretz, 2016) * [[https://replicationindex.wordpress.com/2016/02/10/are-you-planning-a-10-study-article-you-may-want-to-read-this-first/|Are You Planning a 10-Study Article? You May Want to Read This First]] (replicability index, 2016) * [[http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0149794|A Bayesian Perspective on the Reproducibility Project: Psychology]] (PLOSone, 2016) * [[http://www.sciencemag.org/news/2016/02/survey-says-many-surveys-about-one-five-may-contain-fraudulent-data|Many surveys, about one in five, may contain fraudulent data]] (Science, 2016) * [[http://link.springer.com/article/10.1007/s11336-015-9445-1|Encourage Playing with Data and Discourage Questionable Reporting Practices]] (Psychometrika, 2016) * [[http://www.npr.org/sections/thesalt/2015/05/28/410313446/why-a-journalist-scammed-the-media-into-spreading-bad-chocolate-science|Why A Journalist Scammed The Media Into Spreading Bad Chocolate Science]] (NPR) and [[http://www.theguardian.com/lifeandstyle/shortcuts/2016/mar/08/can-chocolate-make-you-smarter-and-thinner-and-healthier|Can chocolate make you smarter? (And thinner? And healthier?)]] (Guardian) * [[http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002165|The Economics of Reproducibility in Preclinical Research]] (PLOS Biology, 2015) * [[https://peerj.com/articles/1935/|Distributions of p-values smaller than .05 in psychology: what is going on?]] (Peerj, 2016) * [[http://www.slate.com/articles/health_and_science/future_tense/2016/04/biomedicine_facing_a_worse_replication_crisis_than_the_one_plaguing_psychology.html|Cancer Research Is Broken There’s a replication crisis in biomedicine—and no one even knows how deep it runs]] (Slate, 2016) * [[http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2773582|The Crisis of Confidence in Research Findings in Psychology: Is Lack of Replication the Real Problem? Or is it Something Else?]] (Archives of Scientific Psychology, 2016) * [[http://link.springer.com/article/10.1007/s10869-016-9456-7|Editorial: Evidence on Questionable Research Practices: The Good, the Bad, and the Ugly]] (Journal of Business and Psychology, 2016) * [[http://www.biorxiv.org/content/early/2016/08/25/071530.abstract|Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature]] (preprint) * [[http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0003081&utm_content=buffer8b935|Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias]] (PLOSOne, 2008) * [[http://www.utstat.utoronto.ca/~brunner/zcurve2016/HowReplicable.pdf|How replicable is psychology? A comparison of four methods of estimating replicability on the basis of test statistics in original studies]] (Brunner and Schimmack, 2016) * [[http://pps.sagepub.com/content/7/6/645.full.pdf">open|Why Science Is Not Necessarily Self-Correcting]] (Ioannidis, PPS, 2016) * [[http://rsos.royalsocietypublishing.org/content/3/9/160384|The natural selection of bad science]] (Smaldino & McElreath, Royal society open, 2016) - reviewed in [[http://nymag.com/scienceofus/2016/09/are-academics-unconsciously-set-up-for-clickbait-research.html?mid=twitter_scienceofus|So, Maybe Academics Are Unconsciously Set Up for Clickbait Research]] (NYMag, 2016) * [[https://medium.com/@richarddmorey/new-paper-why-most-of-psychology-is-statistically-unfalsifiable-4c3b6126365a#.dv1r6qwut|Why most of psychology is statistically unfalsifiable]] (Morey & Lakens, submitted) * [[http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.2000995|Current Incentives for Scientists Lead to Underpowered Studies with Erroneous Conclusions]] (PLOS Biology, 2016) * [[https://replicationindex.wordpress.com/2017/01/15/are-most-published-results-in-psychology-false-an-empirical-study/|Are Most Published Results in Psychology False? An Empirical Study]] (Replication index, 2017) * [[https://osf.io/rkumy/|Preprint of Too good to be false: Nonsignificant results revisited]] (preprint) * [[http://www.nature.com/news/replication-studies-offer-much-more-than-technical-details-1.21311|Replication studies offer much more than technical details They demonstrate the practice of science at its best.]] (Editorial, Nature, 2017) * [[https://osf.io/preprints/psyarxiv/qxaby|Status of our Science - Study 2]] about the failed OSC replications (pre-print, 2017) * [[http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0109019|The N-Pact Factor: Evaluating the Quality of Empirical Journals with Respect to Sample Size and Statistical Power]] (PLOSOne, 2014) ([[https://osf.io/7im3n/|Data]]) * [[http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0114876|Statistical Reporting Errors and Collaboration on Statistical Analyses in Psychological Science]] (PLOSOne, 2014) * [[http://www.mdpi.com/2076-328X/7/3/53/htm|Four Bad Habits of Modern Psychologists]] (BS, 2018) * [[https://www.frontiersin.org/articles/10.3389/fnhum.2018.00037/full|Prestigious Science Journals Struggle to Reach Even Average Reliability]] (Brembs, Front. Hum. Neurosci., 2018) Economics: * [[https://brainsidea.wordpress.com/2016/03/19/replicability-in-economics-psychology/?utm_content=bufferebc46&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer|IS REPLICABILITY IN ECONOMICS BETTER THAN IN PSYCHOLOGY?]] (Brian's ideas) * [[http://science.sciencemag.org/content/351/6280/1433|Evaluating replicability of laboratory experiments in economics]] (Science, 2016) * [[http://onlinelibrary.wiley.com/doi/10.1111/ecoj.12461/pdf|THE POWER OF BIAS IN ECONOMICS RESEARCH]] * [[http://onlinelibrary.wiley.com/doi/10.1111/ecoj.12527/abstract?campaign=woletoc|To Replicate or Not To Replicate? Exploring Reproducibility in Economics through the Lens of a Model and a Pilot Study]] (The Economic Journal, 2017) * [[https://www.bloomberg.com/view/articles/2017-11-27/the-economics-data-revolution-has-growing-pains|The Economics Data Revolution Has Growing Pains Too many studies use small sample sizes that give false positives]] (Bloomberg, 2017) * [[http://www.pnas.org/content/early/2018/03/08/1802324115|Reproducibility of research: Issues and proposed remedies]] (PNAS, 2018) SPECIAL ISSUE Other fields: * [[http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0175635|Questionable science and reproducibility in electrical brain stimulation research]] (PLOSone, 2017) * [[http://journals.sagepub.com.sci-hub.cc/doi/abs/10.1177/1476127017701076|Is there a credibility crisis in strategic management research? Evidence on the reproducibility of study findings]] (Strategic Organization, 2017) * [[https://www.the-scientist.com/features/replication-failures-highlight-biases-in-ecology-and-evolution-science-64475#.W2R8S_JdBdk.twitter|Replication Failures Highlight Biases in Ecology and Evolution Science]] * [[http://www.frontiersin.org/Cognition/10.3389/fpsyg.2013.00863/abstract|Calculating and Reporting Effect Sizes to Facilitate Cumulative Science: A Practical Primer for t-tests and ANOVAs]] (Frontiers in Psychology, 2013) * [[http://osc.centerforopenscience.org/2013/11/03/Increasing-statistical-power/|Increasing statistical power in psychological research without increasing sample size]] (blog post from the Open Science collaboration, Sean Mackinnon) * [[http://www.pnas.org/content/110/48/19313.long|Revised standards for statistical evidence]] (PNAS, 2013) * Bosco, F. A., Aguinis, H., Singh, K., Field, J. G., & Pierce, C. A. (2014, October 13) [[http://dx.doi.org/10.1037/a0038047|Correlational Effect Size Benchmarks. Journal of Applied Psychology. Advance online publication]]. * [[http://pss.sagepub.com/content/early/2015/06/10/0956797615585115.abstract|Using Nonnaive Participants Can Reduce Effect Sizes]] (PsycSci, 2015) * [[http://io9.com/i-fooled-millions-into-thinking-chocolate-helps-weight-1707251800|I Fooled Millions Into Thinking Chocolate Helps Weight Loss. Here's How.]] (IO9 ; May, 2015) * [[http://www.pnas.org/content/early/2015/11/04/1516179112.full.pdf?with-ds=yes|Using prediction markets to estimate the reproducibility of scientific research]] (PNAS, 2015) * [[http://psych-your-mind.blogspot.com/2015/11/three-guys-talking-about-scales.html|Three Guys Talking About Scales]] (Pysc your mind, 2015) - yes/no versus Likert * [[https://theconversation.com/give-p-a-chance-significance-testing-is-misunderstood-20207|Give p a chance: significance testing is misunderstood]] (the conversation, 2015) * [[http://www.psychonomic.org/featured-content-detail/confidence-intervals-more-like-confusion-intervals|Confidence intervals? More like confusion intervals]] (psychometric society, 2015) * [[http://www.vox.com/science-and-health/2016/1/22/10811320/journalists-social-science|What journalists get wrong about social science, according to 20 scientists]] (Vox, 2016) * [[http://journal.frontiersin.org/article/10.3389/fpsyg.2015.01365/abstract|Meta-analyses are no substitute for registered replications: a skeptical perspective on religious priming]] (Front. Psychol., 2015) * [[http://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0152719|Statistically controlling for confounding constructs is harder than you think]] Jacob WestfallTal Yarkoni * [[http://www.theatlantic.com/science/archive/2016/03/save-psychology-by-replicating-studies-before-theyre-published/475983/|How to Make Psychology Studies More Reliable]] (The Atlantic, 2016) * [[http://datacolada.org/17|NO-WAY INTERACTIONS]] (Data Colada) * Zhu, L. L., Aquino, K., & Vadera, A. K. (2016). [[https://www.researchgate.net/profile/Luke_Zhu/publication/297595423_What_Makes_Professors_Appear_Credible_The_Effect_of_Demographic_Characteristics_and_Ideological_Beliefs/links/56ec2c1708ae4b8b5e732f4e.pdf|What Makes Professors Appear Credible: The Effect of Demographic Characteristics and Ideological Beliefs]]. (JAP) * [[http://pss.sagepub.com/content/early/2016/06/28/0956797616647519.long|Researchers’ Intuitions About Power in Psychological Research]] (PsycSci, 2016) * [[http://www.sciencedirect.com/science/article/pii/S002210311630258X|Does expertise matter in replication? An examination of the reproducibility project: Psychology]] (JESP, 2016) * [[http://pss.sagepub.com/content/27/8/1069|Researchers’ Intuitions About Power in Psychological Research]] (PsycSci, 2016) * [[http://science.sciencemag.org/content/355/6325/584.full|Measurement error and the replication crisis]] (Science, 2017) * [[http://www.tandfonline.com/doi/full/10.1080/01621459.2016.1240079|On the reproducibility of psychological science]] (Journal of the American Statistical Association, 2016) - it's actually more like 90% didn't replicate * [[http://journals.sagepub.com/doi/abs/10.1177/1745691617690042|It’s Time to Broaden the Replicability Conversation: Thoughts for and From Clinical Psychological Science]] (PPS, 2017) * [[https://politicalsciencereplication.wordpress.com/2013/03/08/only-18-of-120-political-science-journals-have-a-replication-policy/|ONLY 18 OF 120 POLITICAL SCIENCE JOURNALS HAVE A REPLICATION POLICY]] ===== psychologists don't know stats ===== * [[http://www.mdpi.com/2076-328X/7/3/53/htm|Four Bad Habits of Modern Psychologists]] (Behavioral Sciences 2017) * [[https://psyarxiv.com/zqkyt/|Quantifying Support for the Null Hypothesis in Psychology: An Empirical Investigation]] (preprint, 2018) * [[https://www.theguardian.com/science/2014/jan/19/mathematics-of-happiness-debunked-nick-brown|The British amateur who debunked the mathematics of happiness]] ===== Control variables ===== * [[https://www.researchgate.net/publication/272091481_A_Critical_Review_and_Best-Practice_Recommendations_for_Control_Variable_Usage|A Critical Review and Best-Practice Recommendations for Control Variable Usage]] (Personnel Psychology · January 2016) * [[http://orm.sagepub.com/content/14/2/287|Methodological Urban Legends: The Misuse of Statistical Control Variables (Spector & Brannick, 2010, ORM)]] * [[http://onlinelibrary.wiley.com/doi/10.1002/job.2053/abstract|Statistical control in correlational studies: 10 essential recommendations for organizational researchers]] (Becker et al., 2016, JOM) ===== Peer review ===== * [[http://jrs.sagepub.com/content/99/4/178.short|Peer review: a flawed process at the heart of science and journals]] (J R Soc Med , 2016) * [[http://osc.centerforopenscience.org/2014/05/15/anonymous-peer-review/|How anonymous peer review fails to do its job and damages science]] (OSC, 2014) * [[http://onlinelibrary.wiley.com/doi/10.1111/j.1740-9713.2016.00881.x/abstract|A Bayesian analysis of peer reviewing]] (concludes peer review no better than chance, Significance, 2016) ===== Dealing with mistakes ===== * [[https://peerj.com/preprints/3486/|Fallibility in science: Responding to errors in the work of oneself and others]] (preprint, Peerj, 2017) ===== Data availability ===== * [[https://osf.io/preprints/bitss/39cfb/|Data availability, reusability, and analytic reproducibility: Evaluating the impact of a mandatory open data policy at the journal Cognition]] (preprint, 2018) * [[http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0201856|Populating the Data Ark: An attempt to retrieve, preserve, and liberate data from the most highly-cited psychology and psychiatry articles]] (PLOSOne, 2018) ====== Summary presentations ====== * [[https://github.com/tyarkoni/slides/blob/master/Yarkoni_OpenScience_HackRoom_OHBM2016.pdf|tyarkoni OHBM 2016 open science hack room talk]] * [[https://www.statisticsdonewrong.com/p-value.html|The p value and the base rate fallacy]] * [[https://osf.io/n4ckd/|Keynote EuroSciPy 2017: How to Fix a Scientific Culture]] ===== Tools/demos ===== * [[http://www.nicebread.de/introducing-p-hacker/|Introducing the p-hacker app: Train your expert p-hacking skills]] * [[http://www.preprints.org/manuscript/201608.0191/v1|688,112 Statistical Results: Content Mining Psychology Articles for Statistical Test Results]] (dataset preprint) * [[https://osf.io/u4jgz/|Learn to p-hack like the pros!]] (a great presentation) * [[http://shinyapps.org/apps/PPV/|When does a significant p-value indicate a true effect? Understanding the Positive Predictive Value (PPV) of a p-value]] * [[http://vudlab.com/simpsons/|Simpson's Paradox]] * [[http://journal.frontiersin.org/article/10.3389/fpsyg.2017.00908/full|The Heuristic Value of p in Inductive Statistical Inference]] (Frontiers, 2017) * [[https://osf.io/zbwr4/|Syllabus design for research methods courses]] (SIPS, on the OSF) ===== Solutions ===== How to: * [[https://osf.io/bs23x/|Center for Open Science ambassador program on the OSF]] (lots of resources and presentations here) * [[https://www.scienceopen.com/search#%7B%22order%22%3A0%2C%22context%22%3A%7B%22collection%22%3A%7B%22id%22%3A%22026426d2-7441-46b0-848c-75089ccb1577%22%2C%22kind%22%3A0%7D%2C%22kind%22%3A11%7D%2C%22kind%22%3A77%7D|Remedies to the reproducibility crisis]] - article collection * [[https://osf.io/preprints/psyarxiv/k9mn3/|Maximizing the Reproducibility of Your Research]] (OSF, 2017) * [[https://faculty.wharton.upenn.edu/wp-content/uploads/2016/12/Guidelines-for-science-389-Clean.pdf|Guidelines for Science:Evidence and Checklists]] (working paper, look for the final version when it's out) * [[https://www.nature.com/articles/d41586-017-07522-z|Five ways to fix statistics]] * [[https://psyarxiv.com/rtygm|A practical guide for transparency in psychological science]] Initiatives: * [[https://www.nature.com/articles/s41562-016-0021|A manifesto for reproducible science]] (Nature Human Behavior, 2017) * [[https://osf.io/preprints/psyarxiv/nhuv3|A Manifesto for Open Science in Giftedness Research]] (preprint) * [[http://www.researchtransparency.org/|Commitment to Research Transparency and Open Science]] * [[http://michaelinzlicht.com/getting-better/2016/10/11/check-yourself-again|Check yourself again - Michael Inzlicht]] * [[http://journal.frontiersin.org/article/10.3389/fpsyg.2016.01832/full|Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking]] (Frontiers, 2016) * [[https://osf.io/preprints/psyarxiv/w9e3r|Constraints on Generality (COG): A Proposed Addition to All Empirical Papers]] (preprint) * [[http://journals.sagepub.com/doi/full/10.1177/1948550616673876|The GRIM Test A Simple Technique Detects Numerous Anomalies in the Reporting of Results in Psychology]] (SPPS, 2017) - [[http://www.prepubmed.org/grim_test/|example1]] / [[http://www.prepubmed.org/general_grim/|example2]] * [[https://osf.io/wfc6u/|Collaborative Replications and Education Project (CREP)]] * [[https://twitter.com/JohnAntonakis/status/919182350029869056|Data reporting at The Leadership Quarterly]] * [[http://web.uvic.ca/~dslind/?q=node/209|Nineteen Things Editors of Experimental Psychology Journals can do to Increase the Replicability of the Research they Publish]] * [[http://journals.sagepub.com/doi/abs/10.1177/1745691614549257|Analytic Review as a Solution to the Misreporting of Statistical Results in Psychological Science]] (PPS, 2014) * [[https://psyarxiv.com/fuzkh|Verify Original Results through Reanalysis before Replicating]] (reply to BBS, 2018) * [[https://psyarxiv.com/6ka9z|Psychological Science needs a standard practice of reporting the reliability of cognitive behavioural measurements]] (preprint, 2018) Solutions / platforms: * [[https://curatescience.org/|Curate Science]] * [[https://dataverse.harvard.edu/|Dataverse]] * [[http://bulliedintobadscience.org/|Bullied Into Bad Science]] Articles: * [[http://www.sciencedirect.com/science/article/pii/S0022103115300019|The pipeline project: Pre-publication independent replications of a single laboratory's research pipeline]] (JESP, 2016) * [[https://osf.io/8uz2g/|A pre-registration primer]] (OSF presentation) * [[http://blogs.lse.ac.uk/impactofsocialsciences/2013/07/19/callforparticipants-platform/|The ‘Call for Participants’ platform connects researchers with participants so as to improve the efficiency and accuracy of research trials.]] * [[https://elifesciences.org/content/5/e16800|How open science helps researchers succeed]] (eLife, 2016) * [[http://eml.berkeley.edu/~sdellavi/wp/expertsJul16.pdf|Predicting Experimental Results: Who Knows What?]] (Della and Pope, in review, 2016) * [[http://www3.nd.edu/~ghaeffel/Value.pdf|The Value of Direct Replication]] (PPS, 2014) * [[https://osf.io/preprints/psyarxiv/fcxge/|Benefits of Open and High-Powered Research Outweigh Costs]] (JPSP, 2017) * [[https://osf.io/preprints/psyarxiv/nnkg9|Too True to be Bad: When Sets of Studies with Significant and Non-Significant Findings Are Probably True]] (Lakens & Etz, 2017) * something similar - The ironic effect of significant results on the credibility of multiple-study articles ([[https://www.ncbi.nlm.nih.gov/pubmed/22924598|article]] / [[https://replicationindex.wordpress.com/2018/02/18/why-most-multiple-study-articles-are-false-an-introduction-to-the-magic-index/|blog post]]) * [[http://deevybee.blogspot.com/2017/05/reproducible-practices-are-future-for.html|Reproducible practices are the future for early career researchers]] (Bishopblog) * [[https://www.nature.com/articles/s41559-017-0160|Our path to better science in less time using open data science tools]] (nature ecology & evolution, 2017) * [[https://psyarxiv.com/7wkdn|Transparent science: A more credible, reproducible, and publishable way to do science]] (preprint, 2018) * [[https://psyarxiv.com/2yphf/|Implications of the Credibility Revolution for Productivity, Creativity, and Progress]] (Simine Vazire, PPS, 2018) * [[http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002128|Beyond Bar and Line Graphs: Time for a New Data Presentation Paradigm]] (PLOSOne, 2018) * [[http://www.willem.maartenfrankenhuis.nl/wp-content/uploads/2016/09/Frankenhuis-Nettle-in-press-Open-Science-for-PPS.pdf|Open Science is Liberating and Can Foster Creativity]] (PPS, 2018) Sharing data/code: * [[http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0026828|Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results]] (PLOSOne, 2011) Pre-registration: * [[http://www.psychologicalscience.org/publications/observer/2016/nov-16/seven-selfish-reasons-for-preregistration.html|Seven Selfish Reasons for Preregistration]] (Observer, 2016) * [[https://twitter.com/ceptional/status/954801590069678080|Improving on pre-registration]] Video lectures: * [[http://www.psychologicalscience.org/publications/observer/obsonline/improving-reproducibility.html#.WEF-_S0rKUk|Improving the Reproducibility of Our Research Practices]] (APS, 2016, on Youtube) * [[https://www.youtube.com/watch?v=6KaXVDYv-jw&feature=youtu.be|Professor John Antonakis discusses the 5 diseases of academic publishing]] Meta analyses: * [[https://www.human.cornell.edu/hd/qml/upload/Braver_Thoemmes_2014.pdf|Continuously Cumulating Meta-Analysis and Replicability]] (PPS, 2014) Crowdsourcing/many-labs: * Crowdsourcing Science PhD Course [[https://www.researchgate.net/publication/301770826_Crowdsourcing_Science_PhD_Course_slides|slides]] / [[https://www.researchgate.net/publication/301770593_Crowdsourcing_Science_PhD_Course_syllabus|syllabus]] by [[https://twitter.com/m_schweinsberg?lang=en|Martin Schweinsberg]] Teaching/education: * [[https://osf.io/wfc6u/|Collaborative Replications and Education Project (CREP)]] * [[http://nobaproject.com/modules/the-replication-crisis-in-psychology|The Replication Crisis in Psychology NOVA]] * [[https://docs.google.com/spreadsheets/d/1kzJDrj3dtL9WOz_zRMEhgR7xxo9p3pGQOLJMUVkO1A0/edit#gid=0|Open Science Teaching and Training Resources]] ===== Questionable research practices ===== ===== Replications ===== * [[https://psyarxiv.com/4tg9c|Making Replication Mainstream]] (BBS, 2918) * [[http://science.sciencemag.org/content/351/6277/1037.2|Comment on “Estimating the reproducibility of psychological science”]] (Science, 2016) (covered in [[http://www.slate.com/blogs/the_slatest/2016/03/03/psychology_study_that_induced_the_reproducibility_crisis_was_wrong.html|Slate]]) * [[http://psycnet.apa.org/psycinfo/2013-25331-001/|P-Curve: A Key to the File-Drawer]] JEP-G , example for implementation - "[[http://journals.sagepub.com.sci-hub.cc/doi/pdf/10.1177/0956797616658563|Power Posing: P-Curving the Evidence]]" (PsycSCi, 2017) ([[https://osf.io/ujpyn/|data in OSF]]) * [[http://www.nature.com/news/replication-studies-bad-copy-1.10634|Replication studies: Bad copy]] * [[http://ahp.apps01.yorku.ca/?p=6021|Interactive Timeline: “Replication in Psychology: A History Perspective”]] (Advances in the history of psychology) * [[http://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970?WT.mc_id=TWT_NatureNews|1,500 scientists lift the lid on reproducibility Survey sheds light on the ‘crisis’ rocking research]] (Nature, 2016) * [[http://willgervais.com/blog/2017/3/2/post-publication-peer-review|Post publication peer review 2017 - Will Gervais on his own Science article]] * [[https://www.researchgate.net/publication/278082978_The_Replication_Paradox_Combining_Studies_can_Decrease_Accuracy_of_Effect_Size_Estimates|The Replication Paradox: Combining Studies can Decrease Accuracy of Effect Size Estimates]] (Review of General Psychology,2015) * [[https://www.nature.com/articles/d41586-018-02108-9|How to make replication the norm]] (Nature, 2018) Satire: * [[http://www.psi-chology.com/replication-mirror/|Replication Mirror]] ===== Problems with P values ===== * [[http://journal.frontiersin.org/article/10.3389/fphy.2016.00006/abstract?utm_source=FRN|P Values: Misunderstood and Misused]] (PLOSOne, 2016) * [[http://rolfzwaan.blogspot.com/2015/05/p20-what-now-adventures-of-good-ship.html|p=.20, what now? Adventures of the Good Ship DataPoint]] (Rolf Zwaan, 2015) * On the hazards of significance testing. [[http://www.dcscience.net/2014/03/10/on-the-hazards-of-significance-testing-part-1-screening/|Part 1: the screening problem]] * On the hazards of significance testing. [[http://www.dcscience.net/2014/03/24/on-the-hazards-of-significance-testing-part-2-the-false-discovery-rate-or-how-not-to-make-a-fool-of-yourself-with-p-values/|Part 2: the false discovery rate, or how not to make a fool of yourself with P values]] * [[http://www.chronicle.com/article/Spoiled-Science/239529|Spoiled Science How a seemingly innocent blog post led to serious doubts about Cornell’s famous food laboratory]] (Chronicles, 2017) * [[http://psychbrief.com/im-a-non-methodologist-does-it-matter-if-my-definition-is-slightly-wrong/|I’m a non-methodologist, does it matter if my definition is slightly wrong? PsychBrief]] ===== Problems with peeking ===== * [[https://neuroneurotic.net/2016/08/25/realistic-data-peeking-isnt-as-bad-as-you-thought-its-worse/|Realistic data peeking isn’t as bad as you* thought – it’s worse]] * [[http://daniellakens.blogspot.com/2014/06/data-peeking-without-p-hacking.html|Data peeking without p-hacking]] * [[http://sometimesimwrong.typepad.com/wrong/2015/06/guest-post-excuses-for-data-peeking.html|Excuses for Data Peeking]] ===== Statistical analyses ===== * [[https://twitter.com/aecoppock/status/937364194374111233|Subgroup analyses]] * mediation - [[http://www.sciencedirect.com/science/article/pii/S0022103117300628|Unwarranted inferences from statistical mediation tests – An analysis of articles published in 2015]] (JESP, 2018) * **[[https://link.springer.com/article/10.1007%2Fs00210-014-1037-6|Common misconceptions about data analysis and statistics]] (Naunyn-Schmiedeberg's Archives of Pharmacology, 2014), nice flowchart of p-hacking, generally great intro paper.** * [[https://osf.io/f3yk8/|Journal Data Sharing Policies and Statistical Reporting Inconsistencies in Psychology]] ===== Issues with management research ===== * [[http://opr.sagepub.com/content/3/2/187|Organizational sciences’ obsession with “that’s interesting!”, Consequences and an alternative]] * [[http://asq.sagepub.com/content/61/1/1?etoc|Ruminations on How We Became a Mystery House and How We Might Get Out]] (Barley, JOM, 2016) * [[http://www.siop.org/tip/oct16/integrity.aspx#sthash.SoNjiRFW.uxfs|I-O Psychology’s Lack of Research Integrity]] (SIOP, 2016) * [[http://onlinelibrary.wiley.com/doi/10.1111/peps.12111/abstract|HARKing's Threat to Organizational Research: Evidence From Primary and Meta-Analytic Sources]] (Personnel Psychology, 2015) * [[https://www.researchgate.net/publication/321734216_HARKing_How_Badly_Can_Cherry-Picking_and_Question_Trolling_Produce_Bias_in_Published_Results|HARKing: How Badly Can Cherry-Picking and Question Trolling Produce Bias in Published Results]]? (JBP, 2017) ===== randomized controlled trials ===== * [[https://www.sciencedirect.com/science/article/pii/S0277953617307359|Understanding and misunderstanding randomized controlled trials]] (Social Science & Medicine, 2017) ===== Problems with analysis and interpretation ===== * [[https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3036573|Health Services as Credence Goods: A Field Experiment]] ===== Concerns and negative reactions to the change in psychological science ===== * [[https://www.dropbox.com/s/9zubbn9fyi1xjcu/Fiske%20presidential%20guest%20column_APS%20Observer_copy-edited.pdf?dl=0|Susan Fiske - APS Observer, in pressMob Rule or Wisdom of Crowds?]] * [[http://daniellakens.blogspot.com/2016/09/why-scientific-criticism-sometimes.html|Why scientific criticism sometimes needs to hurt]] (Daniel Lakens) * [[https://medium.com/@jamesheathers/meet-the-new-bad-people-4922137949a1|Meet The New Bad People]] (Medium, 2017) ===== Other ===== * [[http://www.sjscience.org/article?id=46#|Principles of the Self-Journal of Science: bringing ethics and freedom to scientific publishing]] (on an interesting platform). Press releases issues: * [[http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0168217|Exaggerations and Caveats in Press Releases and Health-Related Science News]] (PLOSone, 2016) ===== Courses ===== * [[http://callingbullshit.org/syllabus.html|Calling Bullshit In the Age of Big Data course]] * [[https://www.coursera.org/learn/statistical-inferences|Improving your statistical inferences]] * [[https://www.coursera.org/learn/reproducible-research|Reproducible Research]] * [[https://www.coursera.org/specializations/jhu-data-science|Statistical Inference]] ===== Scandals ===== ==== The food lab ==== * http://steamtraen.blogspot.com/2017/03/strange-patterns-in-some-results-from.html * http://www.chronicle.com/article/Spoiled-Science/239529 * http://steamtraen.blogspot.com/2017/03/some-instances-of-apparent-duplicate.html * http://www.timvanderzee.com/the-wansink-dossier-an-overview/ * http://steamtraen.blogspot.com/2017/03/more-apparent-duplication-from-food-and.html ===== What replicates well ===== * [[https://osf.io/preprints/psyarxiv/rbz29|Some Psychological Effects Replicate Even Under Potentially Adverse Conditions]] (preprint) ==== Grant culture ==== * [[http://journals.sagepub.com/doi/abs/10.1177/1745691616687745|Psychology’s Replication Crisis and the Grant Culture: Righting the Ship]] (PPS, 2017) ==== Opposition ==== * [[http://journals.sagepub.com/doi/pdf/10.1177/1745691617707314|Cultural Change Over Time: Why Replicability Should Not Be the Gold Standard in Psychological Science]] (PPS, 2017) issues_with_research_readings.txt Last modified: 2018/08/05 20:05by filination