hku_psyc2020_and_psyc3052

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
hku_psyc2020_and_psyc3052 [2018/05/09 22:42]
filination
hku_psyc2020_and_psyc3052 [2018/05/14 21:00] (current)
filination
Line 2515: Line 2515:
 </​blockquote>​ </​blockquote>​
  
 +==== Question collection #4 ====
 +
 +<​blockquote>​
 +1. For "​Methods",​
 +
 +(a) Is writing about the process we do pre-registration unneccesary in this section? For example,
 +
 +"We first did an article analysis report about the Experiment 1 of the target article and addressed the section 1 and 2 of the Replication Recipe (Question 1- 16). We then redesigned the survey on Qualtrics..." ​
 +
 +(b)As we will include link for pre-registration plan, should we still mentioned about the experiment design with IV and DV if nothing was changed?
 +
 +(c) According to my understanding,​ we should mention about change that is deviated from pre-registered plan in "​supplementary materials"​. However, should we mention it in the " methods"​ section as well and refer it to the supplementary materials?
 +
 +2. As you have mentioned about writing the result according to before-exclusion data to allow convenient comparison, I wonder if I pre-registered the main effect of time (Pronin & Kugler, 2010) as additional analysis, would this create problem to allow comparison with study done by my peer?
 +
 +
 +3. On the other hand, I wonder if there is any way to define the strength of p-value. This is because while doing peer review, my peer has interpreted the strength of p-value as following. If I want to see how important is a significant p-value to interpret its importance, should I use p-curve to see how the other p-values distribute and judge it according to the overall context. Thus i wonder if using p-curve is relevant here. Also. as I know, p- curve calculator needs input from a lot of study that I wonder if there is any p- curve calculator that allows us to see the trend without the input of result from unlimited replication.
 +
 +
 +4. Also, can I use post- hoc power calculator to see if our study has enough power?
 +This is because I realized that we did power analysis based on original study effect size which we assume the effect size to be the true population effect size. However, the original study effect size might not be true. Thus, as our replication study changed in effect size, I would like to look up for its power. At the same time, i wonder if post- hoc power calculation cause opportunity bias?
 +
 +</​blockquote>​
 +
 +I answered:
 +<​blockquote>​
 +  - No need to mention the article analysis stage, the pre-registration is enough as it is registered, public, and official. The article analysis is just us doing our thing.
 +  - Not sure what you mean here, but the experimental design needs to be clear in the main part of the final report. This is what readers will look at to understand your replication experiment.
 +  - You should mention in great detail what deviated from the pre-registration in the supplementary materials, you should briefly mention that – or even better – summarize that in a table, in the main text, referring readers to details in the supplementary.
 +  - Differences in pre-registration for analyses does not create “problems”,​ it is understandable and expected. You don’t need to worry about these things, these are “higher level” things for me when I integrate your reports, just focus on doing the best in your own report. My only mention of the other’s report was in reference to some of my request. If I didn’t make a request, don’t worry about it.
 +  - A p-curve analysis is relevant for multi-study experiment or multi-study such as in a multi-experiment or meta-analysis. Less relevant for one experiment one study. There’s no need to do a p-curve for this project.
 +  - No need to calculate power post-hoc in this project. Post hoc power has all sorts of issues, that we didn’t have time to go into, and maybe needs to be discussed in a methods course. If you already did that – great, it adds some information and definitely doesn’t ruin anything. If you didn’t – no need, leave this out.
 +</​blockquote>​
  • hku_psyc2020_and_psyc3052.txt
  • Last modified: 2018/05/14 21:00
  • by filination