How to guides


Data cleaning

Visualization galleries:

-@fredericksolt's interplot R, interflex R (through Twitter)

ANOVA two way interaction (with constrasts:

Basic code:


(might want to change the “EMMEANS=TABLES(IV1*IV2) COMPARE(IV2)” to “EMMEANS=TABLES(IV1*IV2) COMPARE(IV1)” to see if more convinient)



Interpreting effect sizes, Andy Field summarizes in his methods book:

Cohen (1992, 1988) has made some widely accepted suggestions about what constitutes a large or small effect: •r = 0.10 (small effect): in this case, the effect explains 1% of the total variance. •r = 0.30 (medium effect): the effect accounts for 9% of the total variance. •r = 0.50 (large effect): the effect accounts for 25% of the variance.

Table: effect-size.png

Readings about effect size estimates in psychology:

Forest plots:

Combining two metas (SEM):

Best way to assess publication bias:

Pooling means and SD/variance:

New directions:


Promising packages:

In the last few lab meetings some of you mentioned that you are interested in collecting data with Amazon Mechanical Turk but that you do not have an account and asked if I might be able to help. I have an account, and have had some experience with this, so I’d be happy to help you run your studies and to help you adjust them to MTurk to increase the chances of getting reliable high-quality data. The email below is to give you an idea of how we can work on that.

First, a quick background. Although MTurk does have some limitations, many of the issues that came up in the various lab meetings can definitely be addressed with MTurk data collection – a replication attempt, scale validation, a pre-test to determine power, quick access to working people and/or underprivileged sectors, etc. I personally believe that MTurk is also good enough for running an independent study complementary to other collected data – if done correctly. In my experience, MTurk data collection is comparable if not better than student participant pool data. I also use a tool called TurkPrime that rides on top on MTurk and intended for academics, that completely automates academic data collection and increases data reliability (preventing duplicates and lots of potential problems, etc.).

My experience is summarized in the following two posts:

If you’d like my help to run something with MTurk, what you’ll need to do is:

  • Have a working Qualtrics survey that you’ve tested. I can share a Qualtrics demo with you designed for MTurk that includes a consent form, funneling section, demographics, and debriefing.
  • Make sure you can get reimbursed and have the money available. Once MTurk runs it automatically pays from my account, so I would ask that you transfer the funds to me before I start running this, so that I don’t have to bear the financial costs of waiting for everyone’s reimbursements. If you can’t, we can talk about that.
  • Plan how many participants you’ll need (N), the intended pay (minimum for IRB in some places is 0.05US$ a minute, most do a minimum of 0.10US$ a minute), sample characteristics (location, qualifications, etc.).

Costs you’ll need to take into account : (N + 10+ participants for pretest run) * pay + 20% Amazon MTurk commission [Note: The default Amazon commission is actually 40%, but TurkPrime uses a feature called Micro-Batch that reduces that to 20%. In some study designs this feature cannot be used, and I’ll alert you if that is the case]

The procedure would usually be:

  • You pretest your study with one or two other persons to make sure it’s working well (preferably a number of times to test all conditions, if you have manipulations).
  • We run a pretest of 10 participants on MTurk to see it all goes well (possibly more if you have many conditions).
  • We run a full sample.

Depending on the kind of survey you’re running and your intended sample size, data collection could take anything from half an hour to a few days.

Please let me know if you have any other questions.


You’ll need to work out your sample size (+small pretest) and payment for each participant with the formula before, confirm your reimbursement, and then transfer that money to me in advance. In the meanwhile, make a copy of your survey on Qualtrics and share the copy with me, and I’ll make sure it’s all set.

Also, it’s not a must, ofcourse, but I would strongly encourage you :

  • Calculate needed sample size based on power analysis using G*Power and an expected effect size.
  • Take 15-30min to pre-register your survey layout and your main predictions on OSF or “As predicted”.

To me, personally, as a reviewer, these two things significantly increase my confidence in the findings (and, in a way, in the researcher).


Public goods:

  • PACO (mobile platform for behavioral science) / info

See all kinds of great resources from the SIPS workshop:

Content mining

Do this macro:

Sub RemoveParaOrLineBreaks()
Dim oRng As Range
Dim oFind As Range
    Set oRng = Selection.Range
    Set oFind = Selection.Range
    With oFind.Find
        .Replacement.Text = "\1"
        Do While .Execute(FindText:="[^13^l]{1,}([a-z])", _
                          MatchWildcards:=True, _
    End With
    Set oRng = Nothing
    Set oFind = Nothing
    Exit Sub
End Sub

Add a keyboard shortcut from options→ribbbon→keyboard→macros→RemoveParaOrLineBreaks

  • how_to_guides.txt
  • Last modified: 2019/04/25 19:10
  • by filination