What does it mean if a categorical moderator is significant overall but has no
significant pairwise contrasts between moderator levels?
I'm using metaphor to conduct a meta-analysis with a categorical moderator with
3 levels; this yields a significant result:
Test of Moderators (coefficient(s) 1,2,3):
F(df1 = 3, df2 = 37) = 4.6052, p-val = 0.0078
Model Results:
estimate se tval pval
ci.lb ci.ub
factor(sample_data$Participants)Adults 0.3920 0.2847 1.3771 0.1768
-0.1848 0.9688
factor(sample_data$Participants)Online 0.1403 0.1283 1.0935 0.2812
-0.1197 0.4004
factor(sample_data$Participants)Students 0.2350 0.0717 3.2747 0.0023
0.0896 0.3803 **
But then I conduct contrasts between each moderator level, and none of these
are significant (no correction for multiple comparisons applied):
Linear Hypotheses:
Estimate Std. Error z value Pr(>|z|)
Online - Adults == 0 -0.2517 0.3123 -0.806 0.420
Students - Adults == 0 -0.1571 0.2936 -0.535 0.593
Students - Online == 0 0.0946 0.1470 0.643 0.520
(Adjusted p values reported -- none method)
Any thoughts or guides to interpretation are appreciated! My code and sample
data are at the end of the email. My interpretation is that while one of the
moderator levels may have be a significant factor in the overall analysis, the
comparisons between moderator levels are noisier because they test to see if
there is a difference in the weights between the two levels. Given this
pattern of results, I conclude the different moderator levels are probably not
strong predictors of effect size. I'm a bit uncertain if this is correct, and
would appreciate any feedback.
Bob
========
Robert Calin-Jageman
Professor, Psychology
Neuroscience Program Director
Dominican University
Parmer 210
7900 West Division
River Forest, IL 60305
rcalinjage...@dom.edu
708.524.6581
http://calin-jageman.net
Sample data link:
https://www.dropbox.com/s/hzz9wmt1d9tcxsm/red_effect_males.csv?dl=0
Code:
#load required libraries
library("metafor")
library("multcomp")
sample_data <- read.csv("red_effect_males.csv")
#Overall test of categorical moderator, reports significant result
mod_test = rma(yi, vi, mods = ~factor(sample_data$Participants) - 1,
data=sample_data, knha = TRUE)
print(mod_test)
#Now do pairwise contrasts - but these show no significant contrasts....why?
cont_holder <- c(1:length(unique(sample_data$Participants)))
names(cont_holder) <- sort(unique(sample_data$Participants))
print(summary(glht(mod_test, linfct=contrMat(cont_holder, "Tukey")),
test=adjusted("none")))
#Now print individual meta-analysis for each subgroub... Effect sizes estimates
and CIs aren't the same as in overall analysis...why?
subgroup_list <- split(sample_data, sample_data$Participants, drop=FALSE)
for (subgroup in subgroup_list) {
print(paste("Individual results for: ", subgroup$Participants[1]))
print(rma(yi, vi, data=subgroup, knha=TRUE))
}
[[alternative HTML version deleted]]
______________________________________________
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
