[R-sig-ME] Different p_values from between groups to within groups

[Luca Danieli]

Hello everybody,

may I ask you a suggestion on how to interpret a weird result I have?

I have 3 groups (ExpertiseType), and through the use of contrast hypotheses, the first model gives me this output:

model = lmer(Score~Closure*ExpertiseType+(1|Participant)+(1|Item), database, REML=TRUE)
Pr(>|t|)
Closurecl_c1:ExpertiseTypeexp_c1 0.42203
Closurecl_c2:ExpertiseTypeexp_c1 0.00601 **
Closurecl_c3:ExpertiseTypeexp_c1 9.32e-08 ***

Another, more detailed model, gives me the following:

model = lmer(Score~Closure*ExpertiseType+ExpertiseType*LastPosition+Closure*LastPosition+(1|Participant)+(1|Item), database, REML=TRUE)

Pr(>|t|)
Closurecl_c1:ExpertiseTypeexp_c1 0.50738
Closurecl_c2:ExpertiseTypeexp_c1 0.01059 *
Closurecl_c3:ExpertiseTypeexp_c1 4.05e-08 ***

As you can notice, I have an interaction in both analyses, but if I look for the same contrast hypotheses within the group for which I have the interaction, the Closurecl_c2 effect disappeares.

model = lmer(Score~Closure*LastPosition+(1|Participant)+(1|Item), subset(database, ExpertiseType==3), REML=TRUE)

Pr(>|t|)
Closurecl_c1 0.4411
Closurecl_c2 0.1419
Closurecl_c3 5.00e-07 ***

Which one should I consider the most reliable output?
Or, alternatively, what does this difference mean? I really don't know how to interpret this outcome. I was expecting that within groups, the analysis would get more defined.

Best
Luca



[[alternative HTML version deleted]]

[Thierry Onkelinx]

Dear Luca,

Those p-values are conditional on the model and not to be used for
model comparison. First find out which model is the most appropriate,
then to the post hoc tests.

Best regards,

ir. Thierry Onkelinx
Statisticus / Statistician

Vlaamse Overheid / Government of Flanders
INSTITUUT VOOR NATUUR- EN BOSONDERZOEK / RESEARCH INSTITUTE FOR NATURE
AND FOREST
Team Biometrie & Kwaliteitszorg / Team Biometrics & Quality Assurance
***@inbo.be
Havenlaan 88 bus 73, 1000 Brussel
www.inbo.be

///////////////////////////////////////////////////////////////////////////////////////////
To call in the statistician after the experiment is done may be no
more than asking him to perform a post-mortem examination: he may be
able to say what the experiment died of. ~ Sir Ronald Aylmer Fisher
The plural of anecdote is not data. ~ Roger Brinner
The combination of some data and an aching desire for an answer does
not ensure that a reasonable answer can be extracted from a given body
of data. ~ John Tukey
///////////////////////////////////////////////////////////////////////////////////////////

Post by Luca Danieli
Hello everybody,
may I ask you a suggestion on how to interpret a weird result I have?
model = lmer(Score~Closure*ExpertiseType+(1|Participant)+(1|Item), database, REML=TRUE)
Pr(>|t|)
Closurecl_c1:ExpertiseTypeexp_c1 0.42203
Closurecl_c2:ExpertiseTypeexp_c1 0.00601 **
Closurecl_c3:ExpertiseTypeexp_c1 9.32e-08 ***
model = lmer(Score~Closure*ExpertiseType+ExpertiseType*LastPosition+Closure*LastPosition+(1|Participant)+(1|Item), database, REML=TRUE)
Pr(>|t|)
Closurecl_c1:ExpertiseTypeexp_c1 0.50738
Closurecl_c2:ExpertiseTypeexp_c1 0.01059 *
Closurecl_c3:ExpertiseTypeexp_c1 4.05e-08 ***
As you can notice, I have an interaction in both analyses, but if I look for the same contrast hypotheses within the group for which I have the interaction, the Closurecl_c2 effect disappeares.
model = lmer(Score~Closure*LastPosition+(1|Participant)+(1|Item), subset(database, ExpertiseType==3), REML=TRUE)
Pr(>|t|)
Closurecl_c1 0.4411
Closurecl_c2 0.1419
Closurecl_c3 5.00e-07 ***
Which one should I consider the most reliable output?
Or, alternatively, what does this difference mean? I really don't know how to interpret this outcome. I was expecting that within groups, the analysis would get more defined.
Best
Luca
[[alternative HTML version deleted]]
_______________________________________________
https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models