SAGE Journal Articles

Click on the following links. Please note these will open in a new window.

Thomas, D. R., & Zumbo, B. D. (2012). Difference scores from the point of view of reliability and repeated-measures ANOVA: In defense of difference scores for data analysis. Educational and Psychological Measurement, 72(1), 37–43. doi:10.1177/0013164411409929.

There is such a doubt in research practice about the reliability of difference scores that granting agencies, journal editors, reviewers, and committees of graduate students’ theses have been known to deplore their use. This most maligned index can be used in studies of change, growth or perhaps discrepancy between two measures taken on the same sampling unit. The most commonly stated problem with difference scores is the supposed associated increase in unreliability of difference scores. In this article, the authors examine difference scores from the point of view of reliability and repeated-measures ANOVA. The authors demonstrate that when the focus of difference scores is data analysis of aggregate models, their use should not be assessed in terms of reliability, and that the complete abolition of difference scores in research practice is unwarranted.

Questions to Consider

1. Summarize the authors’ conclusions regarding difference scores from the point of view of reliability and repeated-measures ANOVA.

Cognitive Domain: Comprehension

Difficulty Level: Medium

 

2. Why is it that the reliability of difference scores has been looked down upon by granting agencies, journal editors, reviewers, and committees of graduate students’ theses?

Cognitive Domain: Knowledge

Difficulty Level: Medium

 

3. What appears to be the most commonly stated problem with difference scores? Why?

Cognitive Domain: Comprehension

Difficulty Level: Medium

 

Rosopa, P. J., Schroeder, A. N., & Doll, J. L. (2016). Detecting between-groups heteroscedasticity in moderated multiple regression with a continuous predictor and a categorical moderator: A Monte Carlo study. SAGE Open, 6(1), 215824401562111. doi:10.1177/2158244015621115.

Moderated multiple regression (MMR) is frequently used to test moderation hypotheses in the behavioral and social sciences. In MMR with a categorical moderator, between-groups heteroscedasticity is not uncommon and can inflate type I error rates or reduce statistical power. Compared with research on remedial procedures that can mitigate the effects of this violated assumption, less research attention has focused on statistical procedures that can be used to detect between-groups heteroscedasticity. In the current article, we briefly review such procedures. Then, using Monte Carlo methods, we compare the performance of various procedures that can be used to detect between-groups heteroscedasticity in MMR with a categorical moderator, including a heuristic method and a variant of a procedure suggested by O’Brien. Of the various procedures, the heuristic method had the greatest statistical power at the expense of inflated type I error rates. Otherwise, assuming that the normality assumption has not been violated, Bartlett’s test generally had the greatest statistical power when direct pairing occurs (i.e., when the group with the largest sample size has the largest error variance). In contrast, O’Brien’s procedure tended to have the greatest power when there was indirect pairing (i.e., when the group with the largest sample size has the smallest error variance). We conclude with recommendations for researchers and practitioners in the behavioral and social sciences.

Questions to Consider

1. Summarize the MMR and its use to test moderation hypotheses in the behavioral and social sciences.

Cognitive Domain: Comprehension

Difficulty Level: Medium–Hard

 

2. What does it mean to violate the homoscedasticity assumption and what sort of error does this often lead to?

Cognitive Domain: Knowledge, Comprehension

Difficulty Level: Hard

 

3. In this study, which test is the most powerful for detecting between-groups heteroscedasticity when the normality assumption is satisfied and direct pairing exists? Why?

Cognitive Domain: Comprehension, Analysis

Difficulty Level: Hard

 

Schlattmann, P., & Dirnagl, U. (September 01, 2010). Statistics in experimental cerebrovascular research: Comparison of more than two groups with a continuous outcome variable. Journal of Cerebral Blood Flow & Metabolism, 30(9), 1558–1563.

A common setting in experimental cerebrovascular research is the comparison of more than two experimental groups. Often, continuous measures such as infarct volume, cerebral blood flow or vessel diameter are the primary variables of interest. This article presents the principles of the statistical analysis of comparing more than two groups using ANOVA. We will also explain post hoc comparisons, which are required to show which groups significantly differ once ANOVA has rejected the null hypothesis. Although statistical packages perform ANOVA and post hoc contrast at a key stroke, in this study, we use examples from experimental stroke research to reveal the simple math behind the calculations and the basic principles. This will enable the reader to understand and correctly interpret the readout of statistical packages and to help prevent common errors in the comparison of multiple means.

Questions to Consider

1. Describe and explain some of the post hoc comparisons included in the article. When and why are these used?

Cognitive Domain: Comprehension, Analysis

Difficulty Level: Medium–Hard

 

2. Why is it important to understand the math behind the calculations and the basic principles? How does this help aid interpreting statistical readouts?

Cognitive Domain: Comprehension

Difficulty Level: Medium

 

3. Summarize the article’s explanation of the ANOVA; to include between-group variability, within-group variability and F-value.

Cognitive Domain: Comprehension

Difficulty Level: Medium–Hard

 

Grier et al. (2003). The vigilance decrement reflects limitations in effortful attention, not mindlessness. Human Factors: The Journal of the Human Factors and Ergonomics Society, 45(3), 349–359.

Robertson et al. (1997) proposed that the decline in performance efficiency over time in vigilance tasks (the vigilance decrement) is characterized by “mindlessness” or a withdrawal of attentional effort from the monitoring assignment. We assessed that proposal using measures of perceived mental workload (NASA-TLX) and stress (Dundee Stress State Questionnaire). Two types of vigilance task were employed: a traditional version, wherein observers made button-press responses to signify detection of rarely occurring critical signals, and a modified version, developed by Robertson et al. to promote mindlessness via routinization, wherein button-press responses acknowledged frequently occurring neutral stimulus events and response withholding signified critical signal detection. The vigilance decrement was observed in both tasks, and both tasks generated equally elevated levels of workload and stress, the latter including cognitions relating to performance adequacy. Vigilance performance seems better characterized by effortful attention (mindfulness) than by mindlessness. Actual or potential applications of this research include procedures to reduce the information-processing demand imposed by vigilance tasks and the stress associated with such tasks.

Questions to Consider

1. Briefly describe the vigilance decrement and why a repeated measures ANOVA would be an appropriate test of this effect.

Learning Objective: One variable with repeated measures

Cognitive Domain: Knowledge

Difficulty Level: Easy

 

2. Ignoring the condition variable, the authors report a significant effect of performance declining over time, F(4, 220) = 10.32. Based on these degrees of freedom, how many groups, or time points, were analyzed? (a) 4. (b) 5. (c) 10. (d) 220.

Learning Objective: Degrees of freedom

Cognitive Domain: Analysis

Difficulty Level: Medium

 

3. If the authors had included additional groups, or time points, the study’s power would typically: (a) increase, (b) decrease, (c) stay the same, (d) impossible to say.

Learning Objective: Power analysis

Cognitive Domain: Application

Difficulty Level: Medium

 

Neimeyer, G. J., Lee, G. A., Saferstein, J., & Pickett, Y. (2004). Effects of graduate preparation program on undergraduate psychology majors. Teaching of Psychology, 31(4), 247–252.

We conducted two studies to assess and enhance levels of graduate preparation. In Study 1, 248 undergraduates and 28 graduate students completed a new measure of graduate preparation, the Grad Prep Quiz. Results documented the psychometric utility and the predictive validity of the measure, with graduate students scoring significantly higher on this measure than undergraduates. Study 2 provided a controlled study of a graduate preparation program. Compared to the control group, participants in the graduate preparation program demonstrated a significant increase in their Grad Prep Quiz scores. These results support efforts to enhance effective preparation for graduate study in psychology.

Questions to Consider

1. Describe what the researchers were trying to accomplish in Study 2 and why a repeated measures analysis would be appropriate.

Learning Objective: One variable with repeated measures

Cognitive Domain: Application

Difficulty Level: Medium

 

2. On p. 251, the authors conducted a within-subjects ANOVA for the control group with nine participants. According to the power analysis information in Chapter 13, this indicates that the study (a) had adequate power, (b) had too few participants, (c) should have focused on the experimental group, (d) controlled for type 1 error rate.

Learning Objective: Power analysis

Cognitive Domain: Knowledge

Difficulty Level: Medium

 

3. When comparing time points in the control group the authors state “p < 0.05, Tukey.” In this case, Tukey is: (a) a repeated measures ANOVA, (b) the name of one of the groups, (c) a post-hoc test to compare groups, (d) a procedure to correct for sphericity.

Learning Objective: Post hoc comparison

Cognitive Domain: Synthesis

Difficulty Level: Easy