Carlson, Kevin D. and Jinpei Wu. 2012. The Illusion of Statistical Control: Control Variable Practice in Management Research. Organizational Research Methods 15 (3): 413-435.
Abstract: The authors extend previous recommendations for improved control variable (CV) practice in management research by mapping the objectives for using statistical control to recommendations for research practice. Including CVs in research designs to permit statistical control of “nuisance” variance is a common research practice that is subject to well-documented and potentially serious problems. Yet because CVs are frequently weakly related to focal variables, they rarely influence the interpretation of results. As a result, current practice offers an illusion of statistical control when in fact little control actually occurs. The authors extend the growing literature on CV practice by examining the ambiguity of researchers' stated purposes for using statistical control that makes it difficult to determine whether common CV practice accomplishes any of these intents effectively. Guidelines for improving research practice are offered, including adopting a conservative stance toward the inclusion of CVs in the analysis of quasiexperimental and correlational designs guided by the principle “When in doubt, leave them out.”
- What problems do the authors identify in using control variables in research, particularly in terms of misunderstanding relationships among variables and misinterpreting findings?
- Discuss how control variables are used in organizational research.
- What role do intervening variables (mediators, moderators) play in understanding relationships between dependent and independent variables?
- Please discuss how researchers may control for extraneous variables / effects through experimental or statistical means.
Goetz, Michal, et. al. 2012. A 12-Month Prospective, Observational Study of Treatment Regimen and Quality of Life Associated With ADHD in Central and Eastern Europe and Eastern Asia. Journal of Attention Disorders 16 (1): 44-59.
Abstract: Objectives: This prospective, observational, non-randomized study aimed to describe the relationship between treatment regimen prescribed and the quality of life (QoL) of ADHD patients in countries of Central and Eastern Europe (CEE) and Eastern Asia over 12 months. Methods: 977 Male and female patients aged 6-17 years seeking treatment for symptoms of ADHD were assessed using the Child and Adolescent Symptom Inventory-4 Parent Checklists, and the Clinical Global Impressions-ADHD-Severity scale. QoL was assessed using the Child Health and Illness Profile-Child Edition parent report form. Patients were grouped according to whether they were prescribed psycho- and/or pharmacotherapy (treatment) or not (no/‘other’ treatment). Results: No statistically significant differences were observed between cohorts (treatment vs. no/‘other’ treatment) in terms of change in QoL, although there was improvement over 12 months, with a greater improvement experienced by patients in the treatment cohort in both study regions (CEE and Eastern Asia). Psychoeducation/counselling and methylphenidate were the predominant ADHD treatments prescribed. Conclusions: Although both treatment and no/‘other’ treatment cohorts showed improvements in mean QoL over 12 months, the difference was small and not statistically significant. A major limitation was the higher than anticipated number of patients switching treatments, predominantly from the no/‘other’ treatment cohort.
- Please discuss how the researchers of this article make use of the observational study approach to test for relationships between variables.
- Please discuss how participants were chosen and grouped for the purposes of this study. Were there any potential biases in the cohort that was created?
- Please discuss what limitations may be apparent in this study (particularly in terms of patient switching).
Spector, Paul E., and Michael T. Brannick. 2011. Methodological Urban Legends: The Misuse of Statistical Control Variables. Organizational Research Methods 14 (2): 287-305.
Abstract: The automatic or blind inclusion of control variables in multiple regression and other analyses, intended to purify observed relationships among variables of interest, is widespread and can be considered an example of practice based on a methodological urban legend. Inclusion of such variables in most cases implicitly assumes that the control variables are somehow either contaminating the measurement of the variables of interest or affecting the underlying constructs, thus distorting observed relationships among them. There are, however, a number of alternative mechanisms that would produce the same statistical results, thus throwing into question whether inclusion of control variables has led to more or less accurate interpretation of results. The authors propose that researchers should be explicit rather than implicit regarding the role of control variables and match hypotheses precisely to both the choice of variables and the choice of analyses. The authors further propose that researchers avoid testing models in which demographic variables serve as proxies for variables that are of real theoretical interest in their data.
- What concerns do the authors discuss on including control variables into multiple regression or other types of analyses, particularly in terms of interpreting the results?
- What role do control variables play as extraneous variables in the research process?
- What recommendations do the authors have for researchers in selecting control variables that are of value to their study?