Andresen, E. M., Malmstrom, T. K., Miller, D. K., & Wolinsky, F. D. (2006). Reliability and validity of observer ratings of neighborhoods. Journal of Aging and Health, 18(1), 28.

Abstract: Objectives: The objectives were to examine the validity and reliability of a five-item neighborhood measurement scale. Methods: Respondents were enrolled from two catchment areas: a poor inner city and a heterogeneous suburban area. Items combine for a total score of 5 (best) to 20 (worst). The authors compared scales across catchment areas and respondent ratings and assessed interviewer effects and retest reliability. Results: Suburban neighborhood scale scores were 3 points lower (higher socioeconomic status, SES) than the inner-city scores. There was a strong relationship

between scores and participants’ neighborhood ratings. The retest correlation was substantial (.81), but only two of five items achieved kappas above .75. In regression models, interviewer experience and residence and individual interviewers contributed to different ratings, although there was still a marked difference between catchment areas. Discussion: Observer ratings of neighborhoods show promise as a measure of neighborhood SES, despite problems with interviewer effects. Future work should improve objective criteria for ratings.

Discussion Questions:

1. How were the concepts of reliability and validity important to this research article?
2. What specific measurements are the authors conceptualizing and using to answer the objectives of their research?
3. Discuss the concept of interrater reliability and its effect on the findings of the study?

Holzer, Marc, and Kaifeng Yang. 2004.  Performance measurement and improvement: an assessment of the state of the art. International Review of Administrative Sciences 70(1): 15-31

Abstract: This article attempts to show the frontier of government performance measurement.  First, it introduces the sophisticated and effective strategies that public agencies haveutilized for performance measurement within a comprehensive approach to productivity improvement. Multiple measures have been developed to improve a variety of management functions such as goal-setting and resource allocation. The article then outlines the cutting edge of performance measurement. Based on a number of case studies throughout the United States, this article discusses the state of the art in three performance areas: citizen-driven government performance, use of citizen surveys and performance reporting. Finally, the article concludes that the essential question in the future is how government can move to full adoption and implementation of citizen-driven, data-driven decision-making.

Discussion Questions:

1. Please discuss how performance measurement has become an important part of organizational strategy in public agencies, particularly in terms of the decision-making process.
2. From the article, please discuss examples of specific measurements that are used by public agencies to track performance over time.
3. Discuss how using effective measures allows government agencies to present information on outputs and outcomes to other interested stakeholders.
4. Please discuss how public agencies use measurement as part of a continuous improvement process to enhance productivity.

Mills, Hayley, Diane Crone, David V. B. James, and Lynne H. Johnston. 2012. Exploring the Perceptions of Success in an Exercise Referral Scheme: A Mixed Method Investigation.  Evaluation Review 36 (6): 407-429.  Available at: http://erx.sagepub.com/content/36/6/407.

Abstract: Background: Exercise referral schemes feature as one of the prevalent primary care physical activity interventions in the United Kingdom, without extensive understanding of how those involved in providing and participating view success. The present research explores and reveals the constituents of “success,” through comparison, contradiction, and integration of qualitative and quantitative research findings.

Method: A population-based cohort design formed the basis for a mixed method approach to the research. The quantitative component used a three-stage binary logistic regression to identify patient sociodemographic characteristics and referral reasons associated with three outcomes (n = 1,315). The qualitative component (n = 28) comprised four focus groups with patients (n = 17), individual interviews with exercise providers (n = 4), and referring health professionals (n = 7). The research components were compared at discussion stage to offer insights into the concept of “success.”

Results: The integrated findings highlighted the multidimensional nature of the concept of success, containing a wide range of concepts such as empowerment, inclusion, and confidence. The traditional notions of success such as, attendance, weight loss, and blood pressure reduction featured amid a more holistic view which incorporated psychological and social aspects as both influences and outcomes.

Conclusion: These findings can enable future development of more representative evaluations of the benefits of exercise referral. This mixed methods research approach can facilitate the development of sophisticated, tailored, evidence-based interventions in the future.

Discussion Questions:

1. How is “success” in primary care physical activity measured in this study?
2. What factors are used to determine the success or failure of an exercise referral scheme?
3. What are specific limitations of the study, particularly in terms of sample selection?

Snyder, Benjamin D., Brian A. Glaser, and Georgia Bunting Calhoun. 2013. The Juvenile Offender Parent Questionnaire: A Structural Validation Study. Measurement and Evaluation in Counseling and Development 46: 38-49. 

Abstract: This study provides evidence for the structural validity of the Juvenile Offender Parent Questionnaire, developed by Rose, Glaser, Calhoun, and Bates. Data analysis indicates that the model does have an adequate level of fit, providing cross-validation for the original exploratory model.  Clinical implications are also discussed.

Discussion Questions:

1. Please discuss how the measurement of parenting factors that inhibit and contribute to juvenile delinquency rates help researchers to better understand trends in this area.
2. What findings does the study present on the structural validity of the Juvenile Offender Parent Questionnaire?
3. How would modification of the original questionnaire improve the nature of the findings?