Activities
Why is Reliability Important?
In Chapter 2 we emphasised three key considerations for evaluating the quality of assessments: validity, reliability and fairness. When reporting to students, parents and others on students’ engagement and achievement with portfolios, teachers need to ensure that results and insights offered about students are stable so that users can have confidence in the information provided. This is a challenge for all assessments, and for use of portfolios, in particular.
It is important to distinguish between portfolios as a pedagogical and motivational tool and their use for assessment purpose. The latter assumes the provision of information or feedback to teachers, students, parents and others. Therefore, assessment using portfolios must be useful and offer consistent information.
What Issues Arise for the Reliability of Portfolios?
Based on their work in Vermont in the USA, Koretz et al. (1994b) identified four key challenges for the use of portfolios in large-scale assessment programmes:
- Dependencies caused when a rater grades all entries within a portfolio before moving to the next portfolio. Sometimes called the halo effect, how a rater views one entry in the portfolio can influence for good or bad assessment of other entries in relation to the same student.
- Systematic differences among raters in how they interpret and apply scoring rubrics.
- The nature and detail associated with rubric.
- Heterogeneous content across portfolios where the same entries are not required for all students.
The authors concluded that achieving levels of reliability typically associated with reporting to students and parents would require the inclusion of very many separate entries in the portfolio and the grading of each piece by two or more raters (2004b, p. 52). Such conditions are virtually impossible to achieve in most classroom or assessment contexts.
Possible approaches to surmounting these challenges include:
- Initial training of raters/teachers and students in the use of portfolios.
- Ongoing discussion and moderation involving teachers/raters (and students, where portfolios are used in association with self-assessment) in a school or assessment programme about their work using portfolios and the application of rubrics in particular.
- Greater standardisation of what students within a class/cohort include in portfolios.
- Clear guidelines to students around criteria for including entries in portfolio.
- Development and fidelity to implementation of clear scoring rubrics.
What are Teachers’ Experiences of Using Portfolios?
It can be useful to ascertain teachers’ and students’ perspectives on using portfolios. You can seek such perspectives either by talking with a teacher who uses portfolios or by reviewing video clips such as the Scottish primary school example or the Canadian community college example found at the web addresses below.
Consider some of the issues in the downloadable Reflection Sheet below in light of (i) discussion with another teacher; (ii) video exemplars above; or (iii) your own use of portfolios.
Activity 9.1– Reflecting on Portfolio Use as an Assessment Tool in Class