Factors influencing raters’ scoring decision and their rating practice development: A study of a high-stakes test in Vietnam

This study explores the issue of rating speaking from a so far underused qualitative research perspective. This qualitative study investigated the experiences of Vietnamese-L1 raters in scoring a localised high-stakes test of English proficiency, Vietnamese standardised test of English proficiency (VSTEP). There were three overarching aims that the thesis aimed to achieve:

- To further an understanding of the rating process experienced by Vietnamese raters in the assessment of speaking performance in English as a foreign language
- To further an understanding of the factors which affect the raters’
scoring decisions
- To further an understanding of the ways raters develop their rating practice over time

All of the 14 participants were Vietnamese teachers of English as a foreign language (EFL), had attended rater training programmes and had rated for at least a year. First, all of the participants were invited to participate in a moderation discussion when the participants rated three bench-marking performances and discussed why they arrived at their scores. After that they were asked to use think-aloud protocols (TAPs) to rate 15 VSTEP speaking performances under the audio rating condition. The participants were then interviewed using a semi-structured approach. Data from moderation discussions, TAPs and interviews were all recorded and then analysed using interpretative phenomenological analysis.

The findings suggested that there were individual differences in perceptions of assessment criteria, rating procedure, score decision time and score decision strategies among the raters though they were recruited from a homogenous group. However, they all experienced a three-stage process while rating: consciously paying attention to the speech features, allocating their first scores and finalising the scores using 5 different decision-making strategies. There existed a number of differences which distinguished novice raters from experienced ones. For example, experienced raters provided more and longer comments on test-takers’ speech than novice raters and with more confidence. Regarding the factors contributing to score variations, the treatment of local speech features, the English standard(s) and the perception of syntactic and lexical accuracy and complexity and discourse competence seemed to play significant roles in generating such differences. The data also showed that the rating experience and the number of trainings contributed considerably to the raters’ rating development. The study reveals the three stages of how novice raters evolved into experienced raters.

The results extend our understanding of the rating process in the assessment of speaking performance in EFL and how rating behaviours evolved with training and practice. The study also contributes to a better understanding of local raters’ perceptions and practices of assessing English in their local context. There were significant implications which could be drawn from this study for speaking constructs assessed in a localised EFL context. Moreover, the study has implications for the enhancement of the effectiveness of rater training programmes and the standardisation procedures.