Assessing English Writing in the Austrian Educational Standards Basel...

The way raters understand rating criteria is an important aspect of assessment literacy. The meaning that raters attribute to the dimensions and level descriptors of rating scales determines to a large extent the test constructs that are measured (cf. Lumley 2002, 2005; Weir 2005; Knoch 2009). Most probably, raters take into account positive as well as negative features, or errors, in the process of assessment. This contribution focuses on the impact of errors on the ratings that performances receive on the analytic rating scale in use since 2009. The focus is on the distinction between the Coherence & Cohesion dimension and the Task Achievement dimension. In order to address these issues, an error analysis of 100 writing performances was carried out. Errors were identified in terms of linguistic category (grammar, lexis, orthography and punctuation, discourse) and surface-structure alterations (addition, misformation, misordering, omission) (cf. Ellis and Barkhuizen 2005; Sigott et al. 2016). The ratings awarded to each performance by the panel of trained raters were submitted to Many-Facet Rasch Analysis (Eckes 2011; Linacre 2003) in order to partial out differences in rater severity. The resulting Fair Measures were used as dependent variables in correlational and regression analyses. Overall, significant moderate to high correlations were found between the error categories and the Total Fair Measure (cf. Weltig 2004; Frey and Heringer 2007) as well as the Fair Measures for the individual rating dimensions (cf. e.g., Sweedler-Brown 1993; Eckes 2008). Orthography and Punctuation errors correlate significantly with all four rating dimensions. Discourse-level errors correlate significantly with both Coherence and with Task achievement (cf. Hyland and Anan 2006). Most importantly, however, coherence errors are a stronger predictor of the Task Achievement rating than are pragmatic errors. This indicates that the raters were interpreting the rating dimensions of Coherence & Cohesion and that of Task Achievement in ways which were not intended by the test constructors. Possibly, raters interpreted flaws in task achievement as breaches in the coherence between the writing prompt and the performance. The results give rise to the question of whether the distinction between Coherence & Cohesion and Task Achievement should be conceptually sharpened and maintained, or be abandoned. In addition, the implications of these findings for the construct validity of the E8 Standards tests as well as for teaching practice are discussed.