• Investigating the effect of classroom-based feedback on speaking assessment: a multifaceted Rasch analysis

    Author(s):
    Khaled Ahmed Abdel‑Al Ibrahim, Houman Bijani, Bahareh Hashempour, Tahereh Heydarnejad, Salim Said Bani Orabah (see profile)
    Date:
    2022
    Item Type:
    Article
    Tag(s):
    Bias, Interrater consistency, Intrarater consistency, Multifaceted Rasch measurement (MFRM), Rater training, Severity/leniency
    Permanent URL:
    https://doi.org/10.17613/577y-es18
    Abstract:
    Due to subjectivity in oral assessment, much concentration has been put on obtaining a satisfactory measure of consistency among raters. However, the process for obtaining more consistency might not result in valid decisions. One matter that is at the core of both reliability and validity in oral assessment is rater training. Recently, multifaceted Rasch measurement (MFRM) has been adopted to address the problem of rater bias and inconsistency in scoring; however, no research has incorporated the facets of test takers’ ability, raters’ severity, task difficulty, group expertise, scale criterion category, and test version together in a piece of research along with their two-sided impacts. Moreover, little research has investigated how long rater training effects last. Consequently, this study explored the influence of the training program and feedback by having 20 raters score the oral production produced by 300 test-takers in three phases. The results indicated that training can lead to more degrees of interrater reliability and diminished measures of severity/leniency, and biasedness. However, it will not lead the raters into total unanimity, except for making them more self-consistent. Even though rater training might result in higher internal consistency among raters, it cannot simply eradicate individual differences related to their characteristics. That is, experienced raters, due to their idiosyncratic characteristics, did not benefit as much as inexperienced ones. This study also showed that the outcome of training might not endure in long term after training; thus, it requires ongoing training throughout the rating period letting raters regain consistency.
    Metadata:
    Published as:
    Journal article    
    Status:
    Published
    Last Updated:
    2 months ago
    License:
    All Rights Reserved
    Share this:

    Downloads

    Item Name: pdf investigating-the-effect-of-classroom-based-feedback-on-speaking-assessment-a-multifaceted-rasch-analysis.pdf
      Download View in browser
    Activity: Downloads: 59