Inter-Rater Reliability of Reflective-Writing Assessment in an Undergraduate Professionalism Course in Medical Education

Research output: Contribution to journalArticlepeer-review

Abstract

Reflective writing is increasingly being used in the teaching of professionalism. Because assessment enhances the learning process, effective evaluation of students' reflective writing is needed. The aim of this study was to examine the inter-rater agreement between two different reflective writing assessment rubrics, which categorised reflective writings into four level of reflection, in an undergraduate medical professionalism course. The reflective writing assignments from 63 medical students enrolled in the 2017 medical professionalism course in the Faculty of Medicine Universitas Indonesia were randomly selected and independently assessed by two raters in September 2019. Intraclass correlation (ICC) analysis (two-way mixed effect, single measure) was carried out to determine the inter-rater agreement of the reflective writing assessment. The less detailed instrument showed a low ICC score of 0.43, which was classified into poor inter-rater agreement, whereas the more detailed rubric showed poor to moderate reliability, with ICC scores of 0.50, 0.50, and 0.36 for the score of each criterion, the total score of each assessed criterion, and the overall score of reflection, respectively. Utilising a more detailed (analytic) rubric to assess students' reflective writing produced a relatively higher score of inter-rater reliability, although the reliability achieved using this rubric was still categorised as moderate.

Original languageEnglish
Pages (from-to)87-97
Number of pages11
JournalEducation in Medicine Journal
Volume14
Issue number1
DOIs
Publication statusPublished - 2022

Keywords

  • Inter-rater reliability
  • Reflective writing assessment
  • Self-reflection

Fingerprint

Dive into the research topics of 'Inter-Rater Reliability of Reflective-Writing Assessment in an Undergraduate Professionalism Course in Medical Education'. Together they form a unique fingerprint.

Cite this