Comparative Study Between Grades Assigned by Professors and Those Generated by Artificial Intelligence: An Example Applied to the Evaluation of Laboratory Reports
Academic Article in Scopus
Assessing argumentative writing in STEM laboratory courses is a persistent challenge due to the high cognitive load required to evaluate scientific reasoning and the well-documented variability among human evaluators. Recent advances in large language models (LLMs) have opened new possibilities for AI-assisted grading, yet empirical evidence comparing AI performance with that of expert instructors in STEM contexts remains limited. This study quantitatively examines the alignment between human evaluators and ChatGPT in the assessment of the discussion and conclusions sections of laboratory reports from an undergraduate chemical engineering course (IQ1001B). Sixty reports were independently graded by four professors and by ChatGPT using a shared rubric. Descriptive statistics, mixed-effects modeling, intraclass correlation coefficients, criterion-level paired t-tests, Bland-Altman analysis, and pairwise comparisons among human evaluators were employed.Results show substantial variability in scoring severity among human evaluators, consistent with prior research on performance-based assessment. ChatGPT assigned systematically higher scores than all human evaluators, with large discrepancies in criteria requiring scientific justification and argumentation quality. Although AI-generated scores fell within the general range of human evaluations, the magnitude of disagreement - reflected in wide limits of agreement - suggests that AI scoring cannot yet substitute human judgment in summative contexts. The findings highlight both the potential and the limitations of AI-assisted grading in STEM education and point toward the need for calibrated, hybrid approaches to leverage AI efficiency while preserving human evaluative rigor.
