The Credible Grading Machine: Automated Essay Scoring in the DoD

Lynn Streeter, Joseph Psotka, Darrell Laham, and Don MacCuish

Abstract

The Intelligent Essay Assessor is commercial software that grades essays as accurately as skilled human graders. It was used to critique senior officers' papers in both the Army's and Air Force's Command and General Staff Colleges. The Army Research Institute has supported development of this software, which understands the meaning of written essays. Automatic essay scoring is well-suited to distance learning environments, and for faculty training and calibration. Because the essay feedback is returned in seconds and can indicate which sections should be rewritten, students can make significant revisions before submitting their final product. This tutorial facility could be exploited in many military courses.

In the Army's Combined Arms and Services Staff School (CAS3), Military Writing assignment memos were graded by both the instructors and a subset by recently retired instructors. The results showed that human-to-human reliabilities (Leavenworth graders-to-retired instructors) were identical to the computer-to-Leavenworth graders reliabilities for the overall grade. In addition, the essay grading software was enhanced to supply written tutorial feedback similar to comments given by instructors, including (1) format checking, (2) section critiquing (e.g. Background, Purpose, etc.), returning recommendations of sections needing revision, and (3) plagiarism detection.

The Air Command and Staff College project is exploring the effectiveness of automating the grading of the written examination used for the "National & International Security Studies" course for both residents and distance learners. In this trial, the Intelligent Essay Assessor was used to assess longer papers, averaging over 2000 words, and grades were compared to two faculty members' grades. Again, the automated method was as reliable as human graders. Plans are underway to use the automated facility for formative evaluation, which means that students, not faculty, will review the assessment provided by the software, and use that feedback to formulate a better response prior to final submission in a portfolio writing exercise.

Full Paper (PDF)