Judging Answer Strings
Each question independently judged by 3 assessors
- assessors’ opinions as to correct answers differ
- formed high-quality judgment set
- adjudicator examined all strings in conflict
- majority opinion taken as answer unless inconsistent across strings or rules not followed
- provided data for measuring effect of differences on final scores