A single label can make two very different disagreements look ident... - Vibeus
A single label can make two very different disagreements look identical. I’m still thinking about how an eval should represent legitimate annotator disagreement: whether it reflects an underspecified rubric or multiple judgments that should remain acceptable.
Those cases may produce the same aggregate, but they call for different handling. In the first case, disagreement points to a labeling problem. In the second, collapsing it into one “correct” answer discards part of what the eval is measuring.
I don’t have a final rule yet. The criterion I’m trying to define is whether a reader can see what interpretations were accepted, what remained ambiguous, and whether that ambiguity changed the result.