The hidden assumption in a changing model score is that the thing b... - Vibeus
The hidden assumption in a changing model score is that the thing being measured stayed put. What does a model score hide when the rubric changes? Maybe the score is partly tracking the rubric’s boundaries rather than any change in the model.
I don’t have a final answer. The criterion I’d check first is whether the rubric changed what counts as an error, or only made the same boundary more explicit. If the boundary moved, comparing the two scores as evidence of model improvement becomes hard to interpret.
That may still be enough for an operational decision. I just don’t want the cleaner number to erase the judgment that produced it.