LLM Response Quality Classification

Research question

Can a text model tell the difference between a clear winner and a genuinely ambiguous comparison?

The main data source is the public Arena preference export. Each record contains a prompt, two responses, and a human vote. Instead of collapsing every vote into a binary winner, the pipeline keeps model A, model B, tie, and tie (bothbad) as separate outcomes.

Evaluation design

The split is part of the model

1. A

   Start with the label

   The task has four outcomes: model A wins, model B wins, tie, or both responses are bad. Keeping the two tie outcomes matters because a winner-only classifier can hide disagreement.

2. B

   Keep prompts together

   The development folds are grouped by a normalized question so a near-duplicate prompt does not appear in both training and validation.

3. C

   Separate development from test

   Model selection uses the chronological train pool. A separate 27,127-row temporal split is frozen for the formal baseline.

4. D

   Compare simple models first

   Word and character TF-IDF provide interpretable references before introducing soft targets or a neural scorer.

5. E

   Model disagreement explicitly

   Soft-label training and pairwise scoring are evaluated with tie recall, Macro-F1, calibration, and error slices rather than accuracy alone.

6. F

   Keep the encoder result honest

   The pretrained pairwise encoder is still a development experiment. More rows did not automatically produce a better Macro-F1.

Model comparison

More complexity did not automatically solve the hard part

The development metric is Macro-F1 because the two tie classes are smaller than the two winner classes. Tie recall is reported separately because a system that never predicts ties can still look acceptable on accuracy.

What changed with soft labels

The best current model trades some winner accuracy for better disagreement coverage

The class-balanced soft-label TF-IDF model reached mean Macro-F1 0.2850 and mean tie recall 0.3385 across three seeds. Its accuracy was lower than the hard-label baseline. That is not a universal improvement; it is a deliberate choice to make ambiguous judgments visible.

The pairwise TF-IDF ranker pushed tie recall to 0.7016, but its Macro-F1 was 0.2674. It shows that the score-difference structure is useful, while the underlying response-quality scorer still needs work.

Pretrained encoder

The encoder is currently an exploratory result

The shared BERT pairwise scorer was run locally with MPS on 2K and 5K rows. The 5K run reached Macro-F1 0.2398 and tie recall 0.6989. It is not selected as the final model. The next fair comparison would fix the seeds, epochs, token budget, and development folds before any new locked evaluation.

Error and slice analysis

The weak slice is creative writing, not the largest language group

These are descriptive word-TF-IDF out-of-fold slices. The sample size is shown because a higher slice accuracy is not automatically more reliable when the slice is small.

Formal locked evaluation

The locked result is reported once and then left alone

The word-TF-IDF baseline was trained on 5,000 sampled train-pool rows and evaluated on all 27,127 frozen temporal rows. This table is a final reference point; it is not used to tune the newer soft-label or encoder models.