Can AI Judges Accurately Evaluate Your Personal Writing Style?

According to Apple Machine Learning:

A recent study accepted at the NeurIPS 2026 workshop investigated how effectively large language models can act as judges for personalized writing tasks. The researchers generated texts using four different LLMs across three long-form writing tasks, gathered human preference labels, and compared them against LLM judge annotations.

The findings reveal that an AI judge's performance correlates only weakly with general reasoning benchmarks like MMLU. Interestingly, the strongest foundation models do not most reliably reproduce human evaluations, likely because they possess an exaggerated sensitivity to subtle nuances that everyday human annotators and smaller models simply overlook.

Furthermore, the study highlights a practical method for improving evaluation: paired comparisons. LLM evaluators proved to be significantly more consistent and reliable when asked to choose the better of two responses rather than scoring individual responses independently.