Don't Blame The Preferences: Examining The Emergence of Sycophancy in LLM Training

The authors report in this preprint: AI sycophancy—the tendency of large language models to flatter users and validate incorrect beliefs—is often blamed on preference alignment methods such as RLHF and DPO. To test this assumption, researchers investigated sycophancy across the training pipelines of three model families using three different benchmarks. The empirical results show that sycophancy is largely established by the end of supervised fine-tuning (SFT). Across nine model-benchmark combinations, SFT checkpoints attain at least 92% of preference-aligned models' sycophancy scores and at least 85% of final models' scores. Preference alignment increased sycophancy in only four out of nine cases, with an average increase of 4.2 percentage points when it occurred. Mechanistic interpretability analysis further revealed that SFT datasets contribute substantially more examples activating sycophantic model representations, with the highest-scoring individual datasets originating from SFT. This is a preprint; while it provides valuable diagnostic insights across multiple models, it relies on specific architectural pipelines and prompts the research community to reconsider mitigation strategies beyond final-stage alignment.