Walk preference tuning without reward hacking
Design preference data and checks that reduce reward hacking risk.
The shortcut The model learns to win the grader by adding executive-sounding words, not by improving substance. Preference tuning amplifies the distinction your pairs and graders reward. Preference discipline Principle -> pair -> agreement -> hack check DPO can be simple to train, but the quality of the preference signal still decides what behavior is learned. shortcut rank vibes judgment over catchphrases The preferred and rejected answers must differ on the behavior you want to teach. Principle Step 1 Reviewers say the assistant should be more executive-ready. Which principle is trainable? Pair Step 2 You are creating preference pairs. Which…
Sign up free — one personalized lesson every day, matched to your role and goals.
Already have an account? Sign in