RLHF
Reinforcement Learning from Human Feedback — training models to align with human preferences.
RLHF is a multi-stage alignment technique. First, human raters rank model outputs by quality. A reward model is trained on these rankings. Finally, the language model is fine-tuned via RL to maximize the reward signal. RLHF is what makes chat models helpful, harmless, and honest rather than just statistically fluent. Variants include DPO (Direct Preference Optimization) which skips the reward model.
Verwandte Begriffe
The field of ensuring AI systems behave according to human values and intentions.
Anthropic's method of training models to self-critique and revise outputs using a set of principles.
Fine-tuning a model on examples of instructions paired with ideal responses.
The discipline of building AI systems that are reliably beneficial and avoid harmful outputs.