#rlaif

Question 1: Single choice

Constitutional AI (Bai et al., 2022) trains a harmless assistant without any human labels of harmfulness; human oversight comes only from a written list of principles. How is the training organized?

Humans first rank answers by harmlessness to train a preference model, and afterwards the principles are used to write a supervised fine-tuning dataset1
A model invents new principles during training, and the policy is fine-tuned to imitate them directly, without any preference model or RL stage2
The model critiques and revises its own answers by the principles and is tuned on the revisions; then AI-labeled pairs train a preference model for RL3
The principles act as a fixed filter at inference time that blocks harmful outputs, so the model's weights are never trained on them at all4

Timed test