Question 1: Short answer
easyAI-checkedAlignment & ReasoningTo collect preference data faster, InstructGPT shows a labeler responses to one prompt and asks for a full ranking, which is then split into pairwise comparisons for the reward model. For , how many pairwise comparisons does one ranking yield?