#knowledge-distillation

Question 1: Select all that apply

Which statements about sequence-level knowledge distillation for language models are true? Select all that apply.

The student is trained on whole output sequences produced by the teacher, e.g. its beam-search outputs1
It can be combined with word-level distillation, which matches the teacher's per-token distributions2
It needs only the teacher's top-1 token at each position, with no sequence-level search3
It requires the teacher to write a chain-of-thought explanation for each example4