#target-modules

Question 1: Select all that apply

Which statements about where to apply LoRA in a Transformer are supported by the LoRA paper and later work? Select all that apply.

LoRA works only for square matrices, so the rectangular MLP projections cannot be adapted1
On GPT-3 with a fixed budget of trainable parameters, adapting $W_q$ and $W_v$ at $r = 4$ beat adapting $W_q$ alone at $r = 8$ 2
Raising the rank of attention-only LoRA fully makes up for leaving the MLP layers untouched3
QLoRA found that LoRA on all linear layers of each Transformer block is needed to match full fine-tuning4
The original LoRA paper adapts only attention weights and keeps the MLP modules frozen5