Skip to content
ml-recall

#grouped-query-attention

1 question from Fine-tuning & PEFT

Timed test

1 question · 1 min · score and explanations at the end

Take a test
Q 1 / 1Score 0 / 0

Question 1: Short answer

mediumAI-checkedFine-tuning & PEFT

A decoder-only model has 16 layers. Each attention block contains q_proj = nn.Linear(2048, 2048, bias=False), k_proj = nn.Linear(2048, 512, bias=False), v_proj = nn.Linear(2048, 512, bias=False) and o_proj = nn.Linear(2048, 2048, bias=False). You wrap the model with PEFT:

from peft import LoraConfig, get_peft_model

config = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"])
model = get_peft_model(base_model, config)
model.print_trainable_parameters()

How many trainable parameters are reported?

a whole number