Skip to content
ml-recall

#feed-forward

1 question from Attention & Transformers

Timed test

1 question · 1 min · score and explanations at the end

Take a test
Q 1 / 1Score 0 / 0

Question 1: Short answer

mediumAI-checkedAttention & Transformers

The feed-forward module of each GPT-2 small block (embedding size 768). What does the last line print?

emb_dim = 768
ff = nn.Sequential(
    nn.Linear(emb_dim, 4 * emb_dim),
    nn.GELU(approximate="tanh"),
    nn.Linear(4 * emb_dim, emb_dim),
)
print(sum(p.numel() for p in ff.parameters()))

an integer