Skip to content
ml-recall

#llms-from-scratch

20 questions from Attention & Transformers, Tokenization, LLM Fundamentals & Prompting, Fine-tuning & PEFT, PyTorch

Timed test

20 questions · 20 min · score and explanations at the end

Take a test
Q 1 / 20Score 0 / 0

Question 1: Short answer

easyAI-checkedAttention & Transformers

What is input_embeddings.shape?

vocab_size, output_dim, context_length = 50257, 256, 4
token_embedding_layer = nn.Embedding(vocab_size, output_dim)
pos_embedding_layer = nn.Embedding(context_length, output_dim)

inputs = torch.randint(0, vocab_size, (8, context_length))   # 8 sequences of 4 token IDs
token_embeddings = token_embedding_layer(inputs)
pos_embeddings = pos_embedding_layer(torch.arange(context_length))
input_embeddings = token_embeddings + pos_embeddings

Three dimensions in order, e.g. (5, 6, 7)