#scaled-dot-product-attention

Attention divides the scores by $\sqrt{d_k}$ before the softmax. What is the larger of the two weights printed by the last line? Round to 3 decimals.

s = torch.tensor([8.0, 16.0])      # two attention scores q·k
d_k = 64                           # keys.shape[-1]
print(torch.softmax(s, dim=-1))              # tensor([3.3535e-04, 9.9966e-01])
print(torch.softmax(s / d_k**0.5, dim=-1))