← Home

Research Engineer · fundamentals screen

Understand it. Then explain it clearly.

An untimed review of 32 fundamentals questions across architecture, optimization, training, distributed systems, inference, and evaluation. Build the intuition, work through an example, then explain it without the reference.

Preparation for broad technical screens—not an official employer question bank. No AI grading.

01 / 32 · Architecture

Attention and the √d scale

What do queries, keys, and values do, and why divide attention scores by √d?

Start with the intuition

A query describes what a token is looking for; keys supply matching features; values supply the information to aggregate. These are learned projections, not hand-assigned semantic roles.

Follow the mechanism

  1. 01

    Project

    Create Q, K, and V from hidden states using learned matrices.

  2. 02

    Match

    Compute scaled query–key dot products and normalize over eligible keys.

  3. 03

    Aggregate

    Use those weights to sum value vectors for each query.

The key equation

Q = XW_Q, K = XW_K, V = XW_V
Attention(Q,K,V) = softmax(QKᵀ / √dₖ + M)V
If independent unit-variance coordinates: Var(q·k) = dₖ

For one head, Q and K have shape [T, dₖ], scores [T, T], and V [T, dᵥ]. M is an additive mask; softmax acts across keys for each query.

Under the simplifying independence assumption, summing dₖ products increases score variance by dₖ. Dividing by √dₖ keeps it near one, reducing dimension-driven softmax saturation. Learned coordinates need not satisfy the assumption exactly: it motivates the scaling rather than proving every attention head has unit score variance.

Work through an example

DERIVED · At dₖ = 64, the unscaled score standard deviation is 8 under these assumptions. Dividing by 8 returns a standard deviation of 1. A softmax row [0.75, 0.25] combines values [2, 0] and [0, 4] into [1.5, 1].

A common interview trap

“Divide by d because there are d dimensions.” Variances add, but standard deviation grows as √d. Also, attention mixes values, not keys.

Turn understanding into an answer

Explain it in your own words

Look away from the lesson and answer the opening question aloud: definition → mechanism → trade-off → example. No timer, recording, or grading.

A follow-up to think through

Does dividing by √d make the attention weights uniform?

Show follow-up reasoning

No. It controls score scale, not equality. Learned similarities can still produce highly concentrated weights; masking also changes the available keys.