Positional encodings from scratch: sinusoidal and rotary (RoPE), step by step

In this article

1. Before you start
2. Why interviewers ask this
3. Positional encodings from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step in the PyTorch interview questions and machine learning (ML) coding round guide.

Why interviewers ask this

Writing positional encodings by hand shows whether a candidate knows what attention can and cannot see. Candidates report prompts such as implementing the sinusoidal encoding of the original Transformer, implementing rotary position embedding (RoPE), and a longer variant that adds a position offset and a context-extension scaling.

A weak answer pastes the formula and cannot say why the vector is added or what the wavelengths do. A strong answer says that attention is blind to order, shows the table, adds it to the embeddings, and explains why the frequencies form a ladder. A Staff-level answer also explains why rotary encoding gives a relative score from absolute positions, which position a new token gets when a cache is used during generation, and why stretching the context needs care. The steps below build exactly that answer, and the worked example lets you check each number by hand.

Positional encodings from first principles

Positional encoding

A positional encoding is a signal, either a vector added to the token vectors, a rotation applied to them, or a bias on attention scores, that gives self-attention information about token position, which attention does not have on its own.

Sinusoidal positional encoding

A sinusoidal positional encoding is a fixed table with one vector per position, in which each dimension is a sine or cosine wave of the position, with a different wavelength for each pair of dimensions, and no trained weights.

In plain words

Think of a clock. The second hand turns fast, the minute hand slowly, the hour hand slower still. Read all three hands and you know the time exactly. A sinusoidal encoding works the same way. Each pair of dimensions is one hand, and each hand turns at its own steady rate. The analogy breaks in one place: a clock has three hands that repeat every twelve hours, while the model has a whole ladder of hands whose slowest one takes very long to repeat.

In one precise sentence, a positional encoding turns the integer position of a token into numbers that the network can combine with the token's content.

Step 1: Attention cannot see order

Self-attention ignores token order because every score comes from the content of two vectors only. The score of one token pair is a dot product of a query and a key, and the same two tokens give the same score wherever they sit. In attention without a mask, as in an encoder, shuffling the input tokens shuffles the outputs the same way; nothing else changes. The sentences "dog bites man" and "man bites dog" would then look like the same bag of words.

Input: token vectors. Output: scores that carry no position. The causal mask used by decoders (see the causal mask and cross-attention) limits which tokens may look at which, and that alone breaks the symmetry: Haviv et al. (2022) report that transformer language models without positional encodings still learn positional information, plausibly because the mask lets a token infer how many tokens precede it. Explicit encodings remain the standard because they give position directly and reliably, and Vaswani et al. (2017) inject it with a positional encoding. You now know why the rest of the page exists.

Step 2: Sinusoidal encoding gives every position its own vector

Vaswani et al. (2017) use sine and cosine functions of different frequencies. Dimension 2i of the encoding at position pos holds the sine of pos divided by 10000 raised to 2i over d_model, and dimension 2i+1 holds the cosine of the same angle. Each dimension is a sinusoid, and the wavelengths form a geometric progression from 2π to 10000·2π, which is about 62,832.¹

PE(pos, 2i)   = sin(pos / 10000^(2i / d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i / d_model))

import math

def sinusoidal_encoding(num_positions, d_model):
    table = []
    for pos in range(num_positions):
        row = []
        for i in range(d_model // 2):
            angle = pos / 10000 ** (2 * i / d_model)
            row += [math.sin(angle), math.cos(angle)]
        table.append(row)
    return table  # add table[pos] to the embedding of the token at pos

The vector has the same length as the embedding, so the two are summed once at the input. The paper hypothesizes that this lets the model attend by relative position, because for any fixed offset the encoding at pos + k is a linear function of the encoding at pos. The paper's section on positional encoding reports that learned embeddings gave nearly identical results, and chose sinusoids because they may extrapolate to longer sequences.

Step 3: Rotary encoding rotates queries and keys

Rotary encoding places position inside the attention score instead of the input. Su et al. (RoFormer) encode the absolute position with a rotation matrix and, at the same time, build the explicit relative position dependency into self-attention. The vector is cut into pairs of dimensions; pair i is rotated by the angle m·θᵢ, where m is the token position. In the paper's one-based form θᵢ = 10000^(−2(i−1)/d), and a zero-based loop uses 10000^(−2i/d), the same frequency ladder as Step 2.

def rotate(vec, pos, base=10000):
    d = len(vec)
    out = []
    for i in range(d // 2):
        angle = pos * base ** (-2 * i / d)
        x, y = vec[2 * i], vec[2 * i + 1]
        out += [x * math.cos(angle) - y * math.sin(angle),
                x * math.sin(angle) + y * math.cos(angle)]
    return out

Rotate the query at position m and the key at position n, then take their dot product. Each pair is rotated by m·θ and n·θ, so the pair's contribution depends on (m−n)·θ only; the paper's two-dimensional form has the score depend on m−n through a complex factor. Values are not rotated. Rotation also keeps vector length unchanged, so only direction carries position.¹ Unlike Step 2, the rotation happens in every attention layer, on queries and keys; the multi-head attention guide shows where those queries and keys come from.

An interview version works on a whole tensor of shape (batch, heads, sequence, head dimension). It builds the cosine and sine tables once and slices them from offset to offset + seq_len, which is what a cache needs. The sketch below uses NumPy; PyTorch code has the same shape.

import numpy as np

def rope_tables(max_len, d, base=10000):
    inv_freq = base ** (-2.0 * np.arange(d // 2) / d)
    angles = np.outer(np.arange(max_len), inv_freq)  # (max_len, d/2)
    return np.cos(angles), np.sin(angles)

def rope(x, cos, sin, offset=0):
    # x: (batch, heads, seq, head_dim); pairs are (2i, 2i+1)
    n = x.shape[-2]
    c, s = cos[offset:offset + n], sin[offset:offset + n]
    x1, x2 = x[..., 0::2], x[..., 1::2]
    out = np.empty_like(x)
    out[..., 0::2] = x1 * c - x2 * s
    out[..., 1::2] = x1 * s + x2 * c
    return out

On the worked-example query and key, this version gives the same rotated vectors and the same score, −2.5336, as the loop above.¹ Two pair layouts exist. The paper pairs neighbours (2i, 2i+1), as here; many open implementations use the rotate-half layout, which pairs dimension i with i + d/2. Weights trained with one layout give wrong scores under the other, so match the layout when loading pretrained weights.

Step 4: Pass the true position when you generate

The position of a new token is the number of tokens already processed (zero only for the very first token). During generation a model keeps a cache. The key-value (KV) cache is a store of the keys and values of tokens already processed, so that each step handles one new token. A KV cache is called a cache because each stored key is reused at every later step. KV cache is called a cache because each stored key is reused at every later step. Each cached key was rotated by its own position when it was stored, so it is never rotated again. The new query must be rotated with its absolute position.

Use position zero by mistake and the score changes. In the worked example below the correct score for a query at position 7 and a key at position 5 is the same as for positions 5 and 3, while a query wrongly given position 0 gives a different score.¹ A decoding algorithm such as beam search keeps several candidate sequences; with equal-length beams they share one position, so per-sequence counters matter mainly for padded batches. The generative pre-trained transformer (GPT) build guide shows where the table or the rotation sits in a full model.

1 · No order yet2 · Sinusoidal table3 · Rotary rotation4 · True positionReplay

Step-by-step description

1. Attention alone ignores order.
2. Add one vector per position.
3. Rotate queries and keys.
4. Decoding passes the real index.

• Attention alone ignores token order.
• A sinusoidal table gives each position its own vector, added once to the token vectors.
• Rotary encoding rotates queries and keys in every attention layer.
• Generation must pass the true position of the new token.

Worked example

Take d_model = 4 (illustrative), so there are two pairs and the frequencies are 1 and 0.01.¹ The encoding of position 0 is [0, 1, 0, 1], because the sine of zero is zero and the cosine of zero is one. The encoding of position 1 is [0.8415, 0.5403, 0.01, 1.0]: the fast pair moved a full radian, the slow pair barely moved.¹

Text description of the diagram

• The frequencies are 1 and 0.01.
• The encoding of position 0 is 0, 1, 0, 1.
• The encoding of position 1 is 0.8415, 0.5403, 0.01, 1.0.

For the shift property, take position 5 and offset 3. The encoding of position 5 is [−0.9589, 0.2837, 0.05, 0.9988] and of position 8 is [0.9894, −0.1455, 0.0799, 0.9968]. Rotating each pair of the first by its own frequency times 3 gives the second, which the script confirms.¹

For rotary, use an illustrative query [1.0, 0.5, −0.5, 2.0] and key [0.5, 1.0, 1.5, −1.0]. At positions 5 and 3 the rotated query is [0.7631, −0.8171, −0.5993, 1.9725] and the rotated key is [−0.6361, −0.9194, 1.5293, −0.9546]. Multiplying pairwise and adding gives about −2.534, which you can redo with a calculator.¹

Trade-offs and edge cases

Each method puts position in a different place and pays a different price.

Text description of the diagram

• Sinusoidal encodings are added to the embeddings once.
• Learned embeddings have one trained row per position.
• Rotary encodings rotate queries and keys.
• ALiBi biases attention scores by distance.

Length beyond training. A learned table has one trained row per position, so it has no row past its last position. Sinusoids can be computed for any position, but Vaswani et al. (2017) say only that they may extrapolate. Attention with Linear Biases (ALiBi) adds no position vector; it penalizes a query-key score in proportion to their distance, with a fixed, head-specific slope that is not trained.

Press et al. (2021) report that a 1.3 billion parameter model trained on length 1024 extrapolates to 2048 with the perplexity of a sinusoidal model trained on 2048, training 11% faster. With an illustrative slope of 0.5, the penalty is 0, 0.5, 1, 2 and 4 at distances 0, 1, 2, 4 and 8.¹

Stretching rotary context. Chen et al. (2023) report that Position Interpolation extends RoPE-based models up to 32768 tokens with minimal fine-tuning, by linearly down-scaling position indices to the original window instead of extrapolating, which can give catastrophically high attention scores. With illustrative lengths of 2048 trained and 8192 wanted, positions are multiplied by 0.25, so position 8000 becomes 2000.¹ Linear interpolation squeezes every frequency by the same factor, which also blurs the fast pairs that separate neighbouring tokens.

YaRN (yet another RoPE extension method; Peng et al., 2023) scales frequencies unequally: it interpolates the slow, low-frequency pairs, leaves the fast, high-frequency pairs alone, and adds a temperature to the attention logits. A simpler relative, scaling named after the neural tangent kernel (NTK) and so called NTK-aware scaling, raises the base instead of dividing positions; a larger base slows the pairs unevenly, as the base follow-up below explains.

2D inputs. For an image grid, the standard recipe splits the embedding dimension in half, encodes the row index with the sinusoidal formula in one half and the column index in the other, and flattens the grid in row-major order into a token sequence. The masked autoencoder (MAE) of He et al. (2021) uses fixed 2D sine-cosine embeddings of this kind. For row 1 and column 2 at four dimensions that gives [0.8415, 0.5403, 0.9093, −0.4161].¹ Heo et al. (2024) study a 2D extension of rotary encoding for vision transformers, the alternative if the interviewer wants rotary; ask which one is expected. A 2D convolution needs none of this, because its kernel shape fixes neighbors.

The base. The base (10000 in the original paper, an illustrative default here) is a hyperparameter that sets the longest wavelength, not a proven optimum.

How to answer it in an interview

The spoken structure: say attention is blind to order, give the table, give the rotation, then the offset and the trade-off.

Common mistakes

• "The sinusoidal encoding is learned." (wrong) It is a fixed function of position with no trained weights. (right) "It is a fixed table; learned embeddings are a separate option that gave nearly identical results in the original paper."
• "Sinusoids are proven to extrapolate." (wrong) The paper says they may. (right) "They were chosen because they might extrapolate; I would test it."
• "RoPE is added to the embeddings." (wrong) It rotates queries and keys inside each layer. (right) "I rotate q and k, leave v alone, and the score depends on the offset."
• "In decoding I rotate the new query with position zero." (wrong) That changes every score. (right) "I use the absolute position, the length of the cache."
• "The zero-based loop uses the paper's i−1." (wrong) That shifts every frequency. (right) "Zero-based uses 10000^(−2i/d); the one-based paper form uses i−1."

Follow-up questions

How do you implement sinusoidal positional encodings in Python?

Loop over positions and over dimension pairs, put the sine of pos / 10000 ** (2 * i / d_model) in slot 2i and the cosine in slot 2i+1, then add the row to the token embedding. Position 0 gives [0, 1, 0, 1] at four dimensions.¹

How do you implement RoPE with a position offset?

Precompute the cosine and sine tables once for the maximum length, slice rows offset to offset + seq_len (the offset is the number of tokens already in the cache), rotate the new query and key with those rows, and store the rotated key. Never rotate a cached key a second time. The vectorised version matches the scalar loop on the worked-example query and key.¹

How does RoPE context-extension scaling work?

Position Interpolation multiplies position indices by trained length over target length, so every angle stays inside the range seen in training, and then fine-tunes briefly (Chen et al., 2023). Frequency-dependent methods such as YaRN (Peng et al., 2023) interpolate the slow pairs and leave the fast ones alone, and NTK-aware scaling raises the base instead.

How do you implement a 2D positional embedding for attention?

Split the dimensions in half, encode the row in one half and the column in the other with the sinusoidal recipe, and flatten the grid in row-major order, as in the fixed 2D sine-cosine embeddings of He et al. (2021). A 2D rotary version exists too (Heo et al., 2024); say which one you chose.

What changes if you change the base of 10000?

The base sets the longest wavelength; a larger base stretches the ladder so the slowest pair turns more slowly, and a smaller one shrinks it. In rotary encoding a larger base slows every pair except the first, which is why raising the base is one context-extension method.

Why is RoPE applied to queries and keys but not to values?

Only the score needs position. The value carries content that is mixed by the scores, so rotating it would add nothing to the offset dependence, and it would put absolute angles into the output mixture.

Sources

1. Attention Is All You Need (arXiv 1706.03762) — arXiv (Vaswani et al.), 12 Jun 2017
2. RoFormer: Enhanced Transformer with Rotary Position Embedding (arXiv 2104.09864) — arXiv (Su et al.), 20 Apr 2021
3. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (arXiv 2108.12409) — arXiv (Press, Smith, Lewis), 27 Aug 2021
4. Extending Context Window of Large Language Models via Positional Interpolation (arXiv 2306.15595) — arXiv (Chen et al.), 28 Jun 2023
5. Transformer Language Models without Positional Encodings Still Learn Positional Information (arXiv 2203.16634) — arXiv (Haviv et al.), 30 Mar 2022
6. YaRN: Efficient Context Window Extension of Large Language Models (arXiv 2309.00071) — arXiv (Peng et al.), 31 Aug 2023
7. Masked Autoencoders Are Scalable Vision Learners (arXiv 2111.06377) — arXiv (He et al.), 11 Nov 2021
8. Rotary Position Embedding for Vision Transformer (arXiv 2403.13298) — arXiv (Heo et al.), 20 Mar 2024