What is rotary positional embedding (RoPE)?

In this article

1. Before you start
2. Why interviewers ask this
3. Rotary positional embedding from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page sits in the transformer architecture guide, which explains where attention and positions fit into a language model.

Why interviewers ask this

A candidate who says "RoPE adds position information" has repeated the label, not the idea. Rotary embeddings sit inside most recent open language models, so a question on them tests whether you understand how attention knows word order at all, and how a model is stretched to read a longer prompt. The question arrives in several forms: "how does RoPE differ from sinusoidal embeddings", "why is it called relative", and "how do you extend a model's context window".

A weak answer names the method. A strong answer says what is rotated, why the dot product then depends on position only through distance, and why extending context means rescaling positions rather than simply continuing them.

Rotary positional embedding from first principles

In plain words

Rotary position embedding

Rotary position embedding (RoPE) is a position method that turns each two-entry slice of a query or key vector by an angle. The angle is the token's position times a fixed frequency. Only queries and keys are turned, so values stay unchanged.

Relative position

A relative position is the signed distance between two tokens, such as three places before. A method is relative when a pair's score depends on this distance, not on where the pair sits.

Rotation frequency

A rotation frequency is the angle, in radians, that one slice turns per step of position. High-frequency slices turn fast. Low-frequency slices turn slowly.

Position interpolation

Position interpolation is a method that extends the context window of a RoPE model. It scales position numbers down so longer inputs fall inside the trained range, then adds a short period of extra training.

Picture a row of clock hands, one per pair of numbers in the vector. Each word gets hands set to a time that depends on its place. To compare two words, check how far apart their hands are. Move both words along the sentence together and that gap does not change.

The clock is a good picture for the speeds, because the hour, minute and second hands turn at different rates, as the pairs do in the third step. Its limit is that a clock shows an absolute time, while RoPE uses the hands only to compare two words.

In one precise sentence, RoPE turns each pair of entries by position times a frequency, so a query-key dot product depends on position only through the distance between the tokens. A positional encoding in general is any signal that tells attention where tokens are; RoPE is one way to supply it.

Step 1: Split each vector into pairs and rotate each pair

RoPE splits a query or key vector of $d$ entries into $d/2$ pairs and turns each pair like a point on a plane. The original paper writes the rotation frequencies as ${\theta }_i={10000}^{-2(i-1)/d}$ for $i$ from 1 to $d/2$ and applies the rotation to queries and keys (Su et al., 2021, section 3.2.2). A pair at position $m$ with frequency $\theta$ is turned through the angle $m\theta$.

The input is a vector and a position number. With the pair $(1,0)$ and a frequency of 1, position 1 gives $(0.540,0.841)$, position 2 gives $(-0.416,0.909)$ and position 3 gives $(-0.990,0.141)$¹. The length of the pair never changes, because a rotation only turns it. The output is the same vector with each pair turned by an angle that grows with position.

Step 2: The query-key dot product depends on position only through the distance m - n

The dot product of a rotated query at position $m$ and a rotated key at position $n$ equals a quantity that depends on $m-n$, because turning both hands by the same extra amount leaves the angle between them unchanged. The RoFormer paper states this as the way the method builds explicit relative position dependency into self-attention (Su et al., 2021).

When the query pair and the key pair both start as $(1,0)$, the contribution of one pair to the score is $\cos (\theta (m-n))$. In general the contribution is $|q||k|\cos (\theta (m-n)+\phi )$, where $|q|$ and $|k|$ are the lengths of the two pairs and $\phi$ is the angle between them before rotation, which comes from their content. Position still enters only through $m-n$.

In our four-entry example the score is 0.5837 for a query at position 3 and a key at position 1, and exactly the same, 0.5837, for positions 5 and 3, 10 and 8, and 100 and 98¹. The pairs share one distance of 2, so they share one score. The position part of the score therefore reflects how far apart two tokens are, not which absolute slot either occupies.

Step 3: Why different pairs rotate at different speeds

Different pairs rotate at different speeds because a single speed would measure only one scale of distance and would repeat itself. A fast pair turns by a large angle per token, so it separates neighbours well but wraps around after a few tokens and then confuses distances that differ by a full turn. A slow pair turns by a tiny angle, so it changes little between neighbours but still distinguishes tokens that are thousands of places apart.

The frequencies follow a geometric ladder with base 10000, meaning each one is the previous one multiplied by the same fixed factor, so they fall steadily from fast to slow. For a head of 64 entries there are 32 pairs: the first has frequency 1 and a full turn every 6.28 tokens (6.28 is the radian count of a full turn, so at frequency 1 one token is one radian), and the last has a frequency of about 0.000133 and a full turn every 47,117 tokens¹.

The same idea appears in the sinusoidal scheme of Vaswani et al. (2017), which RoFormer follows when it picks its frequencies. The chart below uses the first pair (frequency 1) and the slow pair of a four-entry head (frequency 0.01), not the 64-entry ends: the frequency-1 pair swings from 1 to −0.99 within three tokens and is back at 0.96 at six, while the frequency-0.01 pair is still above 0.99 at ten.

Step 4: Position interpolation squeezes positions to extend the context

A RoPE model trained on short inputs cannot simply be fed longer ones (its context window, the number of tokens it reads at once, was fixed in training), because angles beyond the trained range produce score patterns it never saw.

Chen et al. (2023) show that extrapolating past the trained length can give very high attention scores that disrupt self-attention, and propose position interpolation: scale the position indices down so that the longer input lands inside the original range, then fine-tune briefly, meaning continue training for a short time on the longer inputs. Using a model outside the range it saw in training is called extrapolation. Their abstract reports models extended to 32,768 tokens with fine-tuning of about 1,000 steps or fewer, on LLaMA models from 7 to 65 billion parameters.

For a model trained to 2,048 tokens and a target of 8,192, the scale is 0.25, so position 8,191 is treated as position 2,047.75¹. The cost is that neighbouring tokens now differ by a quarter of the old angle, which is why the short extra training is needed. The input is a longer prompt; the output is positions that stay inside the range the model knows.

1 · Pairs and rotation2 · Score uses m - n3 · Frequency ladder4 · InterpolationReplay

Step-by-step description

1. A d-entry vector becomes d/2 pairs; each turns by m x its frequency.
2. The position part of the score depends only on the distance m - n.
3. Frequencies run from 1 down to 0.000133 and feed the rotation.
4. Positions x 0.25 map 8,191 to 2,047.75.

• Each pair at position m turns through the angle m times its frequency.
• The position part of the dot product of a rotated query and key depends only on m minus n.
• Frequencies run from 1 down to about 0.000133 in a 64-entry head.
• Scaling positions by 0.25 treats position 8,191 as position 2,047.75.

Worked example

The setup is a head of four entries, so two pairs, with query and key both equal to $(1,0,1,0)$ and the frequencies from the RoFormer formula with base 10000: 1 and 0.01. These inputs are illustrative. The score is the sum over the two pairs of $\cos (\theta (m-n))$, which for a distance of 2 is $\cos 2+\cos 0.02=-0.416+0.9998=0.5837$¹. The last row has a different distance, so its score differs.

Text description of the diagram

• The frequency-1 pair swings from 1 to −0.99 within three tokens and is back at 0.96 at six.
• At a distance of 10 the fast pair contributes −0.839 and the slow pair 0.995.
• At a distance of 100 the fast pair returns to 0.862 while the slow pair has fallen to 0.540.

The numbers show the two roles. Between distances 0 and 10 (the chart runs to 12) the fast pair goes 1, 0.540, −0.416, −0.990, −0.654, 0.284, 0.960, 0.754, −0.146, −0.911, −0.839 while the slow pair only drifts from 1 to 0.995; at a distance of 100 the fast pair returns to 0.862 while the slow pair has fallen to 0.540¹. A model reads the pattern across all pairs, much as a number is read from several digits of different place value.

Trade-offs and edge cases

RoPE trades a simple input-side addition for a rotation inside every attention layer, and the three methods in use differ in where position enters.

Text description of the diagram

• Sinusoidal embeddings are added to the token embeddings at the input.
• RoPE rotates queries and keys at each layer.
• ALiBi adds a distance penalty to the query-key scores.

The LLaMA paper removes absolute positional embeddings and adds RoPE at each layer of the network (Touvron et al., 2023). The paper on Attention with Linear Biases (ALiBi) takes a different route: it adds a penalty proportional to distance to the query-key scores, with no position embedding at all, and reports that a 1.3 billion parameter model trained on inputs of length 1,024 extrapolates to 2,048 with the same perplexity as a sinusoidal model trained on 2,048 (Press et al., 2021). Reported practice differs on which method extends best, so state the source when you compare them. Perplexity, the score for how well a model predicts held-out text (lower is better), is the measure that paper uses.

Four edge cases matter. First, RoPE needs an even head dimension, because entries are paired. Second, the rotation is applied to queries and keys when they are computed, so a key-value (KV) cache stores keys that are already rotated, and a change of the scaling needs the cached keys recomputed. Third, fast attention kernels such as FlashAttention take the rotated queries and keys as ordinary inputs, so RoPE composes with them without change (a kernel is the low-level routine that runs the attention arithmetic on the chip).

Fourth, position interpolation is not the only way to extend context, and 10000 is RoFormer's default, not a law. Code Llama raised the base period from 10,000 to 1,000,000 for fine-tuning on long inputs (Rozière et al., 2023, arXiv 2308.12950, 24 August 2023). The Llama 3 report raised the RoPE base frequency to 500,000 and says an earlier study found that value effective for contexts up to 32,768 tokens (Llama Team, 2024, arXiv 2407.21783, 31 July 2024).

YaRN is another method for extending a RoPE model's window, which its authors say needs 10 times fewer tokens and 2.5 times fewer training steps than earlier methods (Peng et al., 2023, arXiv 2309.00071, 31 August 2023). Interviewers often ask which of these you would pick, and the honest answer names the trade-off and the source.

How to answer it in an interview

The spoken structure is: say what is rotated, say why the score depends on distance, then the frequency ladder, then the context-extension trade-off.

Common mistakes

• "RoPE adds a position vector to the embedding" (wrong). That describes the sinusoidal scheme at the input; RoPE rotates queries and keys inside attention (right: say it rotates rather than adds).
• "RoPE rotates the value vectors too" (wrong). Only queries and keys are rotated, so the values carry content without position (right).
• "All pairs rotate at the same speed" (wrong). The frequencies form a geometric ladder, which is what lets one head measure short and long distances (right).
• "A RoPE model can read any length because it is relative" (wrong). Angles beyond the trained range produce unfamiliar scores, which is why position interpolation exists (right).
• "RoPE is the only way to get relative position" (wrong). ALiBi gets distance dependence by biasing the scores instead (right).

Follow-up questions

How does rotary positional embedding work?

It rotates each pair of query and key entries by position times a frequency, so the position part of the attention score depends on the distance between tokens. RoFormer introduced it and the LLaMA models use it at each layer.

What is the difference between rotary embedding and positional embedding?

Positional embedding is the general idea of telling attention where tokens are; rotary embedding is one method. The sinusoidal method adds a position vector to the input once, while rotary embedding rotates queries and keys inside every attention layer.

What is the rotary positional embedding formula?

For a pair at position $m$ with frequency ${\theta }_i={10000}^{-2(i-1)/d}$, the pair is turned by the angle $m{\theta }_i$, using the cosine and sine of that angle (Su et al., 2021, section 3.2.2). The score contribution is then $|q||k|\cos ({\theta }_i(m-n)+\phi )$, as in step 2.

What is partial rotary positional embedding?

It is a variant that rotates only some of the entries of each head and leaves the rest unrotated. Implementations of this exist, so check the model's configuration before assuming that all entries rotate. In the standard form above every entry is rotated, in pairs, so a head of 64 entries has 32 rotating pairs.

What if the head has 128 entries instead of 64?

The head then has 64 pairs, the frequencies reach about 0.000115 and the slowest pair takes about 54,410 tokens for a full turn¹, so the ladder stretches to longer distances.

Why can a RoPE model not simply read a longer context than it was trained on?

Because the angles for positions beyond the training range were never seen, and Chen et al. report that extrapolating this way can produce attention scores high enough to disrupt the model. Rescaling positions into the trained range with brief extra training avoids it, and raising the base (Code Llama, Llama 3) or YaRN are other approaches; see large language model (LLM) inference for how long contexts cost memory.

Sources

1. RoFormer: Enhanced Transformer with Rotary Position Embedding (arXiv 2104.09864 v5) — arXiv (Su et al.), 8 Nov 2023
2. LLaMA: Open and Efficient Foundation Language Models (arXiv 2302.13971) — arXiv (Touvron et al.), 27 Feb 2023
3. Extending Context Window of Large Language Models via Positional Interpolation (arXiv 2306.15595) — arXiv (Chen et al.), 27 Jun 2023
4. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (arXiv 2108.12409) — arXiv (Press, Smith, Lewis), 27 Aug 2021
5. Code Llama: Open Foundation Models for Code (arXiv 2308.12950) — arXiv (Rozière et al.), 24 Aug 2023
6. YaRN: Efficient Context Window Extension of Large Language Models (arXiv 2309.00071) — arXiv (Peng et al.), 31 Aug 2023
7. The Llama 3 Herd of Models (arXiv 2407.21783) — arXiv (Llama Team, Meta), 31 Jul 2024
8. Attention Is All You Need (arXiv 1706.03762) — arXiv (Vaswani et al.), 12 Jun 2017