Accepted at EMNLP 2026

Fractional Rotation, Full Potential?

How much of an attention head actually needs rotary position encoding? Our large-scale study finds that a small fraction can be enough.

Read the paper Explore the code

Mohammad Aflah Khan¹ · Krishna P. Gummadi¹ · Manish Gupta² · Abhilasha Ravichander¹

¹ Max Planck Institute for Software Systems ² Microsoft, Hyderabad

64-d attention head interactive view

RoPE fraction 10%

Token position 24

6 of 64 dimensions rotate · 3 frequency pairs Dims 0–1 · highest frequency

Select a dimension pair to inspect its frequency and rotation.

Key observation

Notice how the first few dimensions on the left (high frequency) spin rapidly with each step in position. In contrast, the dimensions on the right (low frequency) barely move.

This allows the model to learn both short-range and long-range relationships simultaneously.

θ_i = 10000^{−2i / d_rope} Interactive RoPE Visualization

≈10%can match full RoPE convergence

up to 10×smaller RoPE cache at extreme context

1B → 8Bconsistent trends across model scales

100Btraining tokens in the main settings

The question

RoPE is usually treated as all or nothing. It doesn’t have to be.

Rotary Positional Embeddings encode token order by rotating query and key dimensions. Model families disagree on how many dimensions to rotate, from a quarter to all of them, yet the choice has rarely been studied directly.

We isolate that choice and test RoPE fractions from NoPE to full RoPE across architectures, scales, datasets, and sequence lengths.

Choices in existing models

The implementation is public. The rationale often isn’t.

Open model families span a wide range, from rotating one quarter of each head to rotating every dimension. Formal ablations or detailed explanations for the fraction are rare.

25%40%50%100%

Model family	RoPE fraction	Rationale in the literature
GPT-J · GPT-NeoX · Pythia Early partial-RoPE lineage	25%	Limited evidence GPT-NeoX reported a small-scale 25% trade-off; later adopters largely inherited the setting.
Phi-2 Microsoft	40%	Not detailed No systematic fraction ablation is reported alongside the configuration.
Nemotron-4-340B NVIDIA	50%	Not detailed The configuration exposes the choice, but not a systematic justification for the fraction.
GLM-4.5 Partial RoPE configuration	50%	Not detailed The model also uses 50% Partial RoPE, without a detailed rationale for selecting that fraction.
LLaMA family · most Qwen models Full-RoPE convention	100%	Default convention Full rotation is standard, while fraction-specific justification is rarely stated.
Qwen3-Next A recent shift back to partial RoPE	25%	Explicit claim The team links the choice to improved extrapolation at longer sequence lengths.

Representative open-weight models. Configuration names differ across training frameworks.

What we found

Three results to remember.

Across the study, a surprisingly small amount of positional rotation carried most of the benefit.

A small fraction is enough

Configurations around 10% or higher track the convergence and final loss of full RoPE in the tested settings.

The clearest performance boundary appeared below 10%.

Memory scales with the fraction

Rotating fewer dimensions shrinks the sine/cosine cache proportionally, up to an order of magnitude at 10% RoPE.

This becomes meaningful mainly at extreme context lengths.

Minimal position signals stabilize

NoPE can produce unrecoverable loss spikes at scale. Even minimal RoPE or QK-Norm mitigates that instability.

Partial RoPE is the more direct positional solution.

Study landscape

One architectural choice, tested from multiple angles.

We pretrained decoder-only models from scratch and varied only the fraction of hidden dimensions receiving RoPE. The pattern held across the axes below.

Architecture Sequential attention Parallel attention

Scale LLaMA-like 1B LLaMA-like 8B Pythia 1B

Sequence 1K 2K 4K 8K

Data FineWeb FineWeb-Edu

RoPE fraction 0% minimal 10% 25–100%

The important caveat

The memory win matters most when context becomes extreme.

At short and moderately long sequence lengths, the RoPE cache is usually not a prominent memory constraint. At million-token scales and beyond, its linear growth turns partial RoPE into a practical design lever, especially when caches are replicated across accelerators.

RoPE cache VRAM

Small at ordinary contexts. Material at extreme scale.

RoPE Cache Size Calculation S_cache = L_max × (D_head × f_RoPE) × P_bytes

The plot defaults to a head dimension of 256 and uses FP32 storage. It shows raw sine/cosine cache storage and excludes memory fragmentation.

256-d head · FP32 · per cache copy

Cache size grows linearly with sequence length

Head dimension 256

Full RoPE95.4 GiB

50% RoPE47.7 GiB

10% RoPE9.5 GiB

At 100M tokens, reducing RoPE from 100% to 10% saves about 85.8 GiB for every replicated cache copy.

Design guidance

A pragmatic starting point.

Start near 10%

Use it as a strong empirical baseline, not a universal constant.

Validate your setting

Check the interaction with head dimension, context length, and block design.

Be careful with NoPE

At scale, retain a minimal positional signal or add QK-Norm for stability.

Go deeper

Paper, code, and artifacts.

Read the full methodology, inspect the training runs, explore released models, or reproduce the training and evaluation pipeline.

Paper arXiv Code Weights & Biases Hugging Face collection

BibTeX

@misc{khan2026fractionalrotationpotentialinvestigating,
  title         = {Fractional Rotation, Full Potential?
                   Investigating Performance and Convergence
                   of Partial RoPE},
  author        = {Mohammad Aflah Khan and Krishna P. Gummadi
                   and Manish Gupta and Abhilasha Ravichander},
  year          = {2026},
  eprint        = {2603.11611},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG}
}