Position Bias in LLMs

U-shaped attention bias in transformer models.

The Problem

LLMs exhibit U-shaped attention bias — tokens at beginning and end receive higher attention than middle, regardless of relevance.

Consequences:

• Important info in middle often ignored
• 10-20% performance drop when key context is middle
• "Lost in the middle" problem

Research

"Found in the Middle" (Hsieh et al., ACL 2024 Findings)

Finding: LLMs have U-shaped attention bias across architectures and sizes.

Results:

• Bias exists regardless of model size
• Persists with instruction tuning
• Affects open-source and proprietary models

arXiv | ACL AnthologyarXiv | "Lost in the Middle at Birth: An Exact Theory of Transformer Position Bias" OpenReview

See Also

• Context Management
• Research Overview