Disentangled Attention in DeBERTa: Separating Content and Position

DeBERTa's disentangled attention keeps token content and token position in separate representations. Standard Transformers usually add a position embedding to a token embedding before computing attention. Once added, the two sources of information are mixed.

For query position $i$ and key position $j$, DeBERTa builds its attention score from three interactions:

Here $c$ means content, $r$ means relative position, and $\delta(i,j)$ converts the distance between tokens into a bounded position index. Because the raw score adds three terms, DeBERTa scales it by $1/\sqrt{3d}$ before softmax, where $d$ is the vector dimension in this single-head expression.

There is deliberately no position-to-position term in this formulation. The model asks how the content at one token relates to another token's content and relative location.

Why separate them?

The meaning of a word depends on both what surrounds it and where those words occur relative to it. Keeping content and position separate lets the model learn asymmetric relations such as “the subject is three positions before this verb” without baking absolute position directly into every token vector.

DeBERTa adds absolute position information later in its enhanced mask decoder, where it is useful for predicting a masked token. That is separate from the relative-position interactions used inside the encoder.

What is easy to misunderstand

• Relative offsets must be clipped or bucketed for long sequences.
• The content-to-position and position-to-content lookups use opposite offset directions; mixing their signs silently changes the mechanism.
• The extra score terms require more work than ordinary content-only attention.
• Padding and attention masks must be applied after all score components are combined.

Primary source

• Pengcheng He et al., DeBERTa: Decoding-enhanced BERT with Disentangled Attention, 2020/2021 revision.