Disentangled Attention

Disentangled Attention in DeBERTa: Separating Content and Position

DeBERTa separates token content from relative position, then combines content-to-content, content-to-position, and position-to-content attention scores.