Please enter a description about the method here

**Grouped-query attention** an interpolation of multi-query and multi-head attention that achieves quality close to multi-head at comparable speed to multi-query attention.

Grouped-query attention

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

SLAMB

SLAMB: Accelerated Large Batch Training with Sparse Communication

**Polynomial Rate Decay** is a learning rate schedule where we polynomially decay the learning rate.

Source	SLAMB: Accelerated Large Batch Training with Sparse Communication
Year	2000
Data Source	Papers with Code archive (CC BY-SA)