Glossary
Explore the latest content and insights.
Squared ReLU: The Tiny Activation Change Used by Primer
Squared ReLU computes the square of a rectified activation. This note explains why Primer used it, how it changes gradients, and when not to adopt it blindly.
Mix-FFN in SegFormer: Adding Local Context to Transformer MLPs
SegFormer Mix-FFN inserts a depthwise 3×3 convolution between two feed-forward projections, giving image tokens local spatial context without explicit positional embeddings.
LayerScale: A Small Change That Stabilizes Deep Vision Transformers
LayerScale multiplies each Transformer residual branch by a learned per-channel gain initialized near zero, helping very deep image Transformers begin close to an identity mapping.
Varifocal Loss: Learning IoU-Aware Classification Scores for Detection
Varifocal Loss trains dense detectors to predict a score that combines object confidence with localization quality, while focusing training on difficult negative examples.
Disentangled Attention in DeBERTa: Separating Content and Position
DeBERTa separates token content from relative position, then combines content-to-content, content-to-position, and position-to-content attention scores.
ReGLU, GEGLU, and SwiGLU: Gated Feed-Forward Layers in Transformers
ReGLU, GEGLU, and SwiGLU multiply one linear projection by an activated gate, giving Transformer feed-forward layers a learned control path.
Ghost Module and Ghost Bottleneck: Cheap Features for Mobile CNNs
GhostNet generates a small set of intrinsic feature maps with a normal convolution, then expands them using cheap depthwise operations. Here is how its module and bottleneck work.
Conditional Batch Normalization: Modulating Features with Side Information
Conditional batch normalization predicts the affine scale and bias from a conditioning input, allowing language, class labels, or another signal to modulate visual features.
Lovász-Softmax Loss: Optimizing IoU for Semantic Segmentation
Lovász-Softmax is a differentiable surrogate for the Jaccard/IoU loss. It sorts pixel errors and weights them by the discrete change they cause in IoU.
CBHG Explained: Convolution Banks, Highway Layers, and a Bidirectional GRU
CBHG is the sequence-processing block used in the original Tacotron: a 1D convolution bank, highway network, and bidirectional GRU working at different context scales.
VGG Perceptual Loss: Comparing Features Instead of Pixels
VGG perceptual loss compares generated and target images in a frozen feature space instead of requiring every output pixel to match.