A log-time alternative to feedforward layers outperforming both the vanilla feedforward and mixture-of-experts approaches.

**One-Shot Aggregation with an Identity Mapping and eSE** is an image model block that extends [one-shot aggregation](https://paperswithcode.com/method/one-shot-aggregation) with a [residual connection](https://paperswithcode.com/method/residual-connection) and [effective squeeze-and-excitation block](https://paperswithcode.com/method/effective-squeeze-and-excitation-block). It is proposed as part of the [VoVNetV2](https://paperswithcode.com/method/vovnetv2) CNN architecture.

The module adds an identity mapping to the OSA module - the input path is connected to the end of an OSA module that is able to backpropagate the gradients of every OSA module in an end-to-end manner on each stage like a [ResNet](https://paperswithcode.com/method/resnet). Additionally, a [channel attention module](https://paperswithcode.com/method/channel-attention-module) - effective Squeeze-Excitation - is used which is like regular [squeeze-and-excitation](https://paperswithcode.com/method/squeeze-and-excitation-block) but uses only one FC layer with $C$ channels instead of two FCs without a channel dimension reduction, which maintains channel information.

OSA (identity mapping + eSE)

CenterMask : Real-Time Anchor-Free Instance Segmentation

Fast Feedforward Networks

A **Deformable Kernels** is a type of convolutional operator for deformation modeling. DKs learn free-form offsets on kernel coordinates to deform the original kernel space towards specific data modality, rather than recomposing data. This can directly adapt the effective receptive field (ERF) while leaving the receptive field untouched. They can be used as a drop-in replacement of rigid kernels. 

As shown in the Figure, for each input patch, a local DK first generates a group of kernel offsets $\{\Delta \mathcal{k}\}$ from input feature patch using the light-weight generator $\mathcal{G}$ (a 3$\times$3 [convolution](https://paperswithcode.com/method/convolution) of rigid kernel). Given the original kernel weights $\mathcal{W}$ and the offset group $\{\Delta \mathcal{k}\}$, DK samples a new set of kernel $\mathcal{W}'$ using a bilinear sampler $\mathcal{B}$. Finally, DK convolves the input feature map and the sampled kernels to complete the whole computation.

Source	Fast Feedforward Networks
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com

Viet-Anh on Software

What is: Fast Feedforward Networks?

Viet-Anh on Software