**ResNet-D** is a modification on the [ResNet](https://paperswithcode.com/method/resnet) architecture that utilises an [average pooling](https://paperswithcode.com/method/average-pooling) tweak for downsampling. The motivation is that in the unmodified ResNet, the 1 × 1 [convolution](https://paperswithcode.com/method/convolution) for the downsampling block ignores 3/4 of input feature maps, so this is modified so no information will be ignored

**CANINE** is a pre-trained encoder for language understanding that operates directly on character sequences—without explicit tokenization or vocabulary—and a pre-training strategy with soft inductive biases in place of hard token boundaries. To use its finer-grained input effectively and efficiently, Canine combines downsampling, which reduces the input sequence length, with a deep [transformer](https://paperswithcode.com/method/transformer) stack, which encodes context.

CANINE

CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation

ResNet-D

Bag of Tricks for Image Classification with Convolutional Neural Networks

**AutoTinyBERT** is a an efficient [BERT](https://paperswithcode.com/method/bert) variant found through neural architecture search. Specifically, one-shot learning is used to obtain a big Super Pretrained Language Model (SuperPLM), where the objectives of pre-training or task-agnostic BERT distillation are used.  Then, given a specific latency constraint, an evolutionary algorithm is run on the SuperPLM to search optimal architectures. Finally, we extract the corresponding sub-models based on the optimal architectures and further train these models.

Source	Bag of Tricks for Image Classification with Convolutional Neural Networks
Year	2000
Data Source	Papers with Code archive (CC BY-SA)