The **Linear Combination of Activations**, or **LinComb**, is a type of activation function that has trainable parameters and uses the linear combination of other activation functions.

$$LinComb(x) = \sum\limits_{i=0}^{n} w_i \mathcal{F}_i(x)$$

**Charformer** is a type of [Transformer](https://paperswithcode.com/methods/category/transformers) model that learns a subword tokenization end-to-end as part of the model. Specifically it uses [GBST](https://paperswithcode.com/method/gradient-based-subword-tokenization) that automatically learns latent subword representations from characters in a data-driven fashion. Following GBST, the soft subword sequence is passed through [Transformer](https://paperswithcode.com/method/transformer) layers.

Charformer

Charformer: Fast Character Transformers via Gradient-based Subword Tokenization

LinComb

Trainable Activations for Image Classification

**BinaryBERT** is a [BERT](https://paperswithcode.com/method/bert)-variant that applies quantization in the form of weight binarization. Specifically, ternary weight splitting is proposed which initializes BinaryBERT by equivalently splitting from a half-sized ternary network. To obtain BinaryBERT, we first train a half-sized [ternary BERT](https://paperswithcode.com/method/ternarybert) model, and then apply a [ternary weight splitting](https://paperswithcode.com/method/ternary-weight-splitting) operator to obtain the latent full-precision and quantized weights as the initialization of the full-sized BinaryBERT. We then fine-tune BinaryBERT for further refinement.

Source	Trainable Activations for Image Classification
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com

Viet-Anh on Software

What is: Linear Combination of Activations?

Viet-Anh on Software