A **Bottleneck Transformer Block** is a block used in [Bottleneck Transformers](https://www.paperswithcode.com/method/bottleneck-transformer) that replaces the spatial 3 × 3 [convolution](https://paperswithcode.com/method/convolution) layer in a [Residual Block](https://paperswithcode.com/method/residual-block) with Multi-Head Self-Attention (MHSA).

**OA-Loss** reduces the domain gap between the original and augmented images. OA-Loss enables the model to learn semantic relations of foreground and background instances from multi-domain.

OA-Loss

Object-Aware Domain Generalization for Object Detection

Bottleneck Transformer Block

Bottleneck Transformers for Visual Recognition

**MPNet** is a pre-training method for language models that combines masked language modeling (MLM) and permuted language modeling (PLM) in one view. It takes the dependency among the predicted tokens into consideration through permuted language modeling and thus avoids the issue of [BERT](https://paperswithcode.com/method/bert). On the other hand, it takes position information of all tokens as input to make the model see the position information of all the tokens and thus alleviates the position discrepancy of [XLNet](https://paperswithcode.com/method/xlnet).

The training objective of MPNet is:

$$ \mathbb{E}\_{z\in{\mathcal{Z}\_{n}}} \sum^{n}\_{t=c+1}\log{P}\left(x\_{z\_{t}}\mid{x\_{z\_{<t}}}, M\_{z\_{{>}{c}}}; \theta\right) $$

As can be seen, MPNet conditions on ${x\_{z\_{<t}}}$ (the tokens preceding the current predicted token $x\_{z\_{t}}$) rather than only the non-predicted tokens ${x\_{z\_{<=c}}}$ in MLM; comparing with PLM, MPNet takes more information (i.e., the mask symbol $[M]$ in position $z\_{>c}$) as inputs. Although the objective seems simple, it is challenging to implement the model efficiently. For details, see the paper.

Source	Bottleneck Transformers for Visual Recognition
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com

Viet-Anh on Software

What is: Bottleneck Transformer Block?

Viet-Anh on Software