**Global-Local Attention** is a type of attention mechanism used in the [ETC](https://paperswithcode.com/method/etc) architecture. ETC receives two separate input sequences: the global input $x^{g} = (x^{g}\_{1}, \dots, x^{g}\_{n\_{g}})$ and the long input $x^{l} = (x^{l}\_{1}, \dots x^{l}\_{n\_{l}})$. Typically, the long input contains the input a [standard Transformer](https://paperswithcode.com/method/transformer) would receive, while the global input contains a much smaller number of auxiliary tokens ($n\_{g}  \ll n\_{l}$). Attention is then split into four separate pieces: global-to-global (g2g), global-tolong (g2l), long-to-global (l2g), and long-to-long (l2l). Attention in the l2l piece (the most computationally expensive piece) is restricted to a fixed radius $r \ll n\_{l}$. To compensate for this limited attention span, the tokens in the global input have unrestricted attention, and thus long input tokens can transfer information to each other through global input tokens. Accordingly, g2g, g2l, and l2g pieces of attention are unrestricted.

Global-Local Attention

ETC: Encoding Long and Structured Inputs in Transformers

ArXiving Before Submission Helps Everyone

**$\epsilon$-Greedy Exploration** is an exploration strategy in reinforcement learning that takes an exploratory action with probability $\epsilon$ and a greedy action with probability $1-\epsilon$. It tackles the exploration-exploitation tradeoff with reinforcement learning algorithms: the desire to explore the state space with the desire to seek an optimal policy. Despite its simplicity, it is still commonly used as an behaviour policy $\pi$ in several state-of-the-art reinforcement learning models.

Image Credit: [Robin van Embden](https://cran.r-project.org/web/packages/contextual/vignettes/sutton_barto.html)

Source	ArXiving Before Submission Helps Everyone
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com

Viet-Anh on Software

What is: Amplifying Sine Unit: An Oscillatory Activation Function for Deep Neural Networks to Recover Nonlinear Oscillations Efficiently?

Viet-Anh on Software