Parameter tuning method for neural network models with adaptive activation functions.

A **Dilated Causal Convolution** is a [causal convolution](https://paperswithcode.com/method/causal-convolution) where the filter is applied over an area larger than its length by skipping input values with a certain step. A dilated causal [convolution](https://paperswithcode.com/method/convolution) effectively allows the network to have very large receptive fields with just a few layers.

Dilated Causal Convolution

WaveNet: A Generative Model for Raw Audio

DSPT

Adaptive hybrid activation function for deep neural networks

**AdaShift** is a type of adaptive stochastic optimizer that decorrelates $v\_{t}$ and $g\_{t}$ in [Adam](https://paperswithcode.com/method/adam) by temporal shifting, i.e., using temporally shifted gradient $g\_{t−n}$ to calculate $v\_{t}$. The authors argue that an inappropriate correlation between gradient $g\_{t}$ and the second-moment term $v\_{t}$ exists in Adam, which results in a large gradient being likely to have a small step size while a small gradient may have a large step size. The authors argue that such biased step sizes are the fundamental cause of non-convergence of Adam.

The AdaShift updates, based on the idea of temporal independence between gradients, are as follows:

$$ g\_{t} = \nabla{f\_{t}}\left(\theta\_{t}\right) $$

$$ m\_{t} = \sum^{n-1}\_{i=0}\beta^{i}\_{1}g\_{t-i}/\sum^{n-1}\_{i=0}\beta^{i}\_{1} $$

Then for $i=1$ to $M$:

$$ v\_{t}\left[i\right] = \beta\_{2}v\_{t-1}\left[i\right] + \left(1-\beta\_{2}\right)\phi\left(g^{2}\_{t-n}\left[i\right]\right) $$

$$ \theta\_{t}\left[i\right] = \theta\_{t-1}\left[i\right] - \alpha\_{t}/\sqrt{v\_{t}\left[i\right]}\cdot{m\_{t}\left[i\right]} $$

Source	Adaptive hybrid activation function for deep neural networks
Year	2000
Data Source	Papers with Code archive (CC BY-SA)