The **Normalized Linear Combination of Activations**, or **NormLinComb**, is a type of activation function that has trainable parameters and uses the normalized linear combination of other activation functions.

$$NormLinComb(x) = \frac{\sum\limits_{i=0}^{n} w_i \mathcal{F}_i(x)}{\mid \mid W \mid \mid}$$

[HHO](https://aliasgharheidari.com/HHO.html) is a popular swarm-based, gradient-free optimization algorithm with several active and time-varying phases of exploration and exploitation. This algorithm initially published by the prestigious Journal of Future Generation Computer Systems (FGCS) in 2019, and from the first day, it has gained increasing attention among researchers due to its flexible structure, high performance, and high-quality results. The main logic of the HHO method is designed based on the cooperative behaviour and chasing styles of Harris' hawks in nature called "surprise pounce". Currently, there are many suggestions about how to enhance the functionality of HHO, and there are also several enhanced variants of the HHO in the leading Elsevier and IEEE transaction journals.

From the algorithmic behaviour viewpoint, there are several effective features in HHO :
Escaping energy parameter has a dynamic randomized time-varying nature, which can further improve and harmonize the exploratory and exploitive patterns of HHO. This factor also supports HHO to conduct a smooth transition between exploration and exploitation.
Different exploration mechanisms with respect to the average location of hawks can increase the exploratory trends of HHO throughout initial iterations.
Diverse LF-based patterns with short-length jumps enrich the exploitative behaviours of HHO when directing a local search.
The progressive selection scheme supports search agents to progressively advance their position and only select a better position, which can improve the superiority of solutions and intensification powers of HHO throughout the optimization procedure.
HHO shows a series of searching strategies and then, it selects the best movement step. This feature has also a constructive influence on the exploitation inclinations of HHO.
The randomized jump strength can assist candidate solutions in harmonising the exploration and exploitation leanings.
The application of adaptive and time-varying components allows HHO to handle difficulties of a feature space including local optimal solutions, multi-modality, and deceptive optima.

🔗 The source codes of HHO are publicly available at https://aliasgharheidari.com/HHO.html

Harris Hawks optimization (HHO)

NormLinComb

Trainable Activations for Image Classification

**AdaSqrt** is a stochastic optimization technique that is motivated by the observation that methods like [Adagrad](https://paperswithcode.com/method/adagrad) and [Adam](https://paperswithcode.com/method/adam) can be viewed as relaxations of [Natural Gradient Descent](https://paperswithcode.com/method/natural-gradient-descent).

The updates are performed as follows:

$$ t \leftarrow t + 1 $$

$$ \alpha\_{t} \leftarrow \sqrt{t} $$

$$ g\_{t} \leftarrow \nabla\_{\theta}f\left(\theta\_{t-1}\right) $$

$$ S\_{t} \leftarrow S\_{t-1} + g\_{t}^{2} $$

$$ \theta\_{t+1} \leftarrow \theta\_{t} + \eta\frac{\alpha\_{t}g\_{t}}{S\_{t} + \epsilon} $$

Source	Trainable Activations for Image Classification
Year	2000
Data Source	Papers with Code archive (CC BY-SA)