AdaFisher – an adaptive second-order optimizer that leverages a block-diagonal approximation to the Fisher information matrix for adaptive gradient preconditioning.

In our proposed concept, we use an Inception layer after a convolutional layer with a kernel size of 1, and then, after performing batch normalization and the SILU activation function, we concatenate the output of this layer with the input of the bottleneck and pass it to the next layer. Moreover, in the other branch of C3-AMP, after performing a convolutional layer, a max pooling layer is employed, and then after concatenating the output of this layer with the output of bottleneck series, another convolutional layer with a kernel size of 1 is utilized.

C3-AMP

Inception-YOLO: Computational cost and accuracy improvement of the YOLOv5 model based on employing modified CSP, SPPF, and inception modules

AdaFisher

AdaFisher: Adaptive Second Order Optimization via Fisher Information

Cyclical log annealing make harsh restarts by creating learning rate spikes to escape a local minimum on weight bias fields while retaining the capability to hang onto a low learning rate unlike cosine annealing, which opens up a discussion of if more greedy stochastic optimization schemes could be viable without as high of a risk of overfitting in late stage training. Though log annealing works similarly to cosine annealing, it's properties are different, as it's base learning rate is much more prevalent and much higher, though this can be adjusted by tweaking minimum decay and restart interval multipliers.

Source	AdaFisher: Adaptive Second Order Optimization via Fisher Information
Year
Data Source	Papers with Code archive (CC BY-SA)