Gravity is a kinematic approach to optimization based on gradients.

**DSelect-k** is a continuously differentiable and sparse gate for Mixture-of-experts (MoE), based on a novel binary encoding formulation. Given a user-specified parameter $k$, the gate selects at most $k$ out of the $n$ experts. The gate can be trained using first-order methods, such as stochastic gradient descent, and offers explicit control over the number of experts to select. This explicit control over sparsity leads to a cardinality-constrained optimization problem, which is computationally challenging. To circumvent this challenge, the authors use a unconstrained reformulation that is equivalent to the original problem. The reformulated problem uses a binary encoding scheme to implicitly enforce the cardinality constraint. By carefully smoothing the binary encoding variables, the reformulated problem can be effectively optimized using first-order methods such as [SGD](https://paperswithcode.com/method/sgd).

The motivation for this method is that  existing sparse gates, such as Top-k, are not smooth. The lack of smoothness can lead to convergence and statistical performance issues when training with gradient-based methods.

DSelect-k

DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning

Gravity

Gravity Optimizer: a Kinematic Approach on Optimization in Deep Learning

**Multi-source Sentiment Generative Adversarial Network** is a multi-source domain adaptation (MDA) method for visual sentiment classification. It is composed of three pipelines, i.e., image reconstruction, image translation, and cycle-reconstruction. To handle data from multiple source domains, it learns to find a unified sentiment latent space where data from both the source and target domains share a similar distribution. This is achieved via cycle consistent adversarial learning in an end-to-end manner. Notably, thanks to the unified sentiment latent space, MSGAN requires a single classification network to handle data from different source domains.

Source	Gravity Optimizer: a Kinematic Approach on Optimization in Deep Learning
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com

Viet-Anh on Software

What is: Gravity?

Viet-Anh on Software