MixLoRA is a type of PEFT method which construct a resource-efficient sparse MoE model based on LoRA.

A method that can recover the weights of the pre-fine-tuning model using a few low-rank (LoRA) fine-tuned models. In contrast to previous attacks that attempt to recover pre-fine-tuning capabilities, Spectral DeTuning aims to recover the exact pre-fine-tuning *weights*. Spectral DeTuning can exploit this vulnerability against large-scale models such as a personalized Stable Diffusion and an aligned Mistral.

Spectral DeTuning

Recovering the Pre-Fine-Tuning Weights of Generative Models

MixLoRA

MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts

Kernel density matrices provide a simpler yet effective mechanism for representing joint probability distributions of both continuous and discrete random variables.  This abstraction allows the construction of differentiable models for density estimation, inference, and sampling, and enables their integration into end-to-end deep neural models.

Source	MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts
Year
Data Source	Papers with Code archive (CC BY-SA)