←
Benjamin Bossan134 posts
Benjamin Bossan avatar

Benjamin Bossan

@BenjaminBossan
◷ Joined March 2015▧ 25 Photos and videos
Lysandre avatar
⟳ Benjamin Bossan retweeted
Lysandre@LysandreJik·Oct 28

evil qwen got us playing with neurons in peft/transformers, and I got to say I really feel like a brain surgeon tweak one neuron AND THE MODEL STARTS SCREAMING IN ALL CAPS tweak another one and it behaves like an oracle; apply more pressure, it talks like a god. tweak both?

Post attachment
Benjamin Bossan avatar
Benjamin Bossan@BenjaminBossan·Sep 16

There is of course much more, including performance improvements, better handling and checks of multiple adapters, etc. Again, check the release notes for details on all of this and let us know what you think. github.com/huggingface/peft/…

Benjamin Bossan avatar
Benjamin Bossan@BenjaminBossan·Sep 16

ShadowPEFT trains a small shadow network (which can be a pretrained model) alongside the frozen base model. It’s a good choice when you care about downstream or OOD performance or when you want the small model on the edge and the frozen base in the cloud. Added by @xmlee97.

Benjamin Bossan avatar
Benjamin Bossan@BenjaminBossan·Sep 16

Super-Tuning uses calibration-free magnitude scoring to allocate a sparse tuning budget. It also allows to add LoRA on top of this sparse update. It is a good fit for exploring sparsity alongside higher-rank adaptation. Also contributed by @smellslikeml.

Benjamin Bossan avatar
Benjamin Bossan@BenjaminBossan·Sep 16

The new LoRA variant KaSA aims at identifying knowledge aware singular values from the base weight. It then removes noisy knowledge from those weights and specifically trains the singular values most relevant to the task. Added by Robbie Walmsley and Bogeum Kim.

Benjamin Bossan avatar
Benjamin Bossan@BenjaminBossan·Sep 16

@smellslikeml added the Riemannian-preconditioned LoRA optimizer. It is a new LoRA-specific optimizer similar to LoRA-FA or LoRA+ that improves feature learning and does not require tuning of learning rates for LoRA's A and B matrices separately.

Benjamin Bossan avatar
Benjamin Bossan@BenjaminBossan·Jul 28

Of course, we also added a lot of new PEFT methods: - HiRA - GLoRA - BEFT - MonteCLoRA - VeLoRA - Uni-LoRA - FRoD - MiCA - DEFT - KappaTune Check out the release notes for more details: github.com/huggingface/peft/…

Benjamin Bossan avatar
Benjamin Bossan@BenjaminBossan·Jul 28

Finally, to make it easier for you to find the best PEFT method for *your use case*, we created the "PEFT Shop". This @Gradio space allows you to browse the PEFT methods according to their metrics and capabilities and to easily compare them: huggingface.co/spaces/peft-i…

Benjamin Bossan avatar
Benjamin Bossan@BenjaminBossan·Jul 28

Determining which PEFT method is really working the best is difficult. To shed more light on this, we extended our existing PEFT benchmark with a new image generation task. Click on this link and then choose "image-gen" as the task to see the results: huggingface.co/spaces/peft-i…

Post attachment
Benjamin Bossan avatar
Benjamin Bossan@BenjaminBossan·Jul 28

PEFT v0.20.0 was just released and as always, it contains a ton of new additions that are worth checking out 🤗. Besides many new PEFT methods, we added new image benchmarks to compare the methods in a fair manner and we made the docs more useful. Check the thread for details.

Post attachment
Benjamin Bossan avatar
Benjamin Bossan@BenjaminBossan·Jun 18

When it comes to fine-tuning, LoRA is dominating. But is it the best option? We investigate this in our latest blog post on HF. You'll learn which PEFT techniques work best on our benchmarks and how to choose the PEFT technique that is best for your use case.

Post attachment