Less is More: Undertraining Experts Improves Model Upcycling

Stefan Horoi, Guy Wolf, Eugene Belilovsky, Gintare Karolina Dziugaite