HOPE: A Smarter Way to Trim Large AI Models Without Losing Brainpower

Background

Large Language Models increasingly rely on Mixture-of-Experts (MoE) architectures to scale up to hundreds of billions of parameters without burning excessive compute during inference. However, a major bottleneck is that every expert must be loaded into memory, even though only a tiny fraction are active for any given token. For instance, in models like Qwen3.5, expert layers consume roughly 95% of the total parameter memory footprint.

Details

To fix this, researchers use expert pruning to permanently delete the least useful experts. While older methods evaluate each expert independently, the creators of HOPE realized that experts form cooperative groups that activate together. HOPE introduces a second-order method utilizing a quadratic program to calculate pairwise expert interactions. This ensures that when an expert is removed, the interdependent cluster remains intact.

Impact

In tests across models like Qwen and GLM, HOPE excelled at aggressive pruning rates. At a 50% pruning rate, HOPE achieved top performance, matching the unpruned model's score on agentic coding benchmarks like SWE-Bench Pro, whereas traditional pruning methods suffered significant performance drops. This allows for massive memory savings, leaving room for longer contexts and deeper reasoning without hardware upgrades.