Exploring Weight-Space Programming: Capabilities, Generalization, and Limitations

Weight-Space Programming (WSP) allows reasoning agents to directly inspect and modify neural network parameters instead of relying on standard gradient-based adaptation. This preprint systematically investigates WSP using 16 editing trajectories across seven benchmarks and two model scales (Qwen3.5-0.8B and Qwen3.5-9B). The findings reveal three main insights: first, WSP can achieve substantial performance gains via highly localized interventions—such as improving accuracy by 29.45 percentage points on MATH-500 by modifying only 1,023 parameters. Second, WSP exhibits promising yet uneven cross-task generalization, where a math-edited checkpoint transfers well to some benchmarks (e.g., SciBench and IFEval) but leaves others unchanged. Third, effective editing remains challenging and inconsistent, with substantial variation across trajectories and several attempts retaining original weights due to unreliable validation. The paper provides open-source code and highlights the fundamental challenges of intervention discovery, reliability, and cross-task transfer in parameter-level editing.