Video-Coder: Reinforcing Agentic Coding for Video Generation

Video-Coder addresses the challenge of precise spatial structure, motion dynamics, and controllable scene composition in video generation by introducing a code-driven agent. The agent autonomously writes and executes code to construct a dynamic 3D scene (Draft Video) that guides downstream video generators. To enable this paradigm, the authors designed a dedicated data pipeline, curated two datasets (Video-Coder-SFT-6k and Video-Coder-RL-2k), and developed the Video-Coder Harness, an interactive environment for code execution, 3D rendering, and feedback. The model is trained using supervised fine-tuning followed by harness-native reinforcement learning with dual reward feedback and an annealing strategy that balances execution and video rewards. Experiments indicate significant performance improvements on benchmarks such as PhyGenBench and VBench-2.0. The code, data, and models are fully open-sourced, though the paper is currently a preprint.