The Definitive Fine-Tuning Guide for 3D Generation Models

Background

3D generation has wide applications in robotics, simulation, and entertainment, but standardizing different file formats and representations remains challenging. While TRELLIS.2 improved 3D structure representation, general-purpose models often fail to produce accurate geometry for specific tasks, such as shoes where the base model frequently generates incorrect soles.

Implementation

The authors used a subset of the Google Scanned Objects dataset containing 188 footwear objects. To adapt TRELLIS.2 efficiently without catastrophic forgetting, they applied LoRA adapters exclusively to the sparse-structure flow DiT and the shape flow DiT, while freezing the texture generation stage. Training was executed using Hugging Face Jobs on A100 GPUs or locally via Docker.

Results

Quantitative evaluation on held-out objects showed major improvements. The fine-tuned model achieved a mean F-score of 0.803 compared to the base model's 0.276, reduced the Chamfer distance from 0.174 to 0.033, and increased normal consistency, proving that targeted fine-tuning successfully adapts general 3D models for specific object categories.