Farm3D is a method for single-view 3D reconstruction and controllable 3D synthesis. It learns an articulated object category entirely from “free” virtual supervision from a 2D diffusion-based image generator, such as Stable Diffusion. The framework produces training data for learning a reconstruction network from the ground up, with the diffusion model incorporated as a scoring mechanism to further improve learning. Farm3D yields a monocular reconstruction network capable of generating controllable 3D assets from a single input image, whether real or generated, in a matter of seconds. The method enables the generation of controllable 3D assets from either a real image or an image synthesized using Stable Diffusion, with the ability to adjust lighting, swap textures between models of the same category, and even animate the shape.
⚡Top 5 Farm3D Features:
Monocular 3D reconstruction: Farm3D is capable of generating controllable 3D assets from a single input image, whether real or generated, in a matter of seconds.
Articulated object reconstruction: The method factorises the input image of an object instance into articulated shape, appearance (albedo and diffuse and ambient intensities), viewpoint, and light direction.
Controllable synthesis: Once generated, the 3D assets can be adjusted with lighting, textures, and even animated.
Texture swapping: The method enables the swapping of textures between models of the same category.
New 3D Articulated Animals Dataset: Farm3D introduced a new dataset to directly evaluate the quality of single-view 3D reconstruction of articulated animals.
⚡Top 5 Farm3D Use Cases:
Real-time applications: Farm3D can be used for analysis, including monocular reconstruction, or for synthesis, generating articulated assets for real-time applications such as video games.
Video game development: The method can be used to create dynamic elements in 3D game layout farms that are responsive to mouse handlers.
Articulated animal reconstruction: Farm3D can reconstruct the shape of a wide range of categories to their fine details, including legs and ears, even without being trained on real images.
Virtual supervision: The framework uses an image generator, such as Stable Diffusion, to produce training data for learning a reconstruction network from the ground up.
Single-view 3D reconstruction: Farm3D enables the generation of controllable 3D assets from either a real image or an image synthesised using Stable Diffusion.
