Unlock the Power of 3D Data: A Deep Dive into SynCD Dataset Generation

Ready to revolutionize your 3D model training? The SynCD dataset offers a streamlined solution for generating high-quality, synthetic data. Whether you're dealing with rigid or deformable objects, this guide breaks down the process step-by-step, empowering you to create custom datasets tailored to your specific needs. This article focuses on leveraging the SynCD tools and utilities for optimal data generation and use.

Download and Dive In: Accessing the SynCD Dataset

Ready to start? You can download the pre-filtered generated SynCD dataset directly from the source. This can serve as a great starting point to understand the data structure and experiment with your models immediately.

Getting Started: Environment Setup for SynCD

Before you can generate your own datasets, ensure you have a powerful GPU (at least 48GB VRAM recommended). Follow the base environment setup instructions provided to prepare your system.

Generating Deformable Datasets with SynCD

Want to create datasets featuring objects that can change shape? Here's how:

1. Navigate to the dataset directory: cd dataset
2. Run the deformable dataset generation script: python gen_deformable.py --save_attn_mask --outdir assets/metadata/deformable_data

This script generates a dataset of deformable objects, complete with attention masks, saved in the specified output directory.

Rigid Dataset Generation: A Step-by-Step Guide

Creating datasets of rigid objects is just as straightforward. Follow these instructions for rigid dataset generation.

Example: Generating a Dataset from a Single Objaverse Asset

1. Download pre-generated prompts: wget https://www.cs.cmu.edu/~syncd-project/assets/prompts_objaverse.pt -P assets/generated_prompts/
2. Unzip rendering data bash assets/unzip.sh assets/metadata/objaverse_rendering/
3. Run the generation script: torchrun --nnodes=1 --nproc_per_node=1 --node_rank=0 --master_port=12356 gen_rigid.py --rootdir ./assets/metadata --promptpath assets/generated_prompts/prompts_objaverse.pt --outdir assets/metadata/rigid_data

This command utilizes torchrun for distributed processing, significantly speeding up the generation process.

Objaverse-Guided Rigid Dataset Generation: Elevate Your Data

Leverage the power of Objaverse for even richer datasets. This approach uses re-rendered Objaverse assets with depth conditioning for enhanced realism. We use FLUX.1-Depth-dev instead of xflux for depth conditioning, different from the original paper.

Preparing for Objaverse Dataset Generation

• Install the necessary libraries:

  pip install objaverse
  pip install ninja
  pip install trimesh
  pip install "git+https://github.com/facebookresearch/pytorch3d.git"
  

  Copy

Multi-View Correspondence Calculation

This part requires pytorch3d.

1. Navigate to the rendering directory: cd assets/metadata/objaverse_rendering
2. Download a subset of Objaverse renderings: wget https://huggingface.co/datasets/nupurkmr9/objaverse_rendering/resolve/main/archive_1.zip
3. Unzip the archive: unzip archive_1.zip
4. Return to the root directory: cd../../../
5. Unzip data using: bash assets/unzip.sh assets/metadata/objaverse_rendering/
6. Run the correspondence generation script: python gen_corresp.py --download --rendered_path ./assets/metadata/objaverse_rendering --objaverse_path ./assets/metadata/objaverse_assets --outdir ./assets/metadata

The parameter --download allows the script to automatically download necessary assets. The script calculates multi-view correspondences, crucial for accurate 3D representation.

Final Dataset Generation

Use this command to generate the final dataset:

torchrun --nnodes=1 --nproc_per_node=1 --node_rank=0 --master_port=12356 gen_rigid.py --rootdir ./assets/metadata --promptpath assets/generated_prompts/prompts_objaverse.pt --outdir <output-path-to-save-dataset>

Replace <output-path-to-save-dataset> with your desired output directory.

Generate Your Own Prompts with LLMs

Customize your datasets further by generating prompts using Large Language Models (LLMs).

• Object and Image Background Description: Use the classes defined in assets/categories.txt to guide prompt generation.

• Background Descriptions for Objaverse Assets:

  1. Download the Cap3D captions: wget https://huggingface.co/datasets/tiange/Cap3D/resolve/main/misc/Cap3D_automated_Objaverse_old.csv?download=true -O Cap3D_automated_Objaverse_old.csv
  2. Run the prompt generation script: python gen_prompts.py --rigid --captions Cap3D_automated_Objaverse_old.csv

This script creates prompts tailored to rigid object generation, enhancing the diversity and quality of your dataset. The use of generated prompts from LLM is pivotal to producing a high-quality dataset.

By following these steps, you can harness the power of the SynCD dataset generation tools to create custom 3D datasets perfectly suited for your machine learning projects. This gives you control over the final data, which when used correctly, can significantly increase the accuracy of your models.