---

Overview

SonicGauss is a novel framework for synthesizing impact sounds from 3D Gaussian Splatting (3DGS) representations by leveraging their inherent geometric and material properties.

---

Getting Started

Prerequisites

# Clone the repository with submodules
git clone https://github.com/AiEson/SonicGauss.git --recursive

cd SonicGauss

# Create conda environment
conda create -n sonicgauss python=3.10
conda activate sonicgauss

Please install the requirements for the following repositories: 
- TangoFlux: https://github.com/declare-lab/TangoFlux.git
- SplatFormer: https://github.com/ChenYutongTHU/SplatFormer.git

---

Dataset

We train and evaluate on the ObjectFolder dataset:

• ObjectFolder 2.0: 1,000 objects with synthetic impact sounds
• ObjectFolder Real: Real-world recordings with diverse materials

Dataset Download

For detailed download instructions, please refer to datas/README.md

The dataset includes:

• ObjectFolder 2.0 and ObjectFolder Real datasets
• 3DGS PLY files
• Rendered images
• Impact sound recordings
• Training/validation split files

Total Size: ~24.7 GB

---

Pretrained Models

For model download instructions, please refer to ckpts/README.md

Total Size: ~3.5 GB

---

Training

Stage 1: Text-to-Audio Pretraining

bash stage1/train.sh

Stage 2: Gaussian-to-Audio Learning

bash stage2/train_2_1.sh
bash stage2/train_2_2.sh

Stage 3: Position-Aware Fine-tuning

bash stage3/train_3.sh

---

Inference

Generate impact sounds from a 3DGS representation:

from stage3.infer_3 import Stage3Inference

# Initialize model
model = Stage3Inference(
    model_path="./ckpts/stage3/best",
    config_path="./configs/stage3.yaml",
    device="cuda"
)

# Generate audio
audio = model.generate(
    ply_path="path/to/object.ply",
    position=[0.1, 0.2, 0.3],  # Impact position (x, y, z)
    duration=3.0,              # Audio duration in seconds
    steps=50,                  # Diffusion steps
    seed=42
)

# Save audio
import torchaudio
torchaudio.save("output.wav", audio, 44100)

Inference with JSON input

python stage3/infer_3.py

Then the generated audio files can be found in the generated/stage3 directory.

Evaluation

We evaluate using standard audio generation metrics:

cd stage3
python eval.py

---

Project Structure

SonicGauss/
├── configs/                 # Configuration files
│   ├── stage1.yaml
│   ├── stage2.yaml
│   └── stage3.yaml
├── stage1/                  # Text-to-Audio pretraining
├── stage2/                  # Gaussian-to-Audio learning
│   ├── train_2_1.py        # Contrastive learning
│   ├── train_2_2.py        # Generative training
│   └── common.py           # Shared utilities
├── stage3/                  # Position-aware fine-tuning
│   ├── train_3.py          # Training script
│   ├── infer_3.py          # Inference script
│   ├── eval.py             # Evaluation script
│   └── common.py           # Position encoder & fusion
├── SplatFormer/            # PTv3 backbone (submodule)
├── TangoFlux/              # Diffusion model (submodule)
├── audioldm_eval/          # Evaluation toolkit (submodule)
├── datas/                  # Dataset files
├── ckpts/                # Training checkpoints
└── assets/                 # Images and resources

---

Citation

If you find this work useful, please cite our paper:

@inproceedings{wang2025sonicgauss,
  title={SonicGauss: Position-Aware Physical Sound Synthesis for 3D Gaussian Representations},
  author={Wang, Chunshi and Li, Hongxing and Luo, Yawei},
  booktitle={Proceedings of the 33rd ACM International Conference on Multimedia},
  pages={10886--10895},
  year={2025}
}

---

Acknowledgements

We thank the authors of the following projects for their excellent work:

• ObjectFolder - Multi-modal object dataset with impact sounds
• TangoFlux - Text-to-audio diffusion model
• SplatFormer - 3DGS feature extraction with PointTransformer
• audioldm_eval - Audio generation evaluation toolkit

---