SonicVista:
Generating a 4D Audio-Visual Scene from One Video via Diffusion Synchronization

Interactive Demo

TL;DR: SonicVista turns a single video into an explorable 4D audio-visual scene, using dual-space audio diffusion synchronization to generate physically grounded spatial audio for static and moving sources, enabling real-time novel-view audio-visual rendering.

Rebuttal


New qualitative experiments added for the rebuttal.

Foreground–background harmonization

We apply DiffusionRenderer (Cosmos) to our renderings as an offline relighting step: its inverse renderer predicts G-buffers of the composited clip, and its forward renderer relights the clip under an environment map rendered from our reconstructed background. Hover over a row to play its videos together.Tap a row to play its videos together. From left to right: input video(s), ours without harmonization, ours with harmonization. Visual comparison only; these clips have no audio.

Input video
Ours w/o harmonization
Ours w/ harmonization
Input video
Ours w/o harmonization
Ours w/ harmonization
Input (red car)
Input (white car)
Ours w/o harmonization
Ours w/ harmonization

Optional use of the input audio

By default our method does not use the input video's audio: the input is RGB frames only, and all audio is generated from visual and text conditioning. The framework can optionally incorporate the input audio when it is available: the right column of each row below, including the multi-source example, shows the result. Hover over a row to play; with Sound on, hover over a video to hear it.Tap a row to play; with Sound on, tap a video to hear it.

Input video
Ours w/o input audio
Ours w/ input audio
Input video
Ours w/o input audio
Ours w/ input audio

Scaling to multiple sound sources

Three musicians playing three instruments: our method generates and synchronizes three object-level audio tracks, without (middle) or with (right) the input audio. As a proof of concept, each musician–instrument pair is a static 3D Gaussian reconstructed from a single frame with TeHOR. Hover over the row to play; with Sound on, hover over a video to hear it.Tap the row to play; with Sound on, tap a video to hear it.

Input video
Ours w/o input audio
Ours w/ input audio

Results


Hover over a row to play both videos.Tap a row to play its videos. Left: input monocular video. Right: our novel-view rendering. Turn on Sound in the top bar (headphones recommended): hovering overtapping the right video plays our binaural audio, and hovering overtapping the left video plays the input audio, if the input has one. The input audio is provided for reference only: our method does not take it as input, and it may be unrelated to the video content (e.g. background music).

Single input video

Each scene is reconstructed from one monocular video, which may contain several dynamic sound sources.

Input video
Ours (novel view)
Input video
Ours (novel view)
Input video
Ours (novel view)
Input video
Ours (novel view)
Input video
Ours (novel view)
Input video
Ours (novel view)
Input video
Ours (novel view)
Input video
Ours (novel view)
Input video
Ours (novel view)
Input video
Ours (novel view)

Multiple input videos

Each dynamic sound source is captured by its own monocular video and the sources are composed into one scene.

Input (red car)
Input (white car)
Ours (novel view)
Input (person 1)
Input (person 2)
Ours (novel view)
AblationsOurs (T2A) / Ours (V2A) / Ours

Comparison with Ours (T2A), which generates each dynamic-source track from text, and Ours (V2A), which generates audio conditioned on the input video. With Sound on, switch versions to compare.

Plane

Input video
Ours

* Plane scene at a novel listener pose. Listen for the Doppler shift as the plane passes the input camera and novel listener: V2A overfits to the input viewpoint and fails to adapt the effect to the listener, while T2A alters timbre.

Version:

Tap dance

Input video
Ours
Version:

Two people

Input (person 1)
Input (person 2)
Ours
Version:
Separate Sound TracksPer-source tracks

Each sound source is generated and spatialized as its own track. Pick a track to hear it alone at the novel listener pose, or the overall mix.

Porsche

Input video
Overall
Sound track:

Tap dance

Input video
Overall
Sound track:

Two cars

Input (red car)
Input (white car)
Overall
Sound track:
Baseline ComparisonsOurs / Sonic4D / OmniAudio

Comparison with Sonic4D and OmniAudio at the same novel listener pose. Sonic4D renders its own video and audio; OmniAudio predicts audio from our novel-view rendering. With Sound on, switch methods to compare.

Porsche

Input video
Ours
Method:

Plane

Input video
Ours
Method:

Tap dance

Input video
Ours
Method:

Robot

Input video
Ours
Method:

Horse riding

Input video
Ours
Method:

Two cars

Input (red car)
Input (white car)
Ours
Method:

Two people

Input (person 1)
Input (person 2)
Ours
Method:
Audio-space mapping demo · PlaneReal audio · source / camera / inverse

Hear how diffusion synchronization moves audio between spaces.

At every sampling step, diffusion synchronization decodes clean-audio predictions in both object space and camera space, then maps them between the two and fuses the results. This demo lets you listen to the forward and inverse rendering operations that make the exchange possible — using a real plane fly-over as the source.

Listen to

Loading plane audio and trajectory…

At the camera · reception-time playback

Input camera video

Reconstructed source / camera trajectory

The reconstructed plane trajectory and the camera's position.
Plane trajectory Source–camera distance — Pitch ratio —
0.00 s

Object source

Plane source audio before propagation.

Forward rendering

Time warp & propagation gain

At the input camera

The plane source after acoustic propagation to the input camera.
bplane=ℛ[aobj]

Compare the plane's source audio with its contribution at the input camera. The reconstructed trajectory determines the time warp and propagation gain.

Plane component only. Camera audio is re-rendered from the source, not the archived full-scene mix. Inverse rendering uses the known plane contribution; it does not run source separation.

SonicVista: Generating a 4D Audio-Visual Scene from One Video via Diffusion Synchronization