SonicVista:
Generating a 4D Audio-Visual Scene from One Video via Diffusion Synchronization
TL;DR: SonicVista turns a single video into an explorable 4D audio-visual scene, using dual-space audio diffusion synchronization to generate physically grounded spatial audio for static and moving sources, enabling real-time novel-view audio-visual rendering.
Rebuttal
New qualitative experiments added for the rebuttal.
Foreground–background harmonization
We apply DiffusionRenderer (Cosmos) to our renderings as an offline relighting step: its inverse renderer predicts G-buffers of the composited clip, and its forward renderer relights the clip under an environment map rendered from our reconstructed background. Hover over a row to play its videos together.Tap a row to play its videos together. From left to right: input video(s), ours without harmonization, ours with harmonization. Visual comparison only; these clips have no audio.
Optional use of the input audio
By default our method does not use the input video's audio: the input is RGB frames only, and all audio is generated from visual and text conditioning. The framework can optionally incorporate the input audio when it is available: the right column of each row below, including the multi-source example, shows the result. Hover over a row to play; with Sound on, hover over a video to hear it.Tap a row to play; with Sound on, tap a video to hear it.
Scaling to multiple sound sources
Three musicians playing three instruments: our method generates and synchronizes three object-level audio tracks, without (middle) or with (right) the input audio. As a proof of concept, each musician–instrument pair is a static 3D Gaussian reconstructed from a single frame with TeHOR. Hover over the row to play; with Sound on, hover over a video to hear it.Tap the row to play; with Sound on, tap a video to hear it.
Results
Hover over a row to play both videos.Tap a row to play its videos. Left: input monocular video. Right: our novel-view rendering. Turn on Sound in the top bar (headphones recommended): hovering overtapping the right video plays our binaural audio, and hovering overtapping the left video plays the input audio, if the input has one. The input audio is provided for reference only: our method does not take it as input, and it may be unrelated to the video content (e.g. background music).
Single input video
Each scene is reconstructed from one monocular video, which may contain several dynamic sound sources.
Multiple input videos
Each dynamic sound source is captured by its own monocular video and the sources are composed into one scene.
AblationsOurs (T2A) / Ours (V2A) / Ours
Comparison with Ours (T2A), which generates each dynamic-source track from text, and Ours (V2A), which generates audio conditioned on the input video. With Sound on, switch versions to compare.
Plane
* Plane scene at a novel listener pose. Listen for the Doppler shift as the plane passes the input camera and novel listener: V2A overfits to the input viewpoint and fails to adapt the effect to the listener, while T2A alters timbre.
Tap dance
Two people
Separate Sound TracksPer-source tracks
Each sound source is generated and spatialized as its own track. Pick a track to hear it alone at the novel listener pose, or the overall mix.
Porsche
Tap dance
Two cars
Baseline ComparisonsOurs / Sonic4D / OmniAudio
Comparison with Sonic4D and OmniAudio at the same novel listener pose. Sonic4D renders its own video and audio; OmniAudio predicts audio from our novel-view rendering. With Sound on, switch methods to compare.
Porsche
Plane
Tap dance
Robot
Horse riding
Two cars
Two people
Audio-space mapping demo · PlaneReal audio · source / camera / inverse
Hear how diffusion synchronization moves audio between spaces.
At every sampling step, diffusion synchronization decodes clean-audio predictions in both object space and camera space, then maps them between the two and fuses the results. This demo lets you listen to the forward and inverse rendering operations that make the exchange possible — using a real plane fly-over as the source.
Loading plane audio and trajectory…
At the camera · reception-time playback
Input camera video
Reconstructed source / camera trajectory
Object source
Forward rendering
Time warp & propagation gain
At the input camera
Compare the plane's source audio with its contribution at the input camera. The reconstructed trajectory determines the time warp and propagation gain.
Plane component only. Camera audio is re-rendered from the source, not the archived full-scene mix. Inverse rendering uses the known plane contribution; it does not run source separation.