Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event Condition for Foley Sound
Junwon Lee, Jaekwon Im, Dabin Kim, Juhan Nam
Paper Code-TBA 🤗 Checkpoints-TBA
TL;DR Controllable Video-To-Sound generation system ensures high temporal and semantic audio-visual synchrony using RMS as a temporal feature.
The two-stage self-supervised learning framework (Video2RMS and RMS2Sound) achieves sota performance in aligning and controlling sound timing, intensity, timbre, and nuance.
shortcuts Abstract Appendix DEMO Unlock the Potential of RMS-ControlNet
Abstract
Foley sound synthesis is crucial for multimedia production, enhancing user experience by synchronizing audio and video both temporally and semantically. Recent studies on automating this labor-intensive process through video-to-sound generation face significant challenges. Systems lacking explicit temporal features suffer from poor controllability and alignment, while timestamp-based models require costly and subjective human annotation. We propose Video-Foley, a video-to-sound system using Root Mean Square (RMS) as a temporal event condition with semantic timbre prompts (audio or text). RMS, a frame-level intensity envelope feature closely related to audio semantics, ensures high controllability and synchronization. The annotation-free self-supervised learning framework consists of two stages, Video2RMS and RMS2Sound, incorporating novel ideas including RMS discretization and RMS-ControlNet with a pretrained text-to-audio model. Our extensive evaluation shows that Video-Foley achieves state-of-the-art performance in audio-visual alignment and controllability for sound timing, intensity, timbre, and nuance.
Appendix
For appendix contents, please refer to the paper on Arxiv: Video-Foley.
DEMO Show
Comparison among differnt Video-to-Sound models
(Video-Foley(ours), SyncFusion, CondFoleyGen - Audio prompt, Diff-Foley - No prompt)
Unlock the Potential of RMS-ControlNet
RMS-ControlNet, trained for additional RMS guidance on top of the pretrained Text-to-Audio model (AudioLDM), shows great potential in controllable audio generation tasks.
We provide demos to showcase its high controllability, which prior TTA models were not able to achieve.
(can be interpreted as a text-to-audio version of T-Foley.)
RMS guidance with a Text Prompt Show
RMS-ControlNet guides AudioLDM to generate audio that matches different input RMS conditions (A-shaped, monotonic decrease, monotonic increase, and V-shaped) while maintaining audio semantics of text prompt. Such intensity dynamics are often used in Foley sound generation, which current text-to-audio models struggle to reflect with sufficient temporal accuracy.
Text Prompt Control with RMS Guidance Show
Through text prompts, users can control audio semantics such as the sound source, timbre, and nuance with the same input RMS. This highlights RMS-ControlNet's ability to guarantee high controllability in RMS guidance for timing and intensity while preserving the power in text-to-audio generation.
Citation
@article{video-foley,
title={Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event Condition For Foley Sound},
author={Lee, Junwon and Im, Jaekwon and Kim, Dabin and Nam, Juhan},
journal={arXiv preprint arXiv:2408.11915},
year={2024}
}
References
video sources
https://youtu.be/01le4Ln8da0?si=L1sv1wudk6EKa8Xu
https://youtu.be/0rlS80hRIVQ?si=THexWAzoVNDEdyPM
https://youtu.be/NpKOVux1qVE?si=Jp_q0Jr8PIujHAQg
https://youtu.be/-2RiNR2fqRY?si=YCpA5hhnsLlBia1f
https://youtu.be/JcN7Ej6QbAU?si=Mh80shPdkxfOSIZN