Precise Video-to-Audio Generation with Cross-Modal Alignment in Latent Space

Thanh V. T. Tran¹     Ngoc-Son Nguyen¹     Luong Tran¹     Long-Khanh Pham¹     Paarth Neekhara²     Shehzeen Hussain ²     Van Nguyen¹

¹ FPT Software AI Center, Vietnam    ² NVIDIA Corporation, USA

Table of Contents

• Introduciton
• Installation
• Training
• Inference
• Citation
• Acknowledgement

Introduction

Flowley is single-stage architecture that synthesizes temporally and semantically aligned soundtracks from silent videos. The key architecture innovation is Progressive Soft-masked Cross-Attention (PSCA), a novel mechanism that embeds precise temporal prior directly within the attention layers.

We also propose SoundCap, a plug-and-play pipeline leveraging audio-visual LLMs to generate detailed, sound-oriented captions that robustly guide our model.

Checkout our project page for more generated samples.

Installation

We have only tested this on Ubuntu 22.04 and PyTorch 2.0+

1. Install dependencies

We use PyTorch 2.2 with CUDA 11.8. Please check if your GPUs/driver support this.

cd Flowley
conda env create -f environment.yml

2. Install project

pip install -e .

Training

See TRAINING.md

Inference

See INFERENCE.md

Citation

If our paper or codebase aids your research, please consider citing us:

@article{tran2026precise,
  author  = {Thanh V. T. Tran and Ngoc-Son Nguyen and Luong Tran and Long-Khanh Pham and Paarth Neekhara and Shezheen Hussain and Van Nguyen},
  title   = {Precise Video-to-Audio Generation with Cross-Modal Alignment in Latent Space},
  journal = {arXiv preprint arXiv:2607.06405},
  year    = {2026},
  url     = {https://arxiv.org/pdf/2607.06405}
}

Acknowledgement

We would like to thank the authors for their great work.

• Make-An-Audio-2
• MMAudio
• rectified-flow-pytorch