Lip Forcing: Few-Step Autoregressive Diffusion for Real-time Lip Synchronization
Authors: Lip Forcing Team |Year: 2026 |arXiv: 2606.11180
二、研究背景
基于扩散的唇语同步方法实现了强视觉质量和音视频对齐，但全序列双向注意力和多步去噪使其难以用于实时推理。
核心挑战：如何在保持视觉质量的同时，将推理速度从教师模型的数秒/帧压缩到实时（>25 FPS）？
四、实验
模型规模：1.3B 和 14B 两个学生规模（均从 14B 教师蒸馏）
指标：FPS（实时性）、SyncNet 分数（同步性）、LPIPS/SSIM（参考保真度）、首帧时延
| 模型 | FPS | 相对加速 |
|---|---|---|
| 14B 教师 | ~0.8 | 1× |
| 1.3B 双向基线 | ~1.8 | 2.3× |
| 1.3B Lip Forcing | 31 | 17.6×（vs 同规模双向） |
| 14B Lip Forcing | ~32 | 39.8×（vs 教师） |
首帧时延：低于 1 毫秒（所有扩散基线均远高于此）。
报告生成时间：2026-06-11 | 论文来源：arXiv:2606.11180
原文摘要:Diffusion-based lip synchronization models achieve strong visual quality and audio-visual alignment, but full-sequence bidirectional attention and many denoising steps make them impractical for real-time inference. We present Lip Forcing, to our knowledge the first autoregressive diffusion method for video-to-video (V2V) lip synchronization, which distills a 14B audio-conditioned bidirectional video diffusion teacher into causal students. At inference, the students generate each chunk in only two denoising steps without inference-time CFG, enabling real-time lip synchronization. A lip-sync-specific teacher-trajectory analysis reveals a CFG fidelity-sync tradeoff: no-CFG predictions favor reference fidelity, whereas CFG-guided predictions favor synchronization within a mid-trajectory band. Lip Forcing translates this finding into three analysis-derived components: Sync-Window DMD, a two-step inference schedule, and a SyncNet-based reward. We validate Lip Forcing at two student scales, both distilled from the 14B teacher. The 1.3B student crosses into real-time streaming at 31 FPS,17.6 × 17.6\times17.6×faster than its same-scale bidirectional model. The 14B student, the largest diffusion model reported for V2V lip synchronization, runs39.8 × 39.8\times39.8×faster than its teacher at comparable reference fidelity. Time-to-first-frame is sub-millisecond at both scales, far below every diffusion baseline.
PDF链接:https://arxiv.org/pdf/2606.11180v1
