Audio-driven portrait animation

StreamAvatar / AROD

A DyStream-based portrait animation system with an Autoregressive One-step Denoising student that reduces motion rollout cost while preserving the frozen teacher and renderer stack.

PanChangxun · GuoZihan · LiSi

GitHub Report PDF Proposal PDF Model on HF

Demo

The current release is an offline Gradio demo with a streaming-capable blockwise motion design. This demo shows multi-identity generation driven by the same audio.

Four AROD talking-head generations driven by the same audio
Same-audio AROD generation across Person1, Person2, and two public-domain NASA/Wikimedia reference portraits.

Method

StreamAvatar keeps DyStream's pretrained audio encoder and renderer, then distills the expensive autoregressive flow-matching motion rollout into a blockwise AROD student.

StreamAvatar system pipeline
System pipeline: audio and reference inputs feed a faster student path before frozen rendering.
AROD student architecture
AROD student architecture: audio, history, and anchors condition short future motion blocks.

Results

The real-anchor student is the primary released model. Mixed/noise-anchor training is reported as an ablation because it can change visible motion strength and long-horizon consistency, but does not consistently improve lip-sync metrics. The original Person2 comparison was rendered as a 60-second sequence and remained visually consistent across the full clip.

8x

Fresh verification speedup

Motion-only AROD rollout was roughly eight times faster than the DyStream teacher in the latest 60-second verification run.

10x

Observed peak setup

The same verification setup has reached about ten times faster motion rollout under earlier runs.

HF

Public checkpoint

The released AROD real-anchor checkpoint is hosted on Hugging Face with a Google Drive mirror.

Person2 three-way comparison clip: DyStream teacher, real-anchor AROD student, and mixed-anchor ablation. This 3-second preview is trimmed from a 60-second verification video, where the reference identity stays consistent across the full long-horizon run.

Reproduce

Clone the repository, install dependencies, download the model assets, and launch the Gradio demo. 

git clone https://github.com/CXP-2024/StreamAvatar.git
cd StreamAvatar
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
bash scripts/download_assets.sh
bash run.sh
Source code GitHub repository Checkpoint Hugging Face model repo Report Final project report Proposal Original project proposal