This page explains the difference between prompt-driven and audio-driven generation, what kinds of sound work, and how to combine audio with a short prompt for control.
Prompt-driven vs audio-driven image to video with audio
General video models are great at scenery and camera moves described in words, but they have no idea when a syllable lands. Audio-driven models read the waveform: plosives close the lips, stressed words trigger emphasis, pauses let the face settle. The result is synchronised to the frame, which matters for any clip where a person speaks or sings on screen.
With OmniHuman 2 you still get a prompt — but it shapes the performance and the camera, while timing comes entirely from the sound.
What audio works
The model is built around human voices. Best results come from:
- Spoken voice-overs, interviews, lessons and podcast segments.
- Singing — solo vocals, duets and choruses.
- Dialogues between two or more people in the same frame.
- Synthetic speech exported from text-to-speech tools.
Combining audio with a short prompt
Use the prompt for what the audio can’t express: where the person looks, how the camera moves, which gesture accompanies a line. Keep it to a sentence or two. “Looks at the camera, opens hands on the question, slow push-in” is plenty for a 20-second clip.
Format-wise, upload MP3 or WAV between 4 seconds and 5 minutes; images can be JPG, PNG, WEBP and more, up to 30 MB.
From image and audio to finished video
The process in the studio:
- 1ImageUpload the still you want to bring to life.
- 2AudioUpload or record the track that should drive it.
- 3DirectionOptionally add one sentence of performance and camera notes.
- 4RenderPreview for free, then render a 720p MP4.
Pairing it with other AI tools
Many creators build the inputs with other tools: an image generator for the character, a voice tool for narration, and this model for the performance. That pipeline keeps each step specialised — design, voice, motion — and lets you swap any one part without redoing the others.
If you generate the image yourself, ask for a front-facing character with the mouth visible and a plain background; those choices make the later animation step much more reliable.
Limitations to know
The model animates people and characters, not whole scenes; background elements stay largely still apart from camera motion. It also cannot change the outfit, add props or move the person to a new location — those decisions belong in the source picture.
Getting the cleanest result
Audio quality sets the ceiling for sync, and image clarity sets the ceiling for realism. A sharp, evenly lit face and a dry, close-mic recording will improve your result more than any prompt wording.
FAQ
Can I add music to the background?
Add it in your editor after rendering. Music under the voice during generation can reduce lip-sync accuracy.
Does the audio set the video length?
Yes. The video is exactly as long as your audio, from 4 seconds to 5 minutes.
Can it animate landscapes from audio?
The model is designed for people and characters. Scenes without a face won’t produce meaningful results.
Which audio formats work?
MP3 and WAV. Browser recordings are converted automatically.
Do I need a prompt?
No. Without one, motion comes purely from the audio; a prompt adds direction.
Is sound included in the downloaded video?
Yes. The MP4 contains your original audio synchronised with the animation.
Can I use a generated voice and a generated image together?
Yes. Many creators combine an image generator, a text-to-speech voice and OmniHuman 2 — just make sure you have rights to each input.