5 Years ago, temporal refugees from the Cretaceous period began arriving. Now, we catch up with them to see how our newly rediscovered cousins have adapted.
--
You may recognize these dinos from here:
Which also lets you see how far things w' Vidu have come in a year.
Made with Vidu-Q3, image-to-video model, I'm in their artist's program.
Details and Tutorial under the fold.
Looks like I don't have an excuse not to make those shows now.
Vidu just launched two models, Q2-Pro with video-prompting and reference-to-video, and Q3, which is currently text-to-video and image-to-video only, but can do up to 16 seconds with dialog and sound effects.
As the video above is longer than 16 seconds, obviously, editing is involved, as it always is when something is more than a quickie one-off image or clip.
But, this does open up a lot of potential.
The basic process of putting together a clip is simple. You start with an image, and then you script out your prompt.
Long time fans might recognize these dinos and some of their interview quotes from here. So I had character art ready, and my own writing to start with.
Generally, where I started actually wasn't in Q3, but Q2-Pro, where I'd use the reference-to-video options to create starting frames. This is helpful because even with reference the AI has to reinterpret things and there's always some style shift. Using the ref-to-video to create base frames makes it so scene-to-scene those changes aren't random, as you're essentially giving the AI an image that's ready-made for Vidu's use.
Vidu even gives you a built-in method to send the images directly to the prompt with its frame extraction feature, but I recommend to always also download it (without the watermark), because you will want to go back to that image, have the option to edit it, etc.
From there, it's a matter of prompting:
Dynamic anime fight scene, multiple cuts and angles. 2005, traditional hand-drawn cel animation, masterpiece sakuga cinematic animation quality. A green-skinned, skullfaced barbarian queen with a huge scythe duels against an orange-skinned goat-anthro with pink flames for hair and trim fighitng with a golden trident. the goat-anthro wears a sparkling burlesque costume and has a long snake-like tail. SHOT 1: the camera circles the fighters slowly, establishing their relative positions and forms. SHOT 2: cut to closeup on the skullfaced barbarian, her voice is sinister and high pitched, dripping with sarcastic evil SKULLFACED: "A goat that's self-braising, how very quaint!" she is voiced by Sarah Skeletoria. SHOT 3: cut to the goat-anthro, front view, middle-closeup, she takes a defensive stance with her trident. GOAT-ANTHRO: "I won't let you hurt anymore dames, Skullsa Doom!" she has a valleygirl accent and sexy, confident voice. SHOT 4: Goat-anthro charges Skullfaced with her trident forward, Skullfaced barely dodges, the trident rips skullfaced's cloak. flame-themed anime motion background. She is driven by righteous fury. GOAT: "And I won't let you hurt Audi!" SHOT 5: Skullfaced swings her scythe, but goat-anthro backflips away, dodging just in time. her flame-hair forms a long arc as she dodges, enhancing the sense of grace an motion. SKULLFACED: "I don't require your permission, Baph-lessk!" SHOT 6: They continue to duel, evenly matched.
In the form of a script, essentially. You'll notice I used the same reference names for the characters, and that I avoided pronouns. This helps reduce confusion. As does phonetically spelling out stuff that's difficult to pronounce.
There is an important early-series bug that needs to be advised on. If the clip isn't the right length for the required dialog, the characters will often flub or babble lines. This means that you're going to want to time-talk-out your dialog before generating (Q3 is credits-hungry compared to Q2-Pro), or you get what I got here:
Which is the clip going off the rails partway through. If you're trial-and-erroring it will get expensive, but with editing, and by re-dubbing with recorded voices, you'd get around that easily.
Generally, I just generate shorter clips for editing, but for showing off the system, a full 16 second demo seemed best.
Vidu does improve their models over time, and I expect this to be ironed out in the next few weeks, but in the meantime, keep it in mind.
The potential of the tool is immense. Not just because of the coherence and quality of the animation produced, but because it's set up for actual production. A team of a handful of people could produce an entire show on a scrap of a budget with this.
If you want to try it yourself, go to Vidu here and use promo code ROBOTQ3 for 100 free credits.
This post is, I'm pretty sure, easily under the umbrella of paid promotion.
