More video tinkering with Vidu 1.5
Vidu hit 1.5 this morning for testing, which I get access to as part of the artist's program. While making raw material for a "Big Professor" project-
(Shown here to establish a 1.0 baseline)
-I nuked most of my credits, I've still got enough to try out the new features and give you all a rundown of them.
First major change is resolution options. Fast, 720, or 1080. Shown here is my first attempt at a 1080 gen, using the same prompt as above. While most vids in this post will be gifs for ease of use, this one is an MP4 to provide full evaluation.
Ignore the odd playback speed, since that's something you'd fix in post. The gif at the start was one of my most stable renderings of Big Professor under v1. Got some minor wonk in the details, little distortion around the prof's claw and the kid in the first half second, but otherwise well above the average of the 1.0 versions I've been working with, especially the clarity on the button text.
The second one is a re-run of the same prompt, minus some detail issues with one shin, the level of consistency is promising.
720 can go up to 8 seconds, 1080 tops off at 4. At present, 1.5 does not have an upscale function. If you're not tinkering to test out prompts I'd generally recommend 1080, even for lower-res projects, for the purposes of being able to crop and/pan.
For most of these, assume I'm using either 720 or 1080 gens.
Multi-Subject Reference-to-Video
THIS is the big winner of the setup. Previously, ref-to-video only could handle a single referenced subject image. The current setup now allows you to reference up to three per generation.
Those can be characters, props, or settings, and it can be used for morphing effects. This is good, because first-frame-last-frame is more finnicky than reference-to-video. The obvious use is to create multi-character scenes, and it works well for that:
But it has other uses. For example, here I used:
And while this one was reversed and trimmed, it displays the potential for using the feature for transformation sequences. The backgrounds are especially helpful, since Vidu's internally-generated toon backgrounds are very modern and digital-painting-looking (the gif of Mrs. Nice turning back and forth into Mrs. Nautilus at the top of the section didn't use a background image reference) and as always, I like it retro.
To that point, using the more comic-booky Mrs. Nice control drawing didn't work near as well as the more period-accurate one:
And while I'll personally be cutting out frames to produce a more accurate 1980s TV animation look, the raw fluidity is bound to appeal to the majority of users, and minor issues like the color of Mrs. Nice's blouse changing mid-transformation are easy enough to fix manually (though if a flaw looks enough like a traditional animation screwup, I might just leave it in)
And another fun feature? Movement amplitude control.
In my last post about it, I said Vidu likes to move, and if you need a more subtle shot, like this one, setting the movement amplitude control to small is your move. (Funny how the robot interpreted the book binding as a mouth.) If something's too static you can up the movement as well, but I haven't had that problem much.
In terms of generation quality, 1.5 is a good but not huge leap up from 1.0. at least in terms of what I've made so far. Aesthetics are very similar, but issues of tearing, distortion, and wonky action are lessened.
And it's way better at understanding triceratops, so thumbs up there.
This is one of my wonk-tests, but man, check that walk cycle. This is why the movement controls are important, sometimes when set to auto a dialog scene becomes a weird chase.
Now, in terms of generation costs (and this can always change), 1.5 seems more expensive but in the end it works out more or less the same.
A 1.0 quality 8-second gen upscaled to max res would be 24 points (16 for the quality 8-seconds, 8 for the upscale), and an 8-second 720 render is also 24 points (as is a 4-second 1080).
Yeah, you don't want to use "Fast" in most cases. While both the 720 and fast versions here both have flaws (Bruce's disappearing popcorn bucket, etc), the 720 res version is substantially more on-model for both characters.
To close out, a bit of fun wonk. To try out the effect, I prompted using a "Specialist Matt Trakker" G.I.Joe/M.A.S.K. crossover figure as a character reference, to see if it could translate a character from toy to live action. And it can-
Though it is very, very literal.
The combination of live action-esq person and accurate G.I.Joe joints is harrowing, but there's potential there for some wild variants on the Toy Story/A Christmas Toy/'Wynona's Big Brown Beaver'-Music-Video-by-Primus concept.
