#sabriel ai adaptation progress notes

[ellynneversweet]

So, in the interests of a diary of the Sabriel short project, here’s where I’m at (not very far), tagged for convenience or Luddite principles or whatever:

So I’ve already addressed the question of why I’m playing with this, kind of, but really I’d been thinking on and off for probably about eight months that I’d love a Sabriel movie, and we’re unlikely to get one and if we did it would probably suck because fantasy film making at the moment has chronic marvel industries irony disease. And I pondered making, say, a trailer with blender 3D (this is a 3D design program which I use for other stuff, very useful) but if I make big complicated files in that my computer starts making scary noises. Then AI video got good enough to be interesting so I am at least going to have a go at making a few scenes. Onwards!

I have a fairly particular internal eye that I want to satisfy and this is surprisingly difficult with AI. Possibly because I’m terrible at prompting, but more, I suspect, because I know exactly what I want and have to nudge programs away from making their own assumptions because either I wasn’t specific enough or I was so specific in one go that I broke the whole thing. A large part of this seems to be that AIs are, as we all know, Large Language Models, which means they neither think nor (this is important) see. Hallucinations and image drift seem to come about in part because of a loss of data that happens when you import an image, the AI translates it into backend text, and then translates that back into an image —just consider how many thousands of black bobs (hair) there are on the internet and you get the problem with letting a program decide what that means with a degradation in consistency at every step. So in order to nail consistency in imagery, what I’m doing at the moment is making a lookbook of consistent references that I can feed back into my prompts to generate images that will act as a sort of storyboard and that can then be animated into a scene.

But first, let me give you part one of a history of fucking around.

So, character design. Obviously I started with Sabriel, because she’s our protagonist, and I plugged a description of her that was basically cut from the book into an AI. The result was, bluntly, terrifying:

Could this be an Abhorsen? Ye-es. Is it Sabriel, a sweet, competent private school girl who doesn’t know how to talk to boys and is not emotionally ready to step into the boots of being a necromantic duke? No.

(This was all months ago and I’ve since lost the exact prompts, I’m just giving a history.)

Attempt two, which tried to at least make her look a bit more age appropriate, wasn’t as bad, but while Sabriel should look a bit unusual due to the death-bleached skin, she’s not even close to the scariest thing in the books.

Version three was still not great, because ‘really, really pale but still, like, cute,’ is a fine line to walk, and her costume was wrong, but she looked more like a protagonist that wasn’t incredibly villain coded, so I spent some time tooling around with her to see if I could get something resembling charter marks. This didn’t work, whether in the form of various prompts for a magical mark, taking the generated image and editing it myself and prompting that this was either a tattoo or makeup that had to be consistently incorporated, or anything else that I can recall, so I resigned myself to the notion that things like baptismal marks would have to be added in manually, which was still much more feasible than doing the whole thing myself.

She also ended up, entirely by accident as a result of something like ‘1920s bob’ in the prompt, with finger waves! Which is incredibly cute and not at all workable for someone who is either in a school uniform or filthy armor up to her knees in corpses. But she probably wears them when she’s attending garden parties in Corvere. Behold CW Sabriel.

Good enough to be going on with for the moment, said I, and moved on to being annoyed at how elements kept melting into each other and wondering if an animated style would be more forgiving of the mid and post processing that I’d have to do eventually. So I prompted this character ‘anime style,’ accidentally hit the video button, and ended up NOT with a line drawing version of the above, but with a grinning, skipping monstrosity that was unintentionally hilarious.

Whoops.

Okay, I think, maybe I need someone for her to interact with a bit. Enter CW Touchstone, in more shiny and less shiny v1 outfits. By the WAY, GARTH, do you know how much an outfit described as a kilt wants to autocomplete as ‘with a sporran!’ something you explicitly didn’t include???

This dude, for the record, is probably going to be supplanted with an imo cuter v2 Touchstone, mostly because (despite him being described as not exactly handsome) my personal headcanon departs from this in favour of the opinion that being the illegitimate son of a royal favourite probably means his father was pretty, and a teenage girl power fantasy means a pretty love interest.

Test shot of two teenage dorks pretending that they’re not not getting along with each other on their post apocalyptic camping trip. Looks like shit like a low budget made for tv series. AI seems to default to a stage blocking layout of characters always facing towards the camera, which is understandable but boring to watch. On review of the video, which is sort of meaningless vague interactions, I actually did manage to get the AI to keep the baptismal marks in place, but only by making them black, which I really disliked. (‘I’m just gonna have to manually edit this shit.’) Plus a bonus still of CW Touchstone reacting in a completely uncharacteristic way to being shown the Book of the Dead while not being a necromancer.

[ellynneversweet]

Tentative Touchstone costume. Red maybe needs to be a bit brighter. *Robert Baratheon voice* bow before king boo-boo the fool, ya shits!

[ellynneversweet]

After a lot of fighting with LLMs in an effort to express the concept of an outfit with layers that involves a lot of generating separate components and taking them away to edit together and then chucking them back into the idiot machine and then manually editing them again, a tentative Abhorsen costume that I’m about 70% happy with…

Aside from, okay, the bell handles mutating into leather because there is little to no reference out there on the internet for a weapons belt full of magical bells, so I’m probably going to have to mock one up in my trusty 3D program and feed shots of those in. And the lace that has spontaneously grown on her cuffs. And the keys, which are not quite dense enough for my liking but in other versions are far too dense, and in some particularly ridiculous versions are clearly turning into mass produced modern house keys. I’m still not sold on the brown leather but if it’s darker it could be good? The boots are also quite a bit heavier and thicker in the sole than your standard medieval boot (yes, even for war gear!), and medieval recreations that could work as references tend to feature people standing around in grassy paddocks, which makes them difficult to get detail from, but the books tend to describe a fairy sturdy kind of boot, and these people have clockwork and magical flying machines, so probably they also believe in orthopedics.

I have another, fancier surcoat sleeve waiting in the wings, but I haven’t done the edit/run back through the LLM, edit again thing on it yet. Abhorsen’s sword and the belt it hangs from don’t yet exist.

Also, a version of Terciel around the time of Sabriel’s birth! Looking a bit glum, the poor bastard, but that’s to be expected. As an earlier version of Sabriel’s face generation is likely to be Elinor, Terciel mostly just gets the same colouring as Sabriel.

[ellynneversweet]

Many moons ago I went back and forth on whether Touchstone’s gaudy-ass outfit merited Sabriel’s initial reaction (annoyance at it being much more flamboyant than her spare clothes) but now I am working on putting it together in a way that doesn’t look like a walking tinselled stop sign and good grief it’s…a challenge.

I’m also bearing in mind that I actually would like to leave open the possibility of a flashback to the royal court pre-breaking of the great stones, which means Touchstone’s look needs to be somewhat out of time with Sabriel’s but in such a way that I can make him look fashionable in his own time.

[ellynneversweet]

I’m currently writing a long post on how I’m putting together the Wyverley uniforms, but, in the meantime, have a Jacinth and Bunny.

[ellynneversweet]

Delicately, carefully asking an AI to please fuck up the fictional face of an AI generated character headshot to create continuity shots for Sabriel spending half her hero’s progress looking like she’s been smacked around the head by a firey primordial magical creature after she gets smacked around the head by a firey primordial magical creature. And succeeding first try without having to resort to the ultimate ‘in Minecraft’ nonsense prompt of asking for the thing I want but as theatrical/practical effects makeup! I call that a win.

Tentative reference image below cut for facial injuries etc.

She’s having a bad day and it’s gonna get way worse. I think she needs more bruising, actually. ‘Can barely see out of one eye’ levels of swelling. ‘Understandable why she’d do something really stupid like wake up the enchanted dude stuck to a ship’ levels of head injury.

[ellynneversweet]

In contrast to borrowing from Leighton, Winterhalter, Sargent et all, something referencing elements in the style of fan art that does numbers on tumblr produces these. Mogget is at least oddly charming.

[ellynneversweet]

When your blind date for dinner after nearly being murdered by an undead mud monster turns out to be an eldritch cat-who-isn’t-actually-a-cat whose topics of conversation are yes, your MIA father is dead and yes, that means that you are in charge of this zombie ridden hellhole.

Anyway. If you have a keen eye you might notice that this isn’t exactly a photorealistic-not-actually-a-photo, and it’s because I’m working on the principle that animating videos from images generated to look like the style of academic oil painters of the 19th century (ie, real but not hyper real) would both help hide artefacts, allow me to work around restrictions on depictions of violence, and give a slight fantasy feel to the whole. Frederic Lord Leighton is a fantastic reference to use in this case, because the way he uses light is very fantasy gleam but he didn’t suffer from same face syndrome.

At present I’m still mostly working on creating references for costume changes, which involves making a catalogue of images without distractions like backgrounds or props, which can then be used in conjunction with backgrounds and prompts to make something that will, theoretically, look consistent over time. In this particular case, a dress for Sabriel to wear at Abhorsen’s House, courtesy of the family Sendings and their boundless storage facilities. I’ll probably post a breakdown of how I generate costumes that look like what I want rather than a random hallucination based on ‘medieval’ later.