#esrgan

[beetlebabber]

ai upscaling is WILD

I have begun the endeavor of upscaling my BJ screenies FOR SCIENCE... they're from my 2160p mkv's anyways but they're still a bit grainy for my preferences so

BEHOLD THE JUICE IN UNHOLY HD GLORY

ヽ(ಠ▄ಠ)ﾉ EEEEEEEE

[nomorepixels]

What software do you use to upscale your images?

Heyas, big fan of your posts. I use mostly use ESRGAN via Cupscale for upscaling, and StyleTransfer/StableDiffusion WebGUI for the repaints. Manual fixes are done in Photoshop.

A variety of ESRGAN models are used, the most usual ones are: 1) Fatality_mkII 2) NMKD-UltraYandere_300k 3) BigFArt_Detail_300000_G 4) BigFace_v3 5) ArtStation1337_bloom 6) SmolFace 7) ArtClarity / ReFocus_v3 8) DoubleDetoon interpolated with ReDetail_v2 9) realesrgan-x4minus / BSRGAN 10) GainRESV3 whenever I need to use a non-pixelart model

[toiletsofbuild]

Duke Nukem Enhanced Resource Pack - first release file

A collection of mods for the ultimate Duke Nukem experience

I forgot to post this. It’s essentially an all in one package of which the upscale pack is made a part of

[ikanmampus]

The dangers of anime in your machine learning

 I have been experimenting with machine learning image upscaling via a program called ESRGAN, a worthy successor to the famous Waifu2X programs used by many to upscale anime art, at twice the image size. The default model ESRGAN came with, while good, was not particularly good with cartoony/anime artworks. This all changed when kingdom akrillic released his manga109 model, which exposed the capabilities of the upscaler with a well-trained model. This opened the floodgates to other people trying machine learning for themselves.

The first model I trained tried to replicate Waifu2X’s dataset which was less manga, and more towards manga/anime styled artworks. The easiest method was to grab the Danbooru dataset that the folks on the web have collated, containing hundreds of gigabytes of data that required a special torrent client to download. Unfortunately, not all of the pictures are of good quality, so the images had to be curated into a proper dataset with only the best. I called my attempt WaifuGAN in honour of Waifu2x, and the main content of the dataset. The first version of WaifuGAN had JPEGs in the dataset, which resulted in learning the noise present in those pictures, and using it as output in the upscales, something definitely unwanted. The later attempts removed JPEGs and only used PNGs, which had better results. It seemed to produce comparable, if not better, results than Waifu2x... for the moment. Then I realised some artifacting happening in the upscales:

Instead of thin or thick lines, certain parts of outlines upscaled to double lines instead. This problem was also present in the manga109 model. I tried to troubleshoot where the model learned this behaviour by looking at the dataset. And why Waifu2x rarely had this problem in its upscales. The I looked at the lowres images that were generated for the training dataset, and managed to come to a theory why this happened: rim-lighting.

Consider the illustration above, which at first glance makes for a good input image for training. Now observe the four parts highlighted below at 4x zoom:

For clarification, when preparing a training dataset, a pair of images called HR (Hi-Res / original sized image) and LR (Low-Res / downscaled images are created. The computer basically has to learn how to convert the LR picture into the HR picture as closely as possible. For a 4x upscaling model, the LR must be 1/4 of the HR size. The 1 pixel wide outlines used in the HR is actually problematic enough, but if you observe the images above, there are also another 1 pixel wide line alongside the outlines with a clearly brighter colour before you see the colour of the area fills. This effect is called rim lighting because you add some brightness to the edge of objects. In traditional painting or 3D, this effect is achieves with usage of glow/highlight near the edges of the objects. This adds more contrast and helps define the shape of the subject matter. in anime art, it is often emulated as a line/strip of brighter colours on the inside of highlights. So how does this stylised effect cause trouble in machine learning situations?  Consider the LR tile generation of the tiles shown above:

This were generated using 4 different downscaling methods:

Nearest neighbour

Bicubic smoother

Bicubic sharper

Photoshop preserve detail 2.0

Observe how the outlines and rim-lighting were scaled. In the places where rim lighting is thick, it downscales correctly and is visible in the LR images. But for the ones that are 1 pixel thick, downscaling it to 25% size results in the highlight merging with the outlines (!) in most cases and this will be used to teach the machine learning that such lines should be upscaled with a highlight added. Most models trained on anime paintings will exhibit the behaviour of adding highlights to some outlines, or in worse case scenarios, result in the lines becoming double-lines. To be succint: Garbage In, Garbage Out.

So what images would make better input? Stills taken from actual anime, or illustrations that mimic this style are good. They usually don’t resort to rim lighting because flat colours are preferred. And if rim lighting is indeed used, they tend to be reasonably thick as well. Otherwise, illustrations that either have thick outlines, or do not use outlines at all are better. This is important if you need your model to be responsive to colour gradients instead of the flat colours that permeate anime screenshots.

On another note, also avoid ‘sketchy’ illustrations such as the one above because the line quality is inconsistent, and sometimes outright have double outlining in the image itself, which will mess up with your machine learning data pronto.

Unless of course, you were wanting these double lines as your goal. Then, everything discussed here is moot. :P

You can find ESRGAN models trained by me and a bunch of brilliant people over at the Upscale Wiki.

[adventurerooms]

A collection of King's Quest backgrounds upscaled with ESRGAN AI Neural Net https://www.facebook.com/photo.php?fbid=10156759302071605&set=pcb.1453894064745313&type=3&theater&ifg=1

[nomorepixels]

ESRGAN models are not trained the same, each have a specific objective they wish to achieve. In terms of models for upscaling pixel art, I gravitate towards the paint-centric models compared to anime, as they trade off line sharpness in favour of blending and more organic shapes. This is a small showcase how differently each model attempts to interpret the base image, including what happens when non-pixelart models upscale pixel-art, as well as what happens pixel art is anti-aliased first before feeding into a non-pixelart model. Finally, is the realesrgan model commonly used by people that post in reddit & YouTube to show ESRGAN upscaling.

[nomorepixels]

How are the upscales done?

Since I’ve gotten requests from reddit as well as on this blog itself, I’ve decided to share my workflow on how I upscale most of the images featured here. Roughly, the main program I use to upscale is ESRGAN, an open-source Python script that uses trained neural network models to process images. I also use IEU Winforms to automate most of the upscaling work (though a lot of people have migrated to Cupscale). Finally, Photoshop is the image editor I use to composite the upscale results.

So let’s get to steps involved. First and foremost is to source the image. I usually either grab them myself from whatever games I have, or get a copy from other sites such as Hardcore Gaming 101, Mobygames etc. So here’s today’s sample image sourced from Retronaissance: The Blog! is a shot from Capcom’s arcade obscurity Red Earth:

I always try to find snapshots stored in pristine pixel art form instead of resized jpeg pics whenever possible. This is because some models are very sensitive to noise and distortion.

Now firing up IEU Winforms, and you can see I have a bunch that are my go-to models when I’m upscaling specifically for No More Pixels. Since this blog is about transforming those shots into supposedly artsy-fartsy canvases, I try to avoid models that are catered to realistic images. Most of these models are obtainable via the Upscale Wiki.

After selecting a bunch of models to filter the image through, I execute the upscaling and get back the results from each model. You may now be asking why do I need that many different upscales in the first place? Well sadly, there is currently no ‘super’ model right now that can produce the best image for all occasions. Some models are good at doing certain things but bad at others. I’ll share some examples:

Normally I start with using Unholy FArt as the base image, and usually followed in conjunction with Fatality Mk II. You can see how different each model interprets various parts of the image. Fatality Faces has the best facial reconstruction, while ArtStation1337 Bloom has the best debanding capability out of all of them (observe Tessa’s fabulous balloon pants). So which model should we choose? The answer is: as many as you like! I take various parts here and there that will be combined together later.

Now let’s observe the horney T-Rex-kun. While organic, the dino has a more geometric feature, so the colour-blending should be less of a priority while having sharper features is better. You can see BigFArt Detail trying to generate some details for the background. BigFace which was trained on an anime ladies dataset made the eye into a T-Rex-chan. FArtFace has the best result here, keeping the lines and features sharp.

The last example deals with geometric non-organic shapes like this pillar. Models that do blending are an obvious no-no. Fatality Mk II looks good, but perhaps too sharp for a background element, while BigFArt Detail is adding in noise. This leaves NMKD UltraYandere as being accurate to the original shape while not being overtly sharp.

Now let’s get compositing. My personal routine is to start with UnholyFArt as base and immediately followed by Fatality MkII as a Darken layer. This gives me Fatality’s sharpness while removing the halo effects it usually produces.

Since I decided to use FArtFace on T-Rex-kun, using the Quick Selection tool helps to quickly set up a mask just for the dino.

Moving on to Tessa (and her cat), I’ll start with BigFArt Detail as base. This keeps detailings sharp.

Then masking in the blending from ArtStation1337 Bloom and transferring the details from Fatality Faces onto Tessa’s mug like one would with those transfer stickers from your childhood.

Now restoring NMKD UltraYandere to the pillars, and BigFArt Detail at 50% opacity to other parts of the background. CleanPixels for the UI and Deviance for some parts of Tessa’s portrait at the top left.

Finally, you might notice that there’s some bits of chunky pixel in the water below T-Rex-kun, which means it’d maintain itself as its angular shape for all models... save one. I originally trained BigFace to dream up hair details whenever possible, but it turned out to be a bad idea because it failed to discern what is hair and what is not. Here the bug becomes a feature, because it’s the only model deblocking that water. Beggars can’t be choosers.

The final composition at last. Normally I’ll add ArtStation1337 Bloom at the top as a soft light layer to increase the colour contrast of the composition. Before exporting, the proper aspect ratio of the original shot must be determined. Here Capcom is using tall pixels for their CPS3 games. So that 384 x 224 internal resolution is actually displayed on a standard CRT arcade monitor ratio.

Researching a bit shows it’s a bit more towards 98:72 than 4:3 standard. That means my final image dimension should be 1220 x 896 pixels. Resize while preserving details, export to web and.... done.

Click here for the full-size image.

Hope this demystifies the whole process, and perhaps spark some interest for anyone else to do upscales of their own. Cheerios!

[toiletsofbuild]

Duke Nukem 3D Upscale Pack 1.3 addon

Upscale Pack for Duke Nukem 3D including Life's a Beach, Nuclear Winter expansions and World Tour - for use with EDuke32, DukeGDX, RedNukem or Raze