How I (used to) make Pre-ablo
Pre-ablo is my mod of the Diablo Pre-Release Demo that aims to be as faithful to the source while allowing it to be played from start to finish. I started it because the only existing mod of the Pre-Release Demo was Alpha4 and, while I enjoyed it, I wanted something with fewer creative liberties. Alpha4 is a new game entirely, I just wanted to play the Pre-Release Demo as it was without crashing!
However, I'm relatively new to the world of decompiling and x86 assembly. I figured the best place to start was the most obvious and I could bootstrap my way from there.
I started with binary patching. It was painful. The workflow looked like this:
Load DIABLO.EXE into IDA. I have a running IDA file where I annotate functions and variables based on Devilution
Identify the broken code. This often requires understanding the x86 assembly, mentally decompiling it to C++, and saving those annotations into the IDA file. Very rarely do I already know exactly what the code is doing, so understanding the code is a large part of this time.
Identify a fix. The best fix is one that doesn't add net new instructions so I tried to favor those. I'd also take some shortcuts which I regret doing later...
Identify where the fix would go. In step 3 I said that I didn't want to add net new instructions. This is largely thanks to the .EXE format. If I add new bytes to the file then all the offsets are now wrong and the game won't work. In these cases I had to repurpose dead code, and figure out how to jump in and out. Every jump is more work down the line...
Turn the fix into asm. My mind works in C++ so I need to mentally translate that into X86 assembly. This is tricky since I'm largely unfamiliar with x86 assembly (though I get better every day)
Turn the asm into machine code. Oh boy this sucked. This sucked so hard. Machine code is meant to be read by the processor, not humans! It ends up being incredibly terse! In addition, x86 has some peculiarities: it uses a multibyte encoding and has a lot of weird edge cases. At one point I switched to using Ghidra to do this for me (but this created its own headaches since I can't use my IDA annotations in Ghidra...). Also I needed to calculate relative offsets to the current instruction pointer...
Insert the machine code into DIABLO.EXE. I did this with a hex editor. By hand. If I made a mistake I'd have to start over.
I made this reproducible by encoding the binary differences using vcdiff. That way, I could take a fresh DIABLO.EXE and reconstruct Pre-ablo by applying the vcdiff patches in order. It also separated the logical changes into a list of discrete patches; changing one patch (usually) had no impact on the other.
This sucked but it worked. I used this approach until v0.4 when I replaced it with something better...
(The "easy way out" was to pay for IDA Pro. Which is several hundred US $. No thanks.)