The blog post reminds me of similar efforts with Shockwave. There's people building decompilers, and runtimes, and if you join enough Discords, you will notice the people in them are cross-contaminating between communities if you will, they share insights with one another, in their efforts towards specific goals. They're hyper focused on making one game or another come back. There's Habbo Hotel, and Coke Studios, as well as other games.
The Coke Studios effort is interesting because there were no "private servers" developed at the time, unlike Habbo which had many, and there are Shockwave Xtras that no open runtime supports currently.
There's several attempts at a full runtime as well, that run in-browser.
Projector Rays (decompiler) really was the biggest release to date, and recently people have been really hacking at it, to some extent AI has helped to reverse engineer bytecode far as I can tell.
For anyone curious, one of the runtimes is called DirPlayer:
I have a lot of respect for little communities where people gather to work obsessively over a goal like this. It's fun to see projects like this that would never see this much effort applied if not for the passion of a few dedicated people.
Many of the smartest people I've known got their starts in little communities like these: Obsessively learning new things that would normally be beyond their academic or career level because they need to know something advanced to move their passion project forward.
Its what started me on the path of my career really. Understanding how systems work. I still enjoy finding cheats in offline games for myself just to see how far I can push game engines.
I'm indirectly involved / hear about a project that buys up old feature phones, mainly from Japan, to try and find ones that have data for old imode games on them, notably a FFVII spinoff called Before Crisis. It's difficult because they would release the game in separate data packs, the idea being people can remove data packs for parts of the game they already played to save space (also a feature on modern smartphone games). But since the servers are long gone, they need to find phones with the data on them to extract it.
Where can I boot up Coke Studios? Enjoyed that game as a kid, including making music (which was so friggen cool! They had tons of samples you could just drag and drop into a timeline).
Edit: fun memory. I figured out how scamming works at some point. IIRC (and I may not) I set up a minigame in my studio, and contestants had to give me their furniture to participate in a game. Since furniture could stack weirdly, people made all sorts of crazy mazes that were actually quite difficult to navigate due to the fixed isometric perspective.
Each stage required handing over a more expensive item than the last. At some point someone handed over a super rare/expensive piece of furniture.
Not gonna erase that. ACTUALLY I think I figured out how to clone items due to a bug. Or mine infinite DBs. Used that as cover to get someone to give me their item. Was a long time ago.
Anyways, a user handed me their super rare item and I then gave them a stern warning on how this is a common scam and I just stole their item(s). Then I handed everything back over to them and they profusely thanked me. lol
Years ago I was searching for why emulators don't recompile to native bytecode and it seemed like the consensus at the time was that JIT etc was faster. I can't find it now, but IIRC I remember replies from maintainers saying it was a bad idea.
What changed since then, since this seems to be a trend? Is this more for more modern systems that have more static code?
Static code. Also, very fine details of the machine because less important.
When emulating a NES (for example), you really have to emulate every register, how registers change, and also weird effects like instructions that take longer to read or write values, as games rely on that stuff. Once you have modern systems where much of the code was originally C, it becomes less important to ensure every register has exactly the right values when a subroutine finishes in most cases, you can rely that (most) of the code follows standard calling conventions.
The problem with NES emulation (having written a fairly bad one) is it can be hard to predict when you need to be careful, particularly in the presence of self-modifying code and interrupts. If it was easy to figure out, emulators wouldn't have to keep such careful track of this stuff either.
Something like a SNES will always have the same hardware no matter what (okay, leave out the expansion chips inside cartridges, or fold them into "same hardware").
Something like a PC will have any old random shite you care to plug into it.
Something like an Xbox 360 is kind of halfway between.
With a SNES you can rely on the fact that a vertical blanking interrupt will happen 50 times a second no matter what, and you get so many instructions before your video output is halfway down the screen no matter what, and then if you want to play silly games like changing a PPU register to dick about with the screen mode you can. You have to take into account things like it taking more time for certain types of memory access like if you cross a 256-byte page boundary, but since when you get down this far you're doing it all in assembler you can place your code and data exactly where you want to get exactly the behaviour you want.
When you emulate this, you must emulate every aspect of this behaviour correctly or things will fall apart. Think about things like the "open bus bugs" that trip up emulation - some games rely on silly tricks like expecting the capacitance on the pins of the chips to hold a value like a crazy single-byte DRAM for a moment while they look somewhere else. If your emulator just goes "okay no devices are enabled so the bus is simply pulled high to 0xff" that will fail.
With a PC you've got a wider variety of hardware, and a deeper level of abstraction. You don't have to care about flipping registers at just the right time, you just throw some textures and polygons at the GPU and let it do its thing. It'll tell you when it's ready for more. This is why you get the same experience between your AMD GPU, your NVidia GPU, your Intel GPU, and hell why not just have software OpenGL too (because it's slow but that's how we used to do it in the olden days).
So if you wanted to run a SNES game on modern hardware you'd be better off emulating the hardware directly, bugs and all, and then running the "real" game binary.
But if you wanted to run something like an Xbox 360 game - written using "modern" development techniques, where a lot of the hardware is abstracted away behind drivers and libraries - you might well just break it down into functional blocks of code, rewrite them for your native CPU, and shim the calls to the libraries to talk to the ones you have on your native system.
A bit of code that rotates a set of vertices and flings them at the GPU will look pretty much the same in C on any platform, but it'll compile to very different assembly language. That's okay though! If you understand how the instructions relate to the data you can just kind of translate one set of opcodes to another and expect it to be more-or-less right. You can even (kind of) do this automatically!
These projects are really neat. Last week, I was able to build (and play) the Xbox 360 release of Sonic Unleashed on a couple of ARM SBCs using Sonic Unleashed Recomp.
You take an Xbox game designed to run on an Xbox 360, a 64 bit PowerPC system and decompile its binaries back into source code. You now have the ability to modify the game as well as port it to other systems and architectures such as Windows on X86_64 or Linux on ARM64.
It's more nuanced than that; the approach you're describing is usually called "decompilation."
The difference is how far one goes in hoisting the "source code;" in this "recompliation" approach the source code, while C++, is basically an IR (intermediate representation) between the original game's assembly and a host platform, and the hardware itself is emulated (for example, the original architecture's CPU registers are represented as variables in the host architecture's memory). The machine code is translated to C++ using a custom tool.
In a "decompilation" approach the game logic is converted (using a decompiler, like IDA or Ghidra's) back into something which resembles the original source code to the game itself, and the source code is usually hand analyzed, marked up, rewritten, and then ported across platforms. The product is something that attempts to resemble the original game's source code.
Of course, they lie on a continuum and both approaches can be mixed, but, while they both involve C++ in the middle, the process is starkly different. Recompilation is much more copyright-friendly, because in many implementations only the modifications are distributed and the original binary is translated by the end user (who owns the software/a license to it), whereas decompilation produces an artifact (source code) which is a derivative work encumbered by the original software's license and generally should not be distributed.
Seeing this [1], I thought it was something related to taking assembly instructions in the original code, emitting C statements that match the instruction, and then compiling that C code.
I wonder whether this "recompilation" technology could be used to run Windows software on Linux, as an alternative/complementary technique to Wine.
Back in the day (year 2000, until 10-15 years ago) we had Project Odin to dynamically translate Windows software to run on OS/2:
https://github.com/netlabsorg/odin32
Very nice. It’d be interesting if recomps of some games that were originally for both 360 and PC become the superior experience… Fable 3 and the whole Games for Windows Live nonsense the PC version had come to mind.
Crazy that it has taken so long. I understand these are small teams trudging off in the dark but you could have imagined there would have been more eyes on this and hands tinkering in the time when the Xbox 360 was being sold than today. Right at that same time in history the iPhone was getting cracked basically every iOS release, sometimes by teenagers. Seems like there were a ton of hackers around back then.
36 comments
The Coke Studios effort is interesting because there were no "private servers" developed at the time, unlike Habbo which had many, and there are Shockwave Xtras that no open runtime supports currently.
There's several attempts at a full runtime as well, that run in-browser.
Projector Rays (decompiler) really was the biggest release to date, and recently people have been really hacking at it, to some extent AI has helped to reverse engineer bytecode far as I can tell.
For anyone curious, one of the runtimes is called DirPlayer:
https://github.com/igorlira/dirplayer-rs
Many of the smartest people I've known got their starts in little communities like these: Obsessively learning new things that would normally be beyond their academic or career level because they need to know something advanced to move their passion project forward.
But they've made really good progress in recent years, to the point that the game is now playable. Not sure if it's complete, but it's playable. See e.g. https://www.oldschoolgamermagazine.com/work-begins-on-restor... for info.
Edit: fun memory. I figured out how scamming works at some point. IIRC (and I may not) I set up a minigame in my studio, and contestants had to give me their furniture to participate in a game. Since furniture could stack weirdly, people made all sorts of crazy mazes that were actually quite difficult to navigate due to the fixed isometric perspective.
Each stage required handing over a more expensive item than the last. At some point someone handed over a super rare/expensive piece of furniture.
Not gonna erase that. ACTUALLY I think I figured out how to clone items due to a bug. Or mine infinite DBs. Used that as cover to get someone to give me their item. Was a long time ago.
Anyways, a user handed me their super rare item and I then gave them a stern warning on how this is a common scam and I just stole their item(s). Then I handed everything back over to them and they profusely thanked me. lol
> cross-contaminating between communities if you will
The positive term for this is cross-polinating
Some exciting stuff has already happened in the week since I wrote the interview. Here's Viva Pinata: Trouble in Paradise: https://www.youtube.com/watch?v=Kih42KlocII
What changed since then, since this seems to be a trend? Is this more for more modern systems that have more static code?
When emulating a NES (for example), you really have to emulate every register, how registers change, and also weird effects like instructions that take longer to read or write values, as games rely on that stuff. Once you have modern systems where much of the code was originally C, it becomes less important to ensure every register has exactly the right values when a subroutine finishes in most cases, you can rely that (most) of the code follows standard calling conventions.
>and also weird effects like instructions that take longer to read or write values
The percentage of a code in the ROM that depends on such things is a small percent.
I also think tooling got better, and with the advent of AI engineering it made all of this a lot more accessible.
Something like a SNES will always have the same hardware no matter what (okay, leave out the expansion chips inside cartridges, or fold them into "same hardware").
Something like a PC will have any old random shite you care to plug into it.
Something like an Xbox 360 is kind of halfway between.
With a SNES you can rely on the fact that a vertical blanking interrupt will happen 50 times a second no matter what, and you get so many instructions before your video output is halfway down the screen no matter what, and then if you want to play silly games like changing a PPU register to dick about with the screen mode you can. You have to take into account things like it taking more time for certain types of memory access like if you cross a 256-byte page boundary, but since when you get down this far you're doing it all in assembler you can place your code and data exactly where you want to get exactly the behaviour you want.
When you emulate this, you must emulate every aspect of this behaviour correctly or things will fall apart. Think about things like the "open bus bugs" that trip up emulation - some games rely on silly tricks like expecting the capacitance on the pins of the chips to hold a value like a crazy single-byte DRAM for a moment while they look somewhere else. If your emulator just goes "okay no devices are enabled so the bus is simply pulled high to 0xff" that will fail.
With a PC you've got a wider variety of hardware, and a deeper level of abstraction. You don't have to care about flipping registers at just the right time, you just throw some textures and polygons at the GPU and let it do its thing. It'll tell you when it's ready for more. This is why you get the same experience between your AMD GPU, your NVidia GPU, your Intel GPU, and hell why not just have software OpenGL too (because it's slow but that's how we used to do it in the olden days).
So if you wanted to run a SNES game on modern hardware you'd be better off emulating the hardware directly, bugs and all, and then running the "real" game binary.
But if you wanted to run something like an Xbox 360 game - written using "modern" development techniques, where a lot of the hardware is abstracted away behind drivers and libraries - you might well just break it down into functional blocks of code, rewrite them for your native CPU, and shim the calls to the libraries to talk to the ones you have on your native system.
A bit of code that rotates a set of vertices and flings them at the GPU will look pretty much the same in C on any platform, but it'll compile to very different assembly language. That's okay though! If you understand how the instructions relate to the data you can just kind of translate one set of opcodes to another and expect it to be more-or-less right. You can even (kind of) do this automatically!
The difference is how far one goes in hoisting the "source code;" in this "recompliation" approach the source code, while C++, is basically an IR (intermediate representation) between the original game's assembly and a host platform, and the hardware itself is emulated (for example, the original architecture's CPU registers are represented as variables in the host architecture's memory). The machine code is translated to C++ using a custom tool.
In a "decompilation" approach the game logic is converted (using a decompiler, like IDA or Ghidra's) back into something which resembles the original source code to the game itself, and the source code is usually hand analyzed, marked up, rewritten, and then ported across platforms. The product is something that attempts to resemble the original game's source code.
Of course, they lie on a continuum and both approaches can be mixed, but, while they both involve C++ in the middle, the process is starkly different. Recompilation is much more copyright-friendly, because in many implementations only the modifications are distributed and the original binary is translated by the end user (who owns the software/a license to it), whereas decompilation produces an artifact (source code) which is a derivative work encumbered by the original software's license and generally should not be distributed.
[1] https://github.com/N64Recomp/N64Recomp
Back in the day (year 2000, until 10-15 years ago) we had Project Odin to dynamically translate Windows software to run on OS/2: https://github.com/netlabsorg/odin32