“Super Mario Spacetime Organ” is a short video by Chris Novello. It depicts the NES classic Super Mario Bros. piped through two devices, an illucia patchbay, which Novello invented, and the multitouch Soundplane created by Madrona labs. Using the Open Sound Control protocol (a modern MIDI alternative) Novello uses the hardware to directly manipulate the game’s state in memory.
It starts off simply enough. Novello plays the game for a bit before stroking the Soundplane to make Mario hover in mid air. He begins physically patching parts of the memory into other parts and the game goes wild. Then the mallets come out…
“I was exploring the RAM state of Super Mario Bros., and I discovered that the game could be controlled in amazing ways by directly manipulating the memory,” says Novello. “It is a game we all know super well, so it was really fascinating to experience it in an entirely different way.”
The game in the video is running on an emulator that runs Lua scripts. By writing OSC hooks into the scripts, software from outside the game can read and write to the game’s memory directly. By mapping the OSC addresses in the scripts on to the illucia’s OSC addresses, Novello can change settings and variables by fiddling knobs and flicking switches.
The process isn’t that different from how the Game Genie alters the ways games are played, except that Novello isn’t in it to win. “I never used Game Genie for extra lives,” he says. “I always played as the bosses in Mortal Kombat for the 30 seconds or so before the game crashed!”