Meet Chris Novello, alias Paper Kettle, who shows in the video above how he can manipulate Super Mario on the NES with his glitch instruments. The smaller one is a USB device with arcade buttons called Illucia made by the man himself, and the other, multitouch thingy is a Soundplane.
The technical details on the codebending are described in detail on Paper Kettle's channel:
I begin by playing the game as one normally would, just using buttons on illucia.. but I also have access to the game's memory, so I use the Y axis on the Soundplane to alter the value in the memory address that determines Mario's Y position onscreen. This is how I make Mario fly and hover during the playthrough.
Also, before I start playing, notice that I flip a switch on illucia. This triggers recording — not video, but actually recording the entire memory state of the NES for each game frame. Because I'm saving the game ~30 times a second (and keeping log of all saves) I'm able to go back to any moment in Mario's life. Sort of like a Super Mario time machine.
So then I use the X-axis of the Soundplane to sweep through the timeline of Mario's universe. Not only that, but the Soundplane is multitouch, so I use a second finger to specify start and endpoints in a playback loop. This is similar to the way samplers and granular synths work, but for recordings of the entire memory state of the NES rather than audio data.
Make sure to watch the whole video, it gets crazier as Mr. Novello sends more and more data from the Illucia to poor Mario's world.