SwiTAS: Making a TAS toolkit for the Switch
Created Sunday, May 17, 2020
Speedrunning
A few months ago, I got myself a Nintendo Switch, hoping to possibly become a proficient speedrunner at a popular and satisfying speedgame. After one run (yes, one), I discovered that I had very little of the necessary stamina to grind for the tens and perhaps hundreds of hours that would be needed just to get me a decent time. Oh well :)
I noticed a small, rather inactive channel near the bottom of the SMO Discord server, one about... TASing
Now, I had encountered TASing before. Like any consumer of speedrunning content, I invariably found entertaining TASes. I had not thought of ever TASing before, however, speedrunning seemed like the natural choice.
The more I thought, however, the more I realized that TASing was a distillation of what made speedrunning great for me. The theorycrafting, the glitch usage, the bypassing of intimidating barriers with complex and ingenious loopholes. Upon finding my interest with speedrunning myself waning, I came into that small channel, the only hub of Nintendo Switch TASing at the time, and tried to make a place for myself. I am a doer after all.
In classic me fashion, I begun by making a loud a verbal presence on the server. ramble, if you will :p. I was semi-useful, I suppose. We migrated to a new independent discord server so we could expand outside of the SMO community and I made a crappy logo for the server. We kept it, which is pretty funny :)
I then tried to pursue my real passion, software development. I first tried to make TASing more accessible by integrating it with a Android app (unsuccessful, the desyncs proved too much). I then tried to develop a TASing solution for a cheap raspi with a fellow dev, MonsterDruide1 (the documentation was nonexistent for creating a USB hid device on linux, go figure). I eventually found myself back where I swore to leave, the homebrew solution. A few weeks of purchasing a hackable switch and setup later, development began on what I aimed to be the Bizhawk for the Switch.
You might be wondering, how could I possibly attempt to create an emulator-based TAS toolkit for the Switch, a physical piece of hardware? The plan is incredibly convoluted and untested as of now, but that is sure to change in the upcoming weeks.
Supercharging Switch TASing
There are a number of aspects of emulators that need to be implemented to allow the illusion of a competent TAS toolkit. They are:
- Frame advance
- Memory viewing
- Savestates
The first is the most simple. Using system calls in a privileged homebrew executable known as a sysmodule, one can get access to event listening. The event important in this case is vsync, which triggers every time the screen is drawn to. As it happens, this provides a convenient way to measure the start of game frames, as most games run their frames at the same rate as the screen is refreshed. Using some debug calls for the HID service, which is tasked with handling controller input, it is trivial to inject arbitrary input into the running application. finally, system calls also allow easily debugging the running application. When a program is being debugged in the context of Horizon, the Nintendo Switch operating system, its execution is halted. While it allows a vast amount of power over the application, which I will abuse for memory viewing, the important part here is that the application is paused. To frame advance, the program simply removes a debug handle on the application to resume execution, waits for a vsync event to signal a frame has just passed, then quickly obtains a new debug handle on the application to pause it again.
Memory viewing is difficult for the reasons of networking. While obtaining the game memory during a frame is trivially simple, system calls once again come to the rescue here, synchronizing the data obtained with the PC client (remember, the TAS client is not being run on the same system the game is executed, there is a non-trivial amount of work required to keep the two aligned) is much harder. Using memory mapped files makes the task somewhat easier and the brunt of the work is offloaded to a competent hex editor which supports detection of file changes.
Savestates?
Savestates: the most outlandish idea here. I saved the best for last, folks! The idea depends wholly on the idea that naturally executing a game from a returnable point, such as a savefile, should eventually replay the game back in real time to the location of the savestate. Piggybacking on nature's savestates, I decided to implement a system where savestates were loaded based on the visual similarity between themselves and the current game's frame.
Two concepts are necessary to make this work. Something I termed Savestate Hooks and Savestates. Savestate hooks are best placed by the user in a place following one of the game's savefile loading points but before any input is to be entered. Good examples of this are the introduction cutscenes to kingdoms in Super Mario Odyssey or the fade-in to black upon pressing play on a Super Mario Maker 2 level. It is important that these frames themselves have visual individuality. A static menu is not usually enough, unless the menu has noticeable idle animations, so it is best to choose a frame with motion that can be returned to. SMO kingdom cutscenes can be returned to with a load of a previous savefile, perhaps with all the previous kingdom's moons collected but prior to placing cappy on the globe. SMM2 level fade-ins can be returned to by leaving the level and pressing play on the level select screen.
The TAS software needs to know whether the frame it is currently on has an acceptable visual similarity to the savestate hook that needs to be loaded. One possible solution is checking the two images pixel-by-pixel. I personally have doubts about this method. The Nintendo Switch is a piece of hardware that runs without any care for TAS-like repeatability. As of now, I don't believe the Switch can be trusted to return frame data that is identical in all 1280 by 720 pixels. The switch performs various scaling and blending operations many times per second which may influence the pixel data by even the tiniest degree. I don't want to take that change. Instead, I settled upon using dHash.
Traditional cryptographic hashing attempts to create hashes as dissimilar as possible for even the most similar inputs. This brings us to the same problems of checking pixel-by-pixel. After some research, it seems image-specific hashing exists to produce the opposite effect: producing more similar hashes for more similar inputs. This class of algorithms is known as perceptual hashing. After some deliberation, I decided upon dHash. While being slightly less accurate then pHash, its beefier cousin, it does the job more quickly. Using hamming distance on the hash returns a number. The smaller this number is, the more similar the images are judged to be. To weed out false positives, values of between 0 and 3 are given to the user to decide rather than accepted outright. Through this method, the loading of a savestate consists of manually navigating to the returnable point in realtime, loading a savefile or otherwise, proceeding by frames when moderately near to the frame, the program checking the dHash of each game frame as it goes along, and the user being prompted when the program believes it has encountered a reasonable match. Upon the user affirming a frame, the TAS has effectively been rewound and the rerecord count is incremented.
Savestates, without the hook, take advantage of the rewinding savestate hooks have accomplished for it. Savestates are loaded by loading the nearest savestate hook prior to it and simply running back each frame up to the savestate's point in realtime, with the input supplied by the user when the savestate is created. With this method, savestates are invalidated when the inputs between it and the nearest savestate hook are changed, thus making it a more of a nicety for the user.
Using these three core TASing principles, I hope to create a usable TAS toolkit for Nintendo's most exciting game console yet! My progress has been steady, but I hold out hope that the Switch's fantastic lineup can finally has representation in the TAS scene before the end of 2020! If you wish to join the Nintendo Switch TASing scene yourself to be a part of Switch and TAS history, make sure to join the discord and join our community!