Mario Maker 2 API
Created Thursday, February 10, 2022
Mario Maker 2 is a game I’ve played for years, since launch, and I’ve always been interested in putting my skills in programming to the test here. I’ve finally done just that. Using the incredible NintendoClients made by Kinnay, I, and individuals from the Ninji Speedrunning discord, 0% discord, and Kinnay himself, we now have an entirely free and public way of accessing data previously only available to the switch itself.
link Background
Nintendo Switch Online games tend to use a specific library with a specific protocol to communicate with Nintendo’s servers. Since the WiiU, this protocol has been NEX. NEX, partially due to the incredible success of many of the games using it like Mario Kart 8 and Splatoon 2, has been extensively documented, to the point where it is fully possible to spoof both the client and server. The Pretendo project has been working on a WiiU NEX server for years, and alongside it NintendoClients, a NEX client, has been becoming more stable and safe to run on your personal account.
In October 2021 I finally took the plunge and started using NintendoClients for a dream I had years prior: to make all Mario Maker 2 data easily accessible to computers. After watching the project for years, debating whether to purchase a digital copy of the game and put my Switch certificate on the line, Tatiaus, a talented kaizo player, contacted me about using the API for the 0% team as I had mentioned it in passing earlier. Once I realized there was interest past my own goals I bought the game and began working. On the 13th, I released the first build, publicly available with no login required.
link First release to a small group of Switch modders
First release on discord
link Word gets out
First, the Ninji server used it to round out their leaderboards, making them 100% accurate for the first time ever. The features I considered most important, the level_info and level_data endpoints, went unused. Not many people outside the Ninji Speedrunning discord knew this existed. Slowly, the API got around, but it stayed a novelty. One of the funniest misconceptions I saw was the guess that my switch was running 24/7 and downloading levels for me. The truth is that this was running on a linux server, using a complete recreation of the NEX client. No switch was involved, except for the various credentials the NEX protocol needs to authenticate with the server.
link Interest peaks
In November a kaizo player well known for performing incredibly difficult "trump jumps", jumps in the game which were pixel perfect, came to me about using the API for a level viewer he had designed. In time Ji Xiaomai released his viewer, and that was the spark that ignited the entire community. Suddenly now anyone could view any level without the game and even without a switch. Tricky levels that were only possible due to information only the creator could know were exposed. Suddenly now no level could hide secrets.
After the release of this viewer, I ported it to C++ and made it portable enough to run in the browser, so now it wasn’t even neccesary to download the app to your PC. With time came new incredible viewers, including one that had animated textures. The count of level viewers was now 5, including the continuation of an old level viewer by streamer and programmer Shoujo and one made from the ground up by Ingram. The community had collectively come together to tear the game apart, utterly destroying the roughly 1 level viewer made for the first game. And in another impressive effort, almost every field of every endpoint was determined, so now stats previously thought impossible to obtain, like the boos on a level or the k/d ratio of a player, were accessible. All this hype has generated an impressive amount of traffic as well:
link Web traffic after release
Web traffic after release
link Archival
Through all this, the API has stayed public and free. Information should always be as accessible as possible and designed to be as easy to parse as possible by people and computers alike. While I am putting my personal Nintendo Switch Online account on the line, this tool is invaluable to research about the game. So I can proudly say, while Youtube made dislikes private, we made Mario Maker 2 dislikes public.
The logical conclusion to this venture is to pull as much data as we can. So I did the only reasonable thing, went out and bought a 14 terabyte hard drive in order to download Mario Maker 2’s entire database. Now it might seem impossible to simply download everything, we can request up to 300 random levels at a time through endpoints like search_endless_mode, but statistically we will recieve less and less unique items over time, almost guaranteed to take years to download 100% of available levels. Nintendo, however, made this easy for us:
1def course_id_to_dataid(id): 2 course_id = id[::-1] 3 charset = "0123456789BCDFGHJKLMNPQRSTVWXY" 4 number = 0 5 for char in course_id: 6 number = number * 30 + charset.index(char) 7 left_side = number 8 left_side = left_side << 34 9 left_side_replace_mask = 0b1111111111110000000000000000000000000000000000 10 number = number ^ ((number ^ left_side) & left_side_replace_mask) 11 number = number >> 14 12 number = number ^ 0b00010110100000001110000001111100 13 return number
Course ids are constructed from a regular algorithm from a consistent incrementing index called a data_id. Internally these ids are used to request many known levels, like a list of your own uploaded levels. Because these data ids increment regularly and we know both the first data id, 3 million, and the last through a call to search_new, we can download every uploaded level by sweeping through all 31 million data ids currently allocated. Because users follow similar rules, downloading every user account is also possible.
Running a python program for 4 straight weeks to generate a 2 terabyte database may seem foolish to most, but I’m not putting this perfectly good PC to waste by turning it off. Instead, I did it, ran my computer for all of those 4 weeks, and the resulting data will be a goldmine into how players interact with levels and each other.
link The resulting database
1CREATE TABLE level ( 2 data_id INTEGER, -- Data IDs are regularly incrementing values, gaps in the table are due to levels deleted by users or Nintendo 3 name TEXT, -- Course name 4 description TEXT, -- Course description 5 uploaded INTEGER, -- UTC timestamp for when the level was uploaded 6 course_id TEXT, -- Course ID as seen in game 7 gamestyle INTEGER, -- Gamestyle, enum below 8 theme INTEGER, -- Theme, enum below 9 difficulty INTEGER, -- Difficulty, enum below 10 tag1 INTEGER, -- The first tag, if it exists, enum below 11 tag2 INTEGER, -- The second tag, if it exists, enum below 12 world_record INTEGER, -- The world record in milliseconds 13 upload_time INTEGER, -- The upload time in milliseconds 14 num_comments INTEGER, -- Number of comments, may not reflect the archived comments if there were more than 1000 comments 15 clear_condition INTEGER, -- Clear condition, enum below 16 clear_condition_magnitude INTEGER, -- If applicable, the magnitude of the clear condition 17 clears INTEGER, -- Course clears 18 attempts INTEGER, -- Course attempts 19 clear_rate REAL, -- Course clear rate as a float between 0 and 1 20 plays INTEGER, -- Course plays, or "footprints" 21 versus_matches INTEGER, -- Course versus matches 22 coop_matches INTEGER, -- Course coop matches 23 likes INTEGER, -- Course likes 24 boos INTEGER, -- Course boos 25 unique_players_and_versus INTEGER, -- All unique players that have ever played this level, including the number of versus matches 26 weekly_likes INTEGER, -- The weekly likes on this course 27 weekly_plays INTEGER, -- The weekly plays on this course 28 one_screen_thumbnail BLOB, -- The one screen course thumbnail, as a JPEG binary 29 one_screen_thumbnail_url TEXT, -- The old URL of this thumbnail 30 one_screen_thumbnail_size INTEGER, -- The filesize of this thumbnail 31 one_screen_thumbnail_filename TEXT, -- The filename of this thumbnail 32 entire_thumbnail BLOB, -- The entire course thumbnail, as a JPEG binary 33 entire_thumbnail_url TEXT, -- The old URL of this thumbnail 34 entire_thumbnail_size INTEGER, -- The filesize of this thumbnail 35 entire_thumbnail_filename TEXT, -- The filename of this thumbnail 36 uploader_pid TEXT, -- The player ID of the uploader 37 first_completer_pid TEXT, -- The player ID of the user who first cleared this course 38 record_holder_pid TEXT, -- The player ID of the user who held the world record at time of archival 39 level_data BLOB, -- The GZIP compressed decrypted level data, kaitai struct file is provided to read the header 40 unk2 INTEGER, -- Unknown 41 unk3 BLOB, -- Unknown 42 unk9 INTEGER, -- Unknown 43 unk10 INTEGER, -- Unknown 44 unk11 INTEGER, -- Unknown 45 unk12 INTEGER -- Unknown 46); 47CREATE TABLE user ( 48 pid TEXT, -- The player ID of this user, an unsigned 64 bit integer as a string 49 data_id INTEGER, -- The data ID of this user, while not used internally user codes are generated using this 50 code TEXT, -- User code 51 region INTEGER, -- User region, enum below 52 name TEXT, -- User name 53 country TEXT, -- User country as a 2 letter code 54 last_active INTEGER, -- UTC timestamp of when this user was last active, not known what constitutes active 55 mii_data BLOB, -- The CHARINFO blob of this user's mii 56 mii_image TEXT, -- An image of this mii as reported by Nintendo's debug endpoint 57 mii_studio_code TEXT, -- The mii studio code of this user 58 pose INTEGER, -- Pose, enum below 59 hat INTEGER, -- Hat, enum below 60 shirt INTEGER, -- Shirt, enum below 61 pants INTEGER, -- Pants, enum below 62 wearing_outfit INTEGER, -- Whether this user is wearing pants 63 courses_played INTEGER, -- How many courses this user has played 64 courses_cleared INTEGER, -- How many courses this user has cleared 65 courses_attempted INTEGER, -- How many courses this user has attempted 66 courses_deaths INTEGER, -- How many times this user has died 67 likes INTEGER, -- How many likes this user has recieved 68 maker_points INTEGER, -- Maker points 69 easy_highscore INTEGER, -- Easy highscore 70 normal_highscore INTEGER, -- Normal highscore 71 expert_highscore INTEGER, -- Expert highscore 72 super_expert_highscore INTEGER, -- Super expert high score 73 versus_rating INTEGER, -- Versus rating 74 versus_rank INTEGER, -- Versus rank, enum below 75 versus_won INTEGER, -- How many courses this user has won in versus 76 versus_lost INTEGER, -- How many courses this user has lost in versus 77 versus_win_streak INTEGER, -- Versus win streak 78 versus_lose_streak INTEGER, -- Versus lose streak 79 versus_plays INTEGER, -- Versus plays 80 versus_disconnected INTEGER, -- Times user has disconnected in versus 81 coop_clears INTEGER, -- Coop clears 82 coop_plays INTEGER, -- Coop plays 83 recent_performance INTEGER, -- Unknown variable relating to versus performance 84 versus_kills INTEGER, -- Kills in versus, unknown what activities constitute a kill 85 versus_killed_by_others INTEGER, -- Deaths in versus from other users, unknown what activities constitute a death 86 multiplayer_unk13 INTEGER, -- Unknown, relating to multiplayer 87 multiplayer_unk14 INTEGER, -- Unknown, relating to multiplayer 88 first_clears INTEGER, -- First clears 89 world_records INTEGER, -- World records 90 unique_super_world_clears INTEGER, -- Super world clears 91 uploaded_levels INTEGER, -- Number of uploaded levels 92 maximum_uploaded_levels INTEGER, -- Maximum number of levels this user may upload 93 weekly_maker_points INTEGER, -- Weekly maker points 94 last_uploaded_level INTEGER, -- UTC timestamp of when this user last uploaded a level 95 is_nintendo_employee INTEGER, -- Whether this user is an official Nintendo account 96 comments_enabled INTEGER, -- Whether this user has comments enabled on their levels 97 tags_enabled INTEGER, -- Whether this user has tags enabled on their levels 98 super_world_id TEXT, -- The ID of this user's super world, blank if they do not have one 99 unk3 INTEGER, -- Unknown 100 unk12 INTEGER, -- Unknown 101 unk16 INTEGER -- Unknown 102); 103CREATE TABLE user_badges ( 104 pid TEXT, -- Player ID 105 type INTEGER, -- The kind of badge, enum below 106 rank INTEGER -- The rank of badge, enum below 107); 108CREATE TABLE level_comments ( 109 data_id INTEGER, -- The data ID of the level this comment corresponds to 110 comment_id TEXT, -- Comment ID 111 type INTEGER, -- Type of comment, enum below 112 pid TEXT, -- Player ID of the comment creator 113 posted INTEGER, -- UTC timestamp of when this comment was created 114 clear_required INTEGER, -- Whether this comment requires a clear to view 115 text TEXT, -- If the comment type is text, the text of the comment 116 reaction_image_id INTEGER, -- If this comment is a reaction image, the id of the reaction image, enum below 117 custom_image BLOB, -- If this comment is a custom drawing, the custom drawing as a PNG binary 118 custom_image_url TEXT, -- If this comment is a custom drawing, the old url of the drawing 119 custom_image_size INTEGER, -- If this comment is a custom drawing, the filesize of the drawing in its old format 120 custom_image_filename TEXT, -- If this comment is a custom drawing, the filename of the drawing 121 has_beaten INTEGER, -- Whether the user had beaten the level when they created the comment 122 x INTEGER, -- The X position of the comment in game 123 y INTEGER, -- The Y position of the comment in game 124 reaction_face INTEGER, -- The reaction the mii of the user has on this comment 125 unk8 INTEGER, -- Unknown 126 unk10 INTEGER, -- Unknown 127 unk12 INTEGER, -- Unknown 128 unk14 BLOB, -- Unknown 129 unk17 INTEGER -- Unknown 130); 131CREATE TABLE level_played ( 132 data_id INTEGER, -- The data ID of the level this play corresponds to 133 pid TEXT, -- The player ID of the user who made this play 134 cleared INTEGER, -- Whether the user cleared the level 135 liked INTEGER -- Whether the user liked the level 136); 137CREATE TABLE level_deaths ( 138 data_id INTEGER, -- The data ID of the level this death corresponds to 139 x INTEGER, -- The X position of where this death occured 140 y INTEGER, -- The Y position of where this death occured 141 is_subworld INTEGER -- Whether this death occured in the subworld 142); 143CREATE TABLE world ( 144 pid TEXT, -- The player ID of the user who created this super world 145 world_id TEXT, -- World ID 146 worlds INTEGER, -- Number of worlds 147 levels INTEGER, -- Number of levels 148 planet_type INTEGER, -- Planet type, enum below 149 created INTEGER, -- UTC timestamp of when this super world was created 150 unk1 BLOB, -- Unknown 151 unk5 INTEGER, -- Unknown 152 unk6 INTEGER, -- Unknown 153 unk7 INTEGER, -- Unknown 154 thumbnail BLOB, -- The thumbnail, as a JPEG binary 155 thumbnail_url TEXT, -- The old URL of this thumbnail 156 thumbnail_size INTEGER, -- The filesize of this thumbnail 157 thumbnail_filename TEXT -- The filename of this thumbnail 158); 159CREATE TABLE world_levels ( 160 pid TEXT, -- The player ID of the user who created the super world with this level 161 data_id INTEGER, -- The data ID of the level 162 ninjis INTEGER -- Number of ninjis shown on this level 163); 164CREATE TABLE user_posted ( 165 pid TEXT, -- The player ID of the user who posted this 166 data_id INTEGER -- The data ID of the level 167); 168CREATE TABLE user_liked ( 169 pid TEXT, -- The player ID of the user who liked this 170 data_id INTEGER -- The data ID of the level 171); 172CREATE TABLE user_played ( 173 pid TEXT, -- The player ID of the user who played this 174 data_id INTEGER -- The data ID of the level 175); 176CREATE TABLE user_first_cleared ( 177 pid TEXT, -- The player ID of the user who first cleared this 178 data_id INTEGER -- The data ID of the level 179); 180CREATE TABLE user_world_record ( 181 pid TEXT, -- The player ID of the user who got world record on this 182 data_id INTEGER -- The data ID of the level 183); 184CREATE TABLE ninji ( 185 data_id INTEGER, -- The data ID of the level this ninji played on 186 pid TEXT, -- The player ID of the user who made this ninji 187 time INTEGER, -- The time of this ninji in milliseconds 188 replay BLOB, -- The replay binary of this ninji 189 replay_url TEXT, -- The old url of this ninji 190 replay_size INTEGER, -- The filesize of this ninji 191 replay_filename TEXT -- The filename of this ninji 192); 193CREATE TABLE ninji_level ( 194 data_id INTEGER, -- The data ID of this ninji level 195 name TEXT, -- Name 196 description TEXT, -- Description 197 uploaded INTEGER, -- UTC timestamp of when this was uploaded 198 ended INTEGER, -- UTC timestamp of when this event ended 199 gamestyle INTEGER, -- Gamestyle, enum below 200 theme INTEGER, -- Theme, enum below 201 medal_time INTEGER, -- Time to get a medal in milliseconds 202 clear_condition INTEGER, -- Clear condition, enum below 203 clear_condition_magnitude INTEGER, -- If applicable, the magnitude of the clear condition 204 unk3_0 INTEGER, -- Unknown 205 unk3_1 INTEGER, -- Unknown 206 unk3_2 INTEGER, -- Unknown 207 unk5 INTEGER, -- Unknown 208 unk6 INTEGER, -- Unknown 209 unk9 INTEGER, -- Unknown 210 level_data BLOB, -- The GZIP compressed decrypted level data, a kaitai struct file is provided to read this 211 one_screen_thumbnail BLOB, -- The one screen course thumbnail, as a JPEG binary 212 one_screen_thumbnail_url TEXT, -- The old URL of this thumbnail 213 one_screen_thumbnail_size INTEGER, -- The filesize of this thumbnail 214 one_screen_thumbnail_filename TEXT, -- The filename of this thumbnail 215 entire_thumbnail BLOB, -- The entire course thumbnail, as a JPEG binary 216 entire_thumbnail_url TEXT, -- The old URL of this thumbnail 217 entire_thumbnail_size INTEGER, -- The filesize of this thumbnail 218 entire_thumbnail_filename TEXT -- The filename of this thumbnail 219); 220CREATE INDEX idx_level_data_id ON level (data_id); 221CREATE INDEX idx_level_comments_data_id ON level_comments (data_id); 222CREATE INDEX idx_level_deaths_data_id ON level_deaths (data_id); 223CREATE INDEX idx_level_played_data_id ON level_played (data_id); 224CREATE INDEX idx_ninji_data_id ON ninji (data_id); 225CREATE INDEX idx_user_pid ON user (pid); 226CREATE INDEX idx_user_badges_pid ON user_badges (pid); 227CREATE INDEX idx_user_first_cleared_pid ON user_first_cleared (pid); 228CREATE INDEX idx_user_liked_pid ON user_liked (pid); 229CREATE INDEX idx_user_played_pid ON user_played (pid); 230CREATE INDEX idx_user_posted_pid ON user_posted (pid); 231CREATE INDEX idx_user_world_record_pid ON user_world_record (pid); 232CREATE INDEX idx_world_pid ON world (pid); 233CREATE INDEX idx_world_levels_pid ON world_levels (pid);
In an update post I’ll cover the data analysis I do on this massive archive, and release it publicly. Make sure to look out for that!