Mario Maker 2 API
Created Thursday, Feb 10, 2022
Mario Maker 2 is a game I've played for years, since launch, and I've always been interested in putting my skills in programming to the test here. I've finally done just that. Using the incredible NintendoClients made by Kinnay, I, and individuals from the Ninji Speedrunning discord, 0% discord, and Kinnay himself, we now have an entirely free and public way of accessing data previously only available to the switch itself.
Background
Nintendo Switch Online games tend to use a specific library with a specific protocol to communicate with Nintendo's servers. Since the WiiU, this protocol has been NEX. NEX, partially due to the incredible success of many of the games using it like Mario Kart 8 and Splatoon 2, has been extensively documented, to the point where it is fully possible to spoof both the client and server. The Pretendo project has been working on a WiiU NEX server for years, and alongside it NintendoClients, a NEX client, has been becoming more stable and safe to run on your personal account.
In October 2021 I finally took the plunge and started using NintendoClients for a dream I had years prior: to make all Mario Maker 2 data easily accessible to computers. After watching the project for years, debating whether to purchase a digital copy of the game and put my Switch certificate on the line, Tatiaus, a talented kaizo player, contacted me about using the API for the 0% team as I had mentioned it in passing earlier. Once I realized there was interest past my own goals I bought the game and began working. On the 13th, I released the first build, publicly available with no login required.
First release to a small group of Switch modders
Word gets out
First, the Ninji server used it to round out their leaderboards, making them 100% accurate for the first time ever. The features I considered most important, the level_info and level_data endpoints, went unused. Not many people outside the Ninji Speedrunning discord knew this existed. Slowly, the API got around, but it stayed a novelty. One of the funniest misconceptions I saw was the guess that my switch was running 24/7 and downloading levels for me. The truth is that this was running on a linux server, using a complete recreation of the NEX client. No switch was involved, except for the various credentials the NEX protocol needs to authenticate with the server.
Interest peaks
In November a kaizo player well known for performing incredibly difficult "trump jumps", jumps in the game which were pixel perfect, came to me about using the API for a level viewer he had designed. In time Ji Xiaomai released his viewer, and that was the spark that ignited the entire community. Suddenly now anyone could view any level without the game and even without a switch. Tricky levels that were only possible due to information only the creator could know were exposed. Suddenly now no level could hide secrets.
After the release of this viewer, I ported it to C++ and made it portable enough to run in the browser, so now it wasn't even neccesary to download the app to your PC. With time came new incredible viewers, including one that had animated textures. The count of level viewers was now 5, including the continuation of an old level viewer by streamer and programmer Shoujo and one made from the ground up by Ingram. The community had collectively come together to tear the game apart, utterly destroying the roughly 1 level viewer made for the first game. And in another impressive effort, almost every field of every endpoint was determined, so now stats previously thought impossible to obtain, like the boos on a level or the k/d ratio of a player, were accessible. All this hype has generated an impressive amount of traffic as well:
Web traffic after release
Archival
Through all this, the API has stayed public and free. Information should always be as accessible as possible and designed to be as easy to parse as possible by people and computers alike. While I am putting my personal Nintendo Switch Online account on the line, this tool is invaluable to research about the game. So I can proudly say, while Youtube made dislikes private, we made Mario Maker 2 dislikes public.
The logical conclusion to this venture is to pull as much data as we can. So I did the only reasonable thing, went out and bought a 14 terabyte hard drive in order to download Mario Maker 2's entire database. Now it might seem impossible to simply download everything, we can request up to 300 random levels at a time through endpoints like search_endless_mode, but statistically we will recieve less and less unique items over time, almost guaranteed to take years to download 100% of available levels. Nintendo, however, made this easy for us:
def course_id_to_dataid(id):
course_id = id[::-1]
charset = "0123456789BCDFGHJKLMNPQRSTVWXY"
number = 0
for char in course_id:
number = number * 30 + charset.index(char)
left_side = number
left_side = left_side << 34
left_side_replace_mask = 0b1111111111110000000000000000000000000000000000
number = number ^ ((number ^ left_side) & left_side_replace_mask)
number = number >> 14
number = number ^ 0b00010110100000001110000001111100
return number
Course ids are constructed from a regular algorithm from a consistent incrementing index called a data_id. Internally these ids are used to request many known levels, like a list of your own uploaded levels. Because these data ids increment regularly and we know both the first data id, 3 million, and the last through a call to search_new, we can download every uploaded level by sweeping through all 31 million data ids currently allocated. Because users follow similar rules, downloading every user account is also possible.
Running a python program for 4 straight weeks to generate a 2 terabyte database may seem foolish to most, but I'm not putting this perfectly good PC to waste by turning it off. Instead, I did it, ran my computer for all of those 4 weeks, and the resulting data will be a goldmine into how players interact with levels and each other.
The resulting database
CREATE TABLE level (
data_id INTEGER, -- Data IDs are regularly incrementing values, gaps in the table are due to levels deleted by users or Nintendo
name TEXT, -- Course name
description TEXT, -- Course description
uploaded INTEGER, -- UTC timestamp for when the level was uploaded
course_id TEXT, -- Course ID as seen in game
gamestyle INTEGER, -- Gamestyle, enum below
theme INTEGER, -- Theme, enum below
difficulty INTEGER, -- Difficulty, enum below
tag1 INTEGER, -- The first tag, if it exists, enum below
tag2 INTEGER, -- The second tag, if it exists, enum below
world_record INTEGER, -- The world record in milliseconds
upload_time INTEGER, -- The upload time in milliseconds
num_comments INTEGER, -- Number of comments, may not reflect the archived comments if there were more than 1000 comments
clear_condition INTEGER, -- Clear condition, enum below
clear_condition_magnitude INTEGER, -- If applicable, the magnitude of the clear condition
clears INTEGER, -- Course clears
attempts INTEGER, -- Course attempts
clear_rate REAL, -- Course clear rate as a float between 0 and 1
plays INTEGER, -- Course plays, or "footprints"
versus_matches INTEGER, -- Course versus matches
coop_matches INTEGER, -- Course coop matches
likes INTEGER, -- Course likes
boos INTEGER, -- Course boos
unique_players_and_versus INTEGER, -- All unique players that have ever played this level, including the number of versus matches
weekly_likes INTEGER, -- The weekly likes on this course
weekly_plays INTEGER, -- The weekly plays on this course
one_screen_thumbnail BLOB, -- The one screen course thumbnail, as a JPEG binary
one_screen_thumbnail_url TEXT, -- The old URL of this thumbnail
one_screen_thumbnail_size INTEGER, -- The filesize of this thumbnail
one_screen_thumbnail_filename TEXT, -- The filename of this thumbnail
entire_thumbnail BLOB, -- The entire course thumbnail, as a JPEG binary
entire_thumbnail_url TEXT, -- The old URL of this thumbnail
entire_thumbnail_size INTEGER, -- The filesize of this thumbnail
entire_thumbnail_filename TEXT, -- The filename of this thumbnail
uploader_pid TEXT, -- The player ID of the uploader
first_completer_pid TEXT, -- The player ID of the user who first cleared this course
record_holder_pid TEXT, -- The player ID of the user who held the world record at time of archival
level_data BLOB, -- The GZIP compressed decrypted level data, kaitai struct file is provided to read the header
unk2 INTEGER, -- Unknown
unk3 BLOB, -- Unknown
unk9 INTEGER, -- Unknown
unk10 INTEGER, -- Unknown
unk11 INTEGER, -- Unknown
unk12 INTEGER -- Unknown
);
CREATE TABLE user (
pid TEXT, -- The player ID of this user, an unsigned 64 bit integer as a string
data_id INTEGER, -- The data ID of this user, while not used internally user codes are generated using this
code TEXT, -- User code
region INTEGER, -- User region, enum below
name TEXT, -- User name
country TEXT, -- User country as a 2 letter code
last_active INTEGER, -- UTC timestamp of when this user was last active, not known what constitutes active
mii_data BLOB, -- The CHARINFO blob of this user's mii
mii_image TEXT, -- An image of this mii as reported by Nintendo's debug endpoint
mii_studio_code TEXT, -- The mii studio code of this user
pose INTEGER, -- Pose, enum below
hat INTEGER, -- Hat, enum below
shirt INTEGER, -- Shirt, enum below
pants INTEGER, -- Pants, enum below
wearing_outfit INTEGER, -- Whether this user is wearing pants
courses_played INTEGER, -- How many courses this user has played
courses_cleared INTEGER, -- How many courses this user has cleared
courses_attempted INTEGER, -- How many courses this user has attempted
courses_deaths INTEGER, -- How many times this user has died
likes INTEGER, -- How many likes this user has recieved
maker_points INTEGER, -- Maker points
easy_highscore INTEGER, -- Easy highscore
normal_highscore INTEGER, -- Normal highscore
expert_highscore INTEGER, -- Expert highscore
super_expert_highscore INTEGER, -- Super expert high score
versus_rating INTEGER, -- Versus rating
versus_rank INTEGER, -- Versus rank, enum below
versus_won INTEGER, -- How many courses this user has won in versus
versus_lost INTEGER, -- How many courses this user has lost in versus
versus_win_streak INTEGER, -- Versus win streak
versus_lose_streak INTEGER, -- Versus lose streak
versus_plays INTEGER, -- Versus plays
versus_disconnected INTEGER, -- Times user has disconnected in versus
coop_clears INTEGER, -- Coop clears
coop_plays INTEGER, -- Coop plays
recent_performance INTEGER, -- Unknown variable relating to versus performance
versus_kills INTEGER, -- Kills in versus, unknown what activities constitute a kill
versus_killed_by_others INTEGER, -- Deaths in versus from other users, unknown what activities constitute a death
multiplayer_unk13 INTEGER, -- Unknown, relating to multiplayer
multiplayer_unk14 INTEGER, -- Unknown, relating to multiplayer
first_clears INTEGER, -- First clears
world_records INTEGER, -- World records
unique_super_world_clears INTEGER, -- Super world clears
uploaded_levels INTEGER, -- Number of uploaded levels
maximum_uploaded_levels INTEGER, -- Maximum number of levels this user may upload
weekly_maker_points INTEGER, -- Weekly maker points
last_uploaded_level INTEGER, -- UTC timestamp of when this user last uploaded a level
is_nintendo_employee INTEGER, -- Whether this user is an official Nintendo account
comments_enabled INTEGER, -- Whether this user has comments enabled on their levels
tags_enabled INTEGER, -- Whether this user has tags enabled on their levels
super_world_id TEXT, -- The ID of this user's super world, blank if they do not have one
unk3 INTEGER, -- Unknown
unk12 INTEGER, -- Unknown
unk16 INTEGER -- Unknown
);
CREATE TABLE user_badges (
pid TEXT, -- Player ID
type INTEGER, -- The kind of badge, enum below
rank INTEGER -- The rank of badge, enum below
);
CREATE TABLE level_comments (
data_id INTEGER, -- The data ID of the level this comment corresponds to
comment_id TEXT, -- Comment ID
type INTEGER, -- Type of comment, enum below
pid TEXT, -- Player ID of the comment creator
posted INTEGER, -- UTC timestamp of when this comment was created
clear_required INTEGER, -- Whether this comment requires a clear to view
text TEXT, -- If the comment type is text, the text of the comment
reaction_image_id INTEGER, -- If this comment is a reaction image, the id of the reaction image, enum below
custom_image BLOB, -- If this comment is a custom drawing, the custom drawing as a PNG binary
custom_image_url TEXT, -- If this comment is a custom drawing, the old url of the drawing
custom_image_size INTEGER, -- If this comment is a custom drawing, the filesize of the drawing in its old format
custom_image_filename TEXT, -- If this comment is a custom drawing, the filename of the drawing
has_beaten INTEGER, -- Whether the user had beaten the level when they created the comment
x INTEGER, -- The X position of the comment in game
y INTEGER, -- The Y position of the comment in game
reaction_face INTEGER, -- The reaction the mii of the user has on this comment
unk8 INTEGER, -- Unknown
unk10 INTEGER, -- Unknown
unk12 INTEGER, -- Unknown
unk14 BLOB, -- Unknown
unk17 INTEGER -- Unknown
);
CREATE TABLE level_played (
data_id INTEGER, -- The data ID of the level this play corresponds to
pid TEXT, -- The player ID of the user who made this play
cleared INTEGER, -- Whether the user cleared the level
liked INTEGER -- Whether the user liked the level
);
CREATE TABLE level_deaths (
data_id INTEGER, -- The data ID of the level this death corresponds to
x INTEGER, -- The X position of where this death occured
y INTEGER, -- The Y position of where this death occured
is_subworld INTEGER -- Whether this death occured in the subworld
);
CREATE TABLE world (
pid TEXT, -- The player ID of the user who created this super world
world_id TEXT, -- World ID
worlds INTEGER, -- Number of worlds
levels INTEGER, -- Number of levels
planet_type INTEGER, -- Planet type, enum below
created INTEGER, -- UTC timestamp of when this super world was created
unk1 BLOB, -- Unknown
unk5 INTEGER, -- Unknown
unk6 INTEGER, -- Unknown
unk7 INTEGER, -- Unknown
thumbnail BLOB, -- The thumbnail, as a JPEG binary
thumbnail_url TEXT, -- The old URL of this thumbnail
thumbnail_size INTEGER, -- The filesize of this thumbnail
thumbnail_filename TEXT -- The filename of this thumbnail
);
CREATE TABLE world_levels (
pid TEXT, -- The player ID of the user who created the super world with this level
data_id INTEGER, -- The data ID of the level
ninjis INTEGER -- Number of ninjis shown on this level
);
CREATE TABLE user_posted (
pid TEXT, -- The player ID of the user who posted this
data_id INTEGER -- The data ID of the level
);
CREATE TABLE user_liked (
pid TEXT, -- The player ID of the user who liked this
data_id INTEGER -- The data ID of the level
);
CREATE TABLE user_played (
pid TEXT, -- The player ID of the user who played this
data_id INTEGER -- The data ID of the level
);
CREATE TABLE user_first_cleared (
pid TEXT, -- The player ID of the user who first cleared this
data_id INTEGER -- The data ID of the level
);
CREATE TABLE user_world_record (
pid TEXT, -- The player ID of the user who got world record on this
data_id INTEGER -- The data ID of the level
);
CREATE TABLE ninji (
data_id INTEGER, -- The data ID of the level this ninji played on
pid TEXT, -- The player ID of the user who made this ninji
time INTEGER, -- The time of this ninji in milliseconds
replay BLOB, -- The replay binary of this ninji
replay_url TEXT, -- The old url of this ninji
replay_size INTEGER, -- The filesize of this ninji
replay_filename TEXT -- The filename of this ninji
);
CREATE TABLE ninji_level (
data_id INTEGER, -- The data ID of this ninji level
name TEXT, -- Name
description TEXT, -- Description
uploaded INTEGER, -- UTC timestamp of when this was uploaded
ended INTEGER, -- UTC timestamp of when this event ended
gamestyle INTEGER, -- Gamestyle, enum below
theme INTEGER, -- Theme, enum below
medal_time INTEGER, -- Time to get a medal in milliseconds
clear_condition INTEGER, -- Clear condition, enum below
clear_condition_magnitude INTEGER, -- If applicable, the magnitude of the clear condition
unk3_0 INTEGER, -- Unknown
unk3_1 INTEGER, -- Unknown
unk3_2 INTEGER, -- Unknown
unk5 INTEGER, -- Unknown
unk6 INTEGER, -- Unknown
unk9 INTEGER, -- Unknown
level_data BLOB, -- The GZIP compressed decrypted level data, a kaitai struct file is provided to read this
one_screen_thumbnail BLOB, -- The one screen course thumbnail, as a JPEG binary
one_screen_thumbnail_url TEXT, -- The old URL of this thumbnail
one_screen_thumbnail_size INTEGER, -- The filesize of this thumbnail
one_screen_thumbnail_filename TEXT, -- The filename of this thumbnail
entire_thumbnail BLOB, -- The entire course thumbnail, as a JPEG binary
entire_thumbnail_url TEXT, -- The old URL of this thumbnail
entire_thumbnail_size INTEGER, -- The filesize of this thumbnail
entire_thumbnail_filename TEXT -- The filename of this thumbnail
);
CREATE INDEX idx_level_data_id ON level (data_id);
CREATE INDEX idx_level_comments_data_id ON level_comments (data_id);
CREATE INDEX idx_level_deaths_data_id ON level_deaths (data_id);
CREATE INDEX idx_level_played_data_id ON level_played (data_id);
CREATE INDEX idx_ninji_data_id ON ninji (data_id);
CREATE INDEX idx_user_pid ON user (pid);
CREATE INDEX idx_user_badges_pid ON user_badges (pid);
CREATE INDEX idx_user_first_cleared_pid ON user_first_cleared (pid);
CREATE INDEX idx_user_liked_pid ON user_liked (pid);
CREATE INDEX idx_user_played_pid ON user_played (pid);
CREATE INDEX idx_user_posted_pid ON user_posted (pid);
CREATE INDEX idx_user_world_record_pid ON user_world_record (pid);
CREATE INDEX idx_world_pid ON world (pid);
CREATE INDEX idx_world_levels_pid ON world_levels (pid);
In an update post I'll cover the data analysis I do on this massive archive, and release it publicly. Make sure to look out for that!