A Mario ML Diffusion Model
Created Tuesday, May 23, 2023
It is clear generative AI has improved immensely in the last few years, even months. Mario Maker 2 poses an interesting problem for current technology, however. It:
And most important of all:
This last point poses an interesting problem. Current diffusion models, the most promising generative image AI model, start with noise and add small increments of color to each pixel to ease the image into something usable. Some of the benefits of this approach include:
Diffusion models depend on the ability to tweak values within the matrix. In Mario Maker 2 object IDs are integers, there is no gradual tweaking towards an object ID. Can I solve this problem while still conserving the properties of seedability and configurable iterations?
The model is a neural network with an input of i neurons, where i is the number of possible object IDs, times j, where j is the number of adjacent objects to include in the training data. The output is a probability distribution of i object ID neurons, where the one with the best probability that has an ID within the allowable range is chosen. The number of adjacent objects to train on is configurable and is called the context size in this model. The visual below demonstrates how a context size of 3 is trained, where 3 represents the 3x3 square surrounding the object ID the model is trying to output.
The theory behind inference in this model comes from a background in cellular automata, where decisions about the next state are made by nearby objects. This approach was applied to Mario Maker 2 by training on a configurable context size of adjacent objects and learning what the expected middle object should be. The level starts as random noise of all allowable objects and this inference is performed for every object in the level, where a context that stretches off the side of the level is considered to be air. This is called one iteration. The same process is performed again on the objects generated by the last iteration until n iterations has been reached. This approach is a blend of cellular automata and diffusion.
This approach inevitably began to prioritize not the objects that actually fit a specific circumstance but instead the objects that just appeared the most. Multiple iterations would not improve the output but just cause a small number of the allowed objects to crowd out all other potential objects. An approach I experimented with was limiting the number of samples per object so that the same number of samples per object was included but this did not fix the overcrowding, it just led to rarer objects becoming dominant instead.
Use the Contact button to the side or join my Discord.