As well as Super Mario Odyssey 64 (which let you possess anything from Mario 64 by chucking your hat at it):
Super Mario 64 Maker:
And Super Mario 64 3D World (where he set up the custom abilities the players for Mario 64 Online have):
So if you're interested in this project, you should also check out his previous work. It's amazing.
In fact, there are level editors for SM64! Kaze, the creator of this hack, occasionally streams development on twitch; it's a lot of cross-referencing different pieces of accumulated documentation.
Edit: An example of some of the documentation, a list of ROM addresses of various textures: http://wiki.origami64.net/super_mario_64/textures
First you need to find code relevant to what you want to do. You can take diffs of memory as you do things in the game to narrow down where the relevant addresses are, and then set breakpoints on read/write to those locations to find relevant code. After that, you can mostly just follow the assembly - you can even read backwards up the call stack by reading to what look like the beginning of a procedure, and searching for jumps to that address. Once you've found the code you want to modify, you just need to find some empty space in the ROM that you can branch out to to write your code.
The trickier bit is if you're in a spot where you need to worry about timing. On older consoles, this included things like changing the HUD or wave effects, etc., since you need to make sure whatever work you do gets done in time for HBlank/VBlank.
Some consoles also had some built in functions that would give you an idea of what code was for, like the lz77 compression on the GBA, which was mostly just used for graphics. Similarly, DMA was most often used for copying sprite data. On newer consoles where code was often compiled from C (rather than assembly with macros), I'm under the impression you can also do things like try to pattern match on the signature of standard library functions.
Mostly of the Lua-script-that-made-cars-rain-from-the-sky-above-random-players kind.
Still, this looks hella cool. Hope Nintendo doesn't get nasty about it, I want to try it with my kids (although I lack enough machines)
There's also additional challenges where they have a limited amount of game RAM to work with and so additional textures and models are tough to fit in.
Threads are starting to form with public servers etc. Could be a good community to come of this, I'm excited.