(Uncreative, probably very bad) examples: Replace brackets with boxes, use colors (think syntax highlighting), icons, or even basic text-like formatting. And that's still being very conservative in relation to what could be done, as exemplified by the OP.
That said, as an experiment, I did about half of my Master's thesis work in TeX. Without quick feedback on the rendered formulae, I had a much harder time reasoning through things, so having a comfortable text editing environment seems crucial.
That said, I wouldn't recommend using TeX as a scratchpad. It just doesn't map well to the sets of tools that we tend to use: mini diagrams, arrows between parts of the page, strikeout and circling text, etc.
What might work is something like e-paper that renders a handwritten document into text. Ideally, we could turn letters into Unicode and diagrams into vector graphics, providing tools to edit and embellish the rendered versions if desired.
It's extremely common to edit equations and diagrams on the fly, so I would probably feel friction with anything other that doesn't seem to just magically "prettify" whatever I handwrite at it. Seems like a really tall order though.
The standard notation is mostly for communicating with other people mediated via papers. But we don't actually use that many brackets in math---that's more of a programming thing. (Because programmers can't really use visual structure all that much, and mostly rely on linearization of their syntax trees. Languages with significant whitespace like Haskell (and to a lesser extent Python) see less parens and braces.)
A bit lowbrow, perhaps (you'd want Penrose notation or such things in that case).
In the coding world, we have "Hello World" and Fizzbuzz. It would be nice if alternative music notation systems had something similar: A simple, familiar melody (candidates that come to mind are "Happy Birthday to You," and "Twinkle, Twinkle, Little Star") that, by rendering it in the new notation system, demonstrated the fundamental features of that system, and could be easily understood by someone knowledgeable in the traditional system.
If the score is figured bass, the player will improvise/create a complete accompaniment from a sketch of the bass line and some hints about the harmony.
And it's not necessarily unchanged. The layout of a piano score and a transposed brass line are only distantly related. The notation will be common, more or less, with the limitation that a horn line is monophonic while a piano can play chords.
But the key won't be the same, which means the pitches won't be the same. And many of the expressive markings will be different too.
I performed this in France with my treble choir in 1990. Schafer sat in the audience looking grumpy. It’s a beautiful piece, and Schafer is odd.
We make web-based interactive music notation. Nothing “experimental” as in the original link, but it may be interesting to the HN crowd as a boundary-pushing web app.