Pandoc's tex_math_dollars, tex_math_single_backslash and raw TeX mode makes it possible to fallback to TeX whenever you need: https://pandoc.org/MANUAL.html#math-input
A Markdown -> LaTeX -> pdf pipeline was how I wrote all of my algorithm assignments in college. I wrote things like:
* **Basis**. Prove for $n = 0$...
* **Induction**. If \mathcal{G} is a graph ...
\begin{equation}
...
\end{equation}
You seem to be well over half-way to LaTeX there, so what's the point in using markdown?
If the verbosity of writing \begin{description} ... \item[basis]... \end{description} is the issue, you can get around that with a couple of shorthand macros.
I wrote my math phd thesis in Markdown/LaTeX in this way.
I also replaced all maths symbols with their Unicode equivalent. The result was very readable markdown
source text, easily compiled to Latex and PDF, using
a mk file.
My thesis also included formalised proofs in a proof
assistant, and they were just written straight into
the markdown files as code blocks, and were using the
same Unicode symbols as the rest.
Later, when publishing the different parts of my thesis,
this separation from Latex made it easier to convert to
whatever cls the publisher wanted, since I would just
change the template.
I wrote mine in plain LaTeX, never had an issue with it. Regarding code, you can use the listings package (\lstincludelisting) to include source code files directly. I fail to see the interest in having a bastardized, not quite TeX, not quite MD file (unless you're using something like Org, of course, and using its other functionalities).
Regarding having unicode for the maths, how did you deal with symbols that needed scaling (brackets, integrals, etc.?) They may look simpler when seen as text, but they certainly won't render nicely as math...
A Markdown -> LaTeX -> pdf pipeline was how I wrote all of my algorithm assignments in college. I wrote things like: