Wolfram Alpha was (at least initially) famously bad* at actually parsing natural language. For math, it was much easier to enter raw Mathematica expression into Wolfram Alpha than structuring the question in natural language. LLMs should give it a boost, by simplifying the kind of the natural language input into the limited forms parsable by Wolfram Alpha.
*: And I say this as a paying customer of Mathematica on my computer and Wolfram Alpha on my phone.
Obviously if you can write Mathematica, that's far more precise than any natural language query, but I think it's always been impressively good at it. You wouldn't say that, say, Python or MySQL is good at parsing natural language - but if you plug a query like:
gravitational constant times density of tungsten carbide times the volume of the earth divided by radius of earth squared
into !wa or Wolfram Alpha it will not only represent the parsed tokens as it interpreted them but also return the correct result of 27 m/s^2, without messing with tedious unit conversions. It's trivial to construct incorrect queries, but where Python just says "SyntaxError" !wa will make an attempt.
I recently tried to get it to convert mpg to l/100km. I spent close to 10 minutes until I stumbled into a combination that gave the answer among a sea of useless conversions.
*: And I say this as a paying customer of Mathematica on my computer and Wolfram Alpha on my phone.