it would certainly make MMOs a little more interesting if raids actually required outthinking an intelligent machine as oppose to exploiting it's preprogrammed responses via rote trial and error
Not raids, but World of Warcraft's "Isand Expeditions" feature, introduced around 2018, does this.
They use an AI trained on PvP player behaviour, so the NPCs jump around, respond to teammates being injured, try to catch you off-rotation, etc.
In my opinion, it wouldn't work for bosses though, as part of the fun is exactly about understanding the tells and patterns. If you want to fight against intelligent enemies, just play PvP! I think if you play without addons like BigWigs/DeadlyBossMods you'll have much more fun, if what you're looking for is unpredictability.
Plus, would there really be that much of a difference between a machine-learned approach and a decision tree when every action the boss can perform needs to be pre-programmed anyway, and have predefined ranges/optimal uses? Maybe a different game could combine that with something like Magicka, where there are spell "recipes" which can be combined, so the AI would have more options to play with.
Another example is things like Dark Souls, the bosses are so engaging to its players _because_ you can learn to understand how they respond to your positioning. If they were a moving target, there would be no point in doing anything but learning the animations for each attack so you can dodge, so in the end they would be _less_ intellectually engaging.
Actually you could have more advanced boss AI while still keeping the player ability to learn its "tells" if the animation system had access to what the AI is thinking. Lights that turn from blue to angry red when the AI is contemplating firing, giving the attack a mandatory charge-up time, etc.
The AI could be unaware of its tells or that it's telegraphing moves. Even more wild would be if the AI could be aware, so that maybe it tries to cover the status light or hide the weapon from view while it charges up. That in itself would be a tell.
that's a good argument. part of the fun is learning the AI behaviors. and in modern games it honestly would take a while for that to get boring, the AIs certainly are getting more complex
