The best way to think of today's AI capability is: automation.
You train it with a lots of examples of "given this input, this is the output i want", and hopefully it learns to get the "correct" (similar input => similar output) output for new inputs that you feed it. i.e. you've now automated the process of figuring the correct output for a given input.
There is also the "reinforcement learning" AI paradigm where the trained AI is choosing actions (from a given repertoire) in order to maximize an action outcome score based on some scoring function you provided. This is appropriate in a situation where you want the "AI" to do something more than just select the correct output, but again no magic - you're having to anticipate and score the potential outcomes.
In my opinion this type of tasks is the typical AI tasks that get most exposed to the general public. The AI/ML model is set to attempt a human task. The aspiration is to get as good as a human (or faster/better more precision etc). Classic statistical inference/ models usually don't perform very well in these scenarios(or do they?).
The typical AI methods train a model that utilises features that don't make much sense to humans (pixels/texture/word tokens etc). In contrast with traditional statistical modelling where each feature were given 'meaning' and their importance explored via investigating the observations (data) using properties of well studied mathematical models, the AI methods that often made the press don't put a lot of emphasis on these properties. The advancement is a push to utilise all available information/data to outstanding 'performance'. The explanation of the inner features within that math space is usually secondary (though I don't mean that authors of novel methods don't care about mathematical modelling).
I might be too naive here but that's how I feel after trying out many methodologies in my field.
The "AI" capabilities that are making headlines nowadays are mostly based on deep (multi-layer) neural networks with millions, or billions, or parameters. These nets do self-organize into a hierarchy of self-defined feature detectors followed by classifiers, but as you suggest for the most part they are best regarded as black boxes. You can do sensitivity analysis to determine what specific interval values are reacting to, and maybe glean some understanding that way, but by-nature the inner workings are not intended/expected to be meaningful - they are just a byproduct of the brute force output-error minimization process by which these nets are trained.
Neural nets are mostly dominant in perceptual domains such as image or speech recognition, where the raw inputs represent a uniform sampling of data values (pixels, audio samples) over space and/or time. For classical business problems where the inputs are much richer and more varied, and already have individual meaning, then decision tree techniques such as random forests may be more appropriate and do provide explainability.
>Neural nets are mostly dominant in perceptual domains such as image or speech recognition, where the raw inputs represent a uniform sampling of data values (pixels, audio samples) over space and/or time.
I mostly agree with that, however there have been advancement in scientific areas such as chemistry and biology (DNA/RNA) where the data are definitely meaningful and a lot times categorical. So the methodologies can be applied in wider areas, they just need a lot of domain knowledge and experience.
You train it with a lots of examples of "given this input, this is the output i want", and hopefully it learns to get the "correct" (similar input => similar output) output for new inputs that you feed it. i.e. you've now automated the process of figuring the correct output for a given input.
There is also the "reinforcement learning" AI paradigm where the trained AI is choosing actions (from a given repertoire) in order to maximize an action outcome score based on some scoring function you provided. This is appropriate in a situation where you want the "AI" to do something more than just select the correct output, but again no magic - you're having to anticipate and score the potential outcomes.