A (non-rational) quadratic Bezier curve is a section of a parabola, which is the trajectory of an initial velocity under constant acceleration, which at most spans a plane as only two vectors are involved.
Basic vector addition lets us visualize that a parallelgram grid is formed from the displacement contributed by initial velocity and displacement contributed by acceleration. So it's pretty easy to see that the middle control point forms the direction of acceleration with the midpoint between start and end[1].
Why then are all the sources describing quadratic Bezier curves make absolutely no reference to any meaningful physical quantities of any sort, instead just treating them like some sort of magical incantation about "interpolations of interpolation"? And similarly cubic Bezier curve can easily be described by the parallelpipid grid spanned by the velocity, acceleration, and jerk vectors, so there's no sense in saying that this strange way of describing them is necessary for generalization. The only likely explanation I can think of is laziness, as it doesn't really make much sense to be motivated by some sort of elitism on something so exceedingly simple.
[1] https://i.imgur.com/vP9fLYh.png
I think a lot of important and frequently applied elements of math are, like Bezier splines, something that can be understood in different ways, in different contexts. e.g. what is a matrix? You can tell a story about systems of equations, or about tensors or maybe bilinear forms, or about coordinate system transforms, etc. All of these are valid, and good understanding comes from being able to hold multiple views of the same underlying thing simultaneously.
You have embraced one way of interpreting a quadratic Bezier curve, as a curve characterized by some constant second derivative (2 degrees of freedom or DOF), an initial velocity (2 DOF), and an initial position (2 DOF). Someone from computer-aided design (CAD) might specify the curve in terms of the start point (2 DOF), end point (2 DOF), and control point (2 DOF). Someone implementing all this in code might evaluate the basis functions, or might use Casteljau's method to do the lerp of the lerps, as you noted. All of these end up being different views of the same math object, and all of them are sensible.
In particular, what is most "physical" may be subjective. An important use of splines was CAD, where a human is saying: I need something to go from here, to here, in a smooth way that I can nudge with this control point. That is how we interact with cubic Bezier splines in drawing programs, and those control points for specifying the spline are nice and tangible, if not physical. A curve of constant jerk is physical too, I guess, but doesn't align as well with how the expressivity of cubic splines frequently gets invoked.
There is definitely elitism in how some people describe their use of math (behold my sophistication); it's a bummer. More charitably, you could understand it as the author respecting the time of the reader assumed to already understand something about the topic. But don't let that discourage you.