Ruts are there because everywhere you look is worse than where you are at, but that just appears to be true because you are in, well, a rut.
I can think of a few ways aviation could break out of its rut. I originally thought the aviation rut would break because of autonomous or near autonomous aircraft enabling safe and more efficient small aircraft. That would have led to cheaper more personal travel and broken the hub and spoke nightmare system we have now. We would have gotten into a whole new optimal for aviation. The rut was too large though and getting to that level of automation isn't happening any time soon without some other massive change. I now think the thing that will do it is electric. Batteries are getting so much better along with electric motors and soon (a few years?) batteries will outpace JET-A enabling a massive shift to smaller, cheaper aviation. I think this push will be so strong that it will finally break the aviation rut. Whatever the eventual reason, and there will be a reason, when we get out of it we will see radical changes to the whole industry and we will look back, just like the space industry is now, and ask 'why did it take so long' and 'how did we do things so slowly/badly?'.
Here is a quick aside about ruts. I almost wrote up the math of this once. Ruts are caused by the algorithm we all use, linear regression. It is powerful and good at what it does. Assess, step, assess step. So simple and so powerful but totally misunderstood. In a fixed world if you throw a dart and use linear regression to find a better optimum you will almost always find a way better place than where the dart initially landed. But we don't live in a fixed world. In a dynamic world you still think you are selecting for optimality, but in reality you are selecting for stability. If the world keeps changing but where you are at is standing still then you are in a rut, a stable point. A rut is a local optimum where the walls are higher than your assessment vision and those walls, for whatever reason, don't get lower than your point even though the world is changing. If they did you would see the route out and take it. An algorithm to bust out of this is to double your step size every time you go back to your starting point. If you do this then eventually you will 'break out' and finally be able to find a new optimum that is likely way better than where you were at before. An interesting counter-intuitive consequence of all this is that the more visionary the industry, the bigger the eventual rut it will get caught in. Aviation, space, medicine have all been in massive ruts -because- the people in them are so smart and can see so far. When they finally got caught in their ruts those ruts were massive. The break out will be equally massive too (as we are seeing with space right now)
There is a lot of fun math behind ruts. I am surprised by how underdeveloped it is. One thing is sure though, aviation is in a deep one and when it does break out it will be amazing.
Electric has many advantages that make a direct comparison of jet-a vs battery a little hard. First, what is the overall weight of the fuel storage/support systems plus the mass associated with getting that fuel to the engines? What about the weight of the engines and all their support systems? How about how efficiently that energy can be turned into work? How efficiently can you place the engines? What about the efficiency that you can maintain in all flight regimes? What about the volume it takes to store it? How about the cost to refuel? What about the environmental advantages (not just air pollution but noise) Finally, what about the simplicity and safety of the design?
Conventional aircraft are barely improving in any of these areas while electric is rapidly improving in all of these areas. It may take a while for batteries to get the same energy density, but there is a good chance they won't need to even come all that close before the other advantages push electric ahead.
That's a lot of handwaving but the reality works out to jet fuel being orders of magnitude more efficient than batteries on the basis of how much of either you can get into an airplane and how far they will take you. One key factor is a load of jet fuel gets much lighter as you burn it, but batteries don't get lighter as you drain them.
One thing I have been thinking about for electric is how it may allow completely different flight possibilities that could leap ahead of what we have now. For example, extremely high altitude flight is a huge challenge for current aircraft on a number of fronts, but a big one is how do you design an engine that can produce power at extreme altitudes. The higher you go the more heat limited you are in a gas turbine engine. Electric doesn't have that issue so it may be able to push aircraft a lot higher which would enable faster/further travel for far less energy use. The energy density comparison is predicated on using that energy in exactly the same way with the same efficiencies. If long distance flight becomes 100x more efficient then the density difference isn't a problem.
I can think of a few ways aviation could break out of its rut. I originally thought the aviation rut would break because of autonomous or near autonomous aircraft enabling safe and more efficient small aircraft. That would have led to cheaper more personal travel and broken the hub and spoke nightmare system we have now. We would have gotten into a whole new optimal for aviation. The rut was too large though and getting to that level of automation isn't happening any time soon without some other massive change. I now think the thing that will do it is electric. Batteries are getting so much better along with electric motors and soon (a few years?) batteries will outpace JET-A enabling a massive shift to smaller, cheaper aviation. I think this push will be so strong that it will finally break the aviation rut. Whatever the eventual reason, and there will be a reason, when we get out of it we will see radical changes to the whole industry and we will look back, just like the space industry is now, and ask 'why did it take so long' and 'how did we do things so slowly/badly?'.
Here is a quick aside about ruts. I almost wrote up the math of this once. Ruts are caused by the algorithm we all use, linear regression. It is powerful and good at what it does. Assess, step, assess step. So simple and so powerful but totally misunderstood. In a fixed world if you throw a dart and use linear regression to find a better optimum you will almost always find a way better place than where the dart initially landed. But we don't live in a fixed world. In a dynamic world you still think you are selecting for optimality, but in reality you are selecting for stability. If the world keeps changing but where you are at is standing still then you are in a rut, a stable point. A rut is a local optimum where the walls are higher than your assessment vision and those walls, for whatever reason, don't get lower than your point even though the world is changing. If they did you would see the route out and take it. An algorithm to bust out of this is to double your step size every time you go back to your starting point. If you do this then eventually you will 'break out' and finally be able to find a new optimum that is likely way better than where you were at before. An interesting counter-intuitive consequence of all this is that the more visionary the industry, the bigger the eventual rut it will get caught in. Aviation, space, medicine have all been in massive ruts -because- the people in them are so smart and can see so far. When they finally got caught in their ruts those ruts were massive. The break out will be equally massive too (as we are seeing with space right now)
There is a lot of fun math behind ruts. I am surprised by how underdeveloped it is. One thing is sure though, aviation is in a deep one and when it does break out it will be amazing.