The problem is that much of the demand for cheap, shelf-stable backup media has gone away over the last decade or two. Most backup jobs either go to cloud or to disk, not tape. And because of that, tape drives are more expensive than ever, which takes away much of the benefit of the cheap media.
LTO, for example, is relegated to archival use-cases nowadays, such as video production. You'd think that would make sense, because the media's shelf stable... but it's actually really awful for archival. Why? Because new drives can't read old tapes past two generations, and old drives stop getting made. So every time a new tape drive generation comes out, a movie studio can't just upgrade drives in their libraries and use the new tapes. Nor can they just hold onto their old drives forever - what happens when they start breaking, and they can't read their old tapes anymore? (Or worse, an old/failing drive eats a tape?)
Every movie studio with an archival program is on a constant upgrade treadmill, including costly media migrations. That is, they're copying data from old tapes onto new, so they can maintain compatibility with new drives. This entirely defeats the purpose of shelf-stable media; if I can't buy newly manufactured drives for old tape formats then who cares if the tapes themselves can sit in a box for 20 years?
Now, let's say you're not a movie studio with a massive digital archival problem. You just want to backup 5 terabytes. Unfortunately, 5 terabytes is so little that tape vendors have forgotten how to count that low. It's far cheaper (not to mention, more performant) to just buy a bunch of disk drives and migrate data between them. Or just pay Amazon to do it. Which is what everyone wound up doing.
To add: LTO speeds are currently in the 300MB/s range: if you want > TB size storage you likely don't want ~100MB/s for regular use. I work on systems where we are regularly writing multiple tapes of 5+ TB and if you get only 100MB/s you're waiting hours and if something fails or gets stuck you can be thrown off schedule by more than a day very quickly. Even at the current 300MB/s speeds it's very slow. If personal archival or even small-scale archival is your interest getting network storage is probably easier, faster and cheaper. What's more, is that you have to keep some kind of inventory of your tapes so you know where to find data if you ever need it. Depending on the size of your collection you can be looking for even more expensive equipment to manage the archive of your archives.
Not to mention that 300MB/s is considered best-case conditions for sequentially stored data. Anything even remotely random is going to have far slower archival times, unless you have archival software that does parallel I/O. (In my experience, not many archivers do this.)
Most production tape installations include dedicated disk storage to act as a buffer for the drives. You archive to disk first, so that your drives aren't tied up, and then write the now-sequential data to tape.
Tape drive backwards compatibility is an issue for sure but optical drives have had decent backwards compatibility so far. DVD drives can read CD's. BR drives can read CD, DVD and BR. I think backwards compatibility is more feasible for optical as the larger pits on older media can still be read by narrower wavelengths. They'll just look like giant craters flashing by instead of nice neat pits.
AFAIK I believe that they actually do use different lasers for different disc types, but the cost to put two or three laser diodes in one drive is basically nil these days.
The big problem with optical storage is the same that tape struggles with - it's primary market shrank.
This was a bit of a self-inflicted wound. CDs and DVDs were basically ubiquitous, drives came down in price dramatically, and the storage densities and pricing was competitive with competing technologies. BD is where that all started breaking down. BD had a format war with HD-DVD, and both formats relied upon new and then-expensive technologies such as blue lasers and quad-layer stacking. PC manufacturers (notably Apple) balked at the cost of either format. Microsoft backed HD-DVD, but wisely decided not to bleed money by not sticking an HD-DVD drive in the Xbox 360. This created an odd situation where the Xbox 360 sold better as a game console, but Blu-Ray outsold HD-DVD because Sony had let the PlayStation division bleed out for a couple of years.
Of course, Microsoft still wanted the 360 to be able to play HD movies, and their response was to partner with Netflix to make streaming a reliable (arguably better) substitute for movies on optical media. In other words, Sony killed HD-DVD, so Microsoft killed the whole home video market by pushing digital distribution for everything.
If you're wondering, there are successor formats to Blu-Ray. Interestingly enough, the whole quad-layer stacking thing was rebranded as BD-XL and UHD-BD (which the PS4 doesn't even support), and there's also a proper successor format for data storage called Archival Disc. Except the latter is basically an LTO competitor - you can't buy bare AD drives and discs and use them like Blu-Rays. The technology only exists in archival systems like Sony's ODA system (Gen 2 and Gen 3; Gen 1 used Blu-Rays), which has discs stacked in cartridges that have to be loaded into special, very expensive readers and media libraries.
> This entirely defeats the purpose of shelf-stable media; if I can't buy newly manufactured drives for old tape formats then who cares if the tapes themselves can sit in a box for 20 years?
Why can't you buy new drives for old tape formats? What new features are these new formats providing?
The new formats have higher storage density than the old ones. First generation LTO (LTO-1) stored 100GB on a tape: how much would you pay for a drive like that today? Unsurprisingly, they don't make them any more. LTO drives are read-compatible with tapes from two generations earlier, so you can read LTO-1 tapes on LTO-3 drives, but LTO-3's capacity was 400GB, so no one buys those any more either. The current generation (LTO-8) has 12TB per tape, so you can see the upgrade treadmill chugging along.
LTO, for example, is relegated to archival use-cases nowadays, such as video production. You'd think that would make sense, because the media's shelf stable... but it's actually really awful for archival. Why? Because new drives can't read old tapes past two generations, and old drives stop getting made. So every time a new tape drive generation comes out, a movie studio can't just upgrade drives in their libraries and use the new tapes. Nor can they just hold onto their old drives forever - what happens when they start breaking, and they can't read their old tapes anymore? (Or worse, an old/failing drive eats a tape?)
Every movie studio with an archival program is on a constant upgrade treadmill, including costly media migrations. That is, they're copying data from old tapes onto new, so they can maintain compatibility with new drives. This entirely defeats the purpose of shelf-stable media; if I can't buy newly manufactured drives for old tape formats then who cares if the tapes themselves can sit in a box for 20 years?
Now, let's say you're not a movie studio with a massive digital archival problem. You just want to backup 5 terabytes. Unfortunately, 5 terabytes is so little that tape vendors have forgotten how to count that low. It's far cheaper (not to mention, more performant) to just buy a bunch of disk drives and migrate data between them. Or just pay Amazon to do it. Which is what everyone wound up doing.