You can create a new bucket or switch your existing one to "Static Website Hosting" mode to enable the ability to 301 your content going forward. But the URL for the "website" version isn't the same as the REST URL. And again, there's no way to redirect from the old naming scheme to the new one.
If you have content that you've ever linked with one of those URLs, it's stuck there forever.
For customers, no. For Amazon itself, yes. And I think that is what parent commenter meant. That Amazon should 301 all requests that are using old paths.
The certificate concern some are raising is also a furphy.
You can do inline generation of LetsEncrypt certificates with bucket-name-specific CN/SAN.
The fact that bucket names could contain characters which are wholly invalid as DNS labels is a bigger issue.
Back when the www was first released the most common criticism was the lack of back links. This is such a stupid and obvious deficiency that it really wasn't worth even looking into a system that was an obvious stillbirth. It wasn't just the "experts" saying this, but so many of them did because it was just so dumb. So you've probably never heard of this "world wide web" thing -- not only were there only one-way links, but it had its own homegrown markup language dialect and instead of using an ordinary protocol like FTP or even gopher it pointlessly used its own http protocol.
(Also back then there was this research protocol called TCP/IP, which was another waste of time given that the OSI protocol stack was poised to dominate the networks just as soon as a working one was written. I wonder what the modern equivalents are).
"The Book": The Elements of Networking Style: And Other Essays & Animadversions of the Art of Intercomputer Networking, by
M. A. Padlipsky (1985)
The World's Only Know Constructively Snotty Computer Science Book: historically, its polemics for TCP/IP and against the international standardsmongers' "OSI" helped the Internet happen; currently, its principles of technoaesthetic criticism are still eminently applicable to the States of most (probably all) technical Arts-all this and Cover Cartoons, too but it's not for those who can't deal with real sentences.
Standards: Threat or Menace, p. 193
A final preliminary: Because ISORM is more widely touted than TCP/IP, and hence the clearer present danger, it seems only fair that it should be the target of the nastier of the questions. This is in the spirit of our title, for in my humble but dogmatic opinion even a good proposed Standard is a prima facie threat to further advance in the state of the art, but a sufficiently flawed standard is a menace even to maintaining the art in its present state, so if the ISORM school is wrong and isn't exposed the consequences could be extremely unfortunate. At least, the threat / menace paradigm applies, I submit in all seriousness, to protocol standards; that is, I wouldn't think of being gratuitously snotty to the developers of physical standards -- I like to be able to use the same cap to reclose sodapop bottles and beer bottles (though I suspect somebody as it were screwed up when it came to those damn "twist off" caps) -- but I find it difficult to be civil to advocates of "final," "ultimate" standards when they're dealing with logical constructs rather than physical ones. After all, as I understand it, a fundamental property of the stored program computer is its ability to be reprogrammed. Yes, I understand that to do so costs money and yes, I've heard of ROM, and no I'm not saying that I insist on some idealistic notion of optimality, but definitely I don't think it makes much sense to keep trudging to an outhouse if I can get indoor plumbing . . . even if the moon in the door is exactly like the one in my neighbor's.
Appendix 3, The Self-Framed Slogans Suitable for Mounting
IF YOU KNOW WHAT YOU'RE DOING,
THREE LAYERS IS ENOUGH;
IF YOU DON'T,
EVEN SEVENTEEN LEVELS WON'T HELP
On the occasion of The Book's reissuance, Peter Salus wrote a review in Cisco's Internet Protocol Journal which included the following observations:
Padlipsky brought together several strands that managed to result in the perfect chord for me over 15 years ago. I reread this slim volume (made up of a Foreword, 11 chapters (each a separate arrow from Padlipsky's quiver) and three appendixes (made up of half a dozen darts of various lengths and a sheaf of cartoons and slogans) several months ago, and have concluded that it is as acerbic and as important now as it was 15 years ago. [Emphasis added] The instruments Padlipsky employs are a sharp wit (and a deep admiration for François Marie Arouet), a sincere detestation for the ISO Reference Model, a deep knowledge of the Advanced Research Projects Agency Network (ARPANET)/Internet, and wide reading in classic science fiction.
In a lighter vein, The Book has been called "... beyond doubt the funniest technical book ever written."
Also, thanks a lot for the reference, strangely the first time I hear of this book! However, I have to most strongly disagree with the claim of "three layers is enough.", and you'll hopefully come to understand why, after checking the above comment. Which, by the way, DOESN'T speak in favor of OSI AT ALL - nor in favor of IP, for that matter. Two brief quotes from over there:
This does not mean that we should be doing OSI. Good grief, no. …
 Someone will ask, What about IPv6? It does nothing for these problems but make them worse and the problem it does solve is not a problem.
There was nothing particularly special about HTTP or HTML, or even the concept of the web. What made it a success was the availability of a server reference architecture, and, more importantly, a browser. It was easy to try it out, see the value, and get up and running with your own server if you had something to publish.
Discoverability was a problem in the early days. There were printed catalogs of wesites! Backlinks might have helped, but clearly were not a fundamental requirement for the web’s success.
> Discoverability was a problem in the early days.
Yeah, I remember being at a conference in which a smart person (actually a smart person, no snark) said that discoverability would be over, as indexing the web would require keeping a copy of everything which, of course, is completely impossible. And we all nodded, because indeed, that did make sense. And about six months later, when altavista launched, it seemed only to confirm this belief.
Please, consider looking A BIT more at the history of TCP/IP:
http://rina.tssg.org/docs/How_in_the_Heck_do_you_lose_a_laye... Day, John - How in the Heck Do You Lose a Layer!? (2012)
"Recently, Alex McKenzie published an anecdote in the IEEE Annals of the History of Computing on the creation of INWG 96, a proposal by IFIP WG6.1 for an international transport protocol. McKenzie concentrates on the differences between the proposals that lead to INWG 96. However, it is the similarities that are much more interesting. This has lead to some rather surprising insights into not only the subsequent course of events, but also the origins of many current problems, and where the solutions must be found. The results are more than a little surprising."
And here, a rather lengthy excerpt from later in the paper, as I suspect a lot of people might presume that the paper would go for some points it definitely DOESN'T go for:
This does not mean that we should be doing OSI.
Good grief, no. This only implies that the data OSI had to work with brought them to the same structure INWG had come to. OSI would have brought along a different can of worms. OSI was the state of understanding in the early 80s. We have learned a lot more since. There was much unnecessary complexity in OSI and recent insights allow considerable simplification over even current practice.
OSI also split the addresses from the error and flow control protocol. This creates other problems. But the Internet’s course is definitely curious. Everyone else came up with an internet architecture for an internet, except them. These were the people who were continually stressing that they were building an Internet.
Even more ironic is that the Internet ceased to be an Internet on the day most people would mark as the birth of the Internet, i.e. on the flag day January 1, 1983 when NCP was turned off and it became one large network.
It was well understood at the time that two levels of addressing were required. This had been realized in the ARPANET when the first host with redundant network connections was deployed in 1972. The INWG structure provided the perfect solution. It is clear why the Internet kept the name, but less clear why they dropped the Network Layer. Or perhaps more precisely, why they renamed the Network Layer, the Internet Layer. Did they think, just calling it something different made it different?
 There was little or no overlap between SC6/WG2 and INWG.
 And if the politics had not been so intense and research had
continued to develop better understanding, we would have learned it a
 Someone will ask, What about IPv6? It does nothing for these
problems but make them worse and the problem it does solve is not a
http://csr.bu.edu/rina/KoreaNamingFund100218.pdf more slides!
And much more, here:
http://rina.tssg.org/ (I find it rather very strange the RINA folks and the GNUnet folks seem to each pull their own thing instead of working together, it very much seems like a—hopefully NOT inevitable—repeat of the very thing John Day describes in the slides & articles above…)
See also, for a network security perspective:
http://bitsavers.informatik.uni-stuttgart.de/pdf/bbn/imp/BBN... see also Appendix H here, starting on PDF page 180
Something in the back of my mind & the depths of my guts tells me I should link the following here, albeit I remain completely clueless as to why, or how it could seem relevant to—& topical for—any of the above, so, I'll just drop it here without explanation:
(Interesting standards compliance section there, by the way.)
Sorry, but no. It is perfectly factual.
> leave so much detail
I was on mobile (hence the typo). I included the level of detail necessary to make my point.
Speaking of points – did you have one?
With the power of hind-sight; Imagine an internet where Comcast, AT&T, et al, have such granular control over access to your infrastructure. Ala Cart billing based on each addressable service you happen to run for example. DNS hijacking on steroids as another. The development of new protocols for all but biggest "participating" organizations would be stillborn. Capitalism would have strangled the Internet baby in the crib long ago if we had "done it right."
The road to hell is paved with good intentions and these are some of the best intentions.
They already do, tho, we call it port filtering & deep package inspection, and when they do that, people get mighty angry. ;)
Also, you seem to implicitly assume that they'd have come to gain as much power as they have now, but that seems doubtful, given how different this would work.
Also, I think you (unintentionally!) attack a strawman there - the point here consists more of matters like congestion control.
I said what I said about hoping for cooperation between RINA & GNUnet for a reason:
RINA lacks the anti-censorship mechanisms of GNUnet, while GNUnet lacks some of the insights from RINA research.
And those anti-censorship mechanisms would make your point entirely moot.
Yes, they do all that now but it's kludgy, easy to detect, and a much more obvious overreach of their presumed authority. See Comcast/Sandvine fiasco.
As to the implicit assumption of the Telco's powers under such a system. History is my biggest "proof". It's a reasonable assumption given economic and games theory. At best it wouldn't have been the Telco's directly that ended up with the control. Someone would have and the result would still be the same.
How about this: How many terrible government regulations about filtering and censorship that are technologically infeasible with the current internet been not only technically possible, but fully fledged features of an objectively better design?
Again I'm not arguing against research and better designs, just rationalizing what we got.
The IETF/RFC/Working Code/Interop(RIP) approach has given V4 incredibly long legs. At least the OSI model itself kinda survived.
Hindsight is 20/20, of course, but I'm (unpleasantly) surprised they didn't take the thousands of ways a hard cut-over would break the web before drafting and (softly) announcing the initial plan.
> The legacy rules for bucket names in the US East (N. Virginia) Region allowed bucket names to be as long as 255 characters, and bucket names could contain any combination of uppercase letters, lowercase letters, numbers, periods (.), hyphens (-), and underscores (_).
: I assume that new bucket names that break subdomains have been made illegal now so they work with a finite and static list of names. I am sure they can come up with a substitution that doesn't create collisions and with a reserved prefix you can avoid future collisions.
1. it's a few years too late for that.
2. it makes very little sense and is not really workable, the set of valid characters in DNS name is limited and quite finite, you're suggesting amazon just bans e.g. z from bucket names (and not explaining what happens to all existing bucket names with a z in them).
s3.amazonaws.com/abc-xyz -> abc-xyz.s3.amazonaws.com
s3.amazonaws.com/abc-x.y.z -> only punycodeof(abc-x.y.z).encoded.s3.amazonaws.com
So it simply shifts the problem to DNS. To keep requests confidential from sniffers on your LAN or somewhere along the path, you're expected to use something like DoH.
Err, no, countries will just block the legacy bucket URL style and say that only the bad guys would still be using it.
Who's next, domain fronting on Microsoft Azure?
The fact you can get around it by ignoring the cert is a bit irrelevant. It's like saying locks don't work because people can break your window.
And it's not the window. It's like saying locks don't work if the state has a master key, which they do.
If you are talking about China, yeah, Google used to carry that burden. Now GAE, GCloud, Youtube, Gmail are all gone. The whole IP range was blacklisted.
Just because something accidentally works does not mean it will last forever.
The Internet is not what it use to be, and it's going to become more and more difficult to deal with censorship.
The Internet treats censorship as a malfunction and routes around it.
- John Perry Barlow
Responding to feedback, publicly, and explaining what they were trying to do and why they needed to do it, is incredibly refreshing.
This seems like a big PR win for AWS. I'm left trusting and liking them more, not less.
The change here was deprecating an access pattern, not destroying data or anything remotely similar.
And thousands of other books like that.
> designed to provide 99.999999999% durability of objects over a given year. This durability level corresponds to an average annual expected loss of 0.000000001% of objects. For example, if you store 10,000,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000 years
They are referring to durability as corruption in the underlying data store due to known storage-technology risks. If they were to take into account all other risks, they would also have to include the risks of nuclear war, cyber warfare, the US government classifying your organization as a terrorist threat, etc etc.
I'm not supporting the old deprecation policy, in fact I thought it was insane. But if anyone publishes a URL assuming that it won't disappear they've not been paying attention. If the peace corps just migrated to Azure instead the links would also die.
It's very likely that there would be something that replaces URLs in the future.
Amazon promise seems to be that they will always keep your data integrity and keep it accessible. The way I see is that they will be moving my data to newer or better storage types and will keep my data unchanged regardless of any technology change.
To keep with the analogy, Amazon's promise here is that they will always keep the data that you originally stored in your floppy disk, but they cannot promise to give it back to you in a floppy disk. Next year they might give it back to you in a CD and the year after in a cartridge.
The data has been always there, intact. They are just giving it to you in a different medium.
A move operation is representationally an atomic copy and delete.
The original object is clearly no longer there.
It's clear cut breaking backwards compat for no reason at all.
I don't even know why they aren't supporting both API versions indefinitely it doesn't really make any sense it's literally a url rewrite/301 for anything hitting the old domain. Want to avoid our bottleknecked legacy LBs for better performance? Hit the new LBs. Hell even the sdk doing this upfront will alleviate a crap tonne of legacy requests.
People need to stop allowing corps making breaking backwards compat so nonchalantly it's unprofessional. And on top of that, AWS has a really good track record of maintaining backwards compat, allowing them to get away with this is just asking for more down the line.
The think that I wish companies would do in this case is the following. Set a hard date for when the change will happen (hopefully giving at least 18 months). Then, send a weekly (monthly) report detailing the number of things a customer has that isn't compliance. For example, every week AWS could send a report summary to the account holder of any URL accessed via the path structure. They could then login to see more details. A lot of systems are sprawling and people are busy putting out fires so a constant reminder and hard end date keeps it top of mind for people so you don't end up working through the weekend trying to get something back up and running. I wish Apple would do this for deprecated APIs (e.g.), email me regularly that I have an app in the store that is using deprecated APIs and they will stop working on date X.
I’m mystified how they’re planning on doing this. Anybody care to speculate?
~They probably couldn't take the Cloudflare approach of jamming 100 customer domains onto each certificate, since that would leak bucket names too easily.~
Issuing one certificate at a time wouldn't make a difference since they're all submitted to public CT logs. Bucket names shouldn't contain sensitive information and security through obscurity is a bad idea.
Obscurity is a good and sensible layer for defense in depth. Systems A and A' were the only difference for A' is added obscurity will result in A' being more difficult to attack.
Maybe there will be a name translation scheme for bucket names with periods (kinda like punycode for IDNs).
only if that's your only defense, which it was for a lot of old crypto schemes & why the crypto community consensus was to publish algorithms/assume the attacker had the implementation. that mindset isn't universally applicable
In that case, you can create a CloudFront endpoint with your bucket as the origin, and point your domain name at the CloudFront endpoint instead. CloudFront can already handle TLS with arbitrary domain names, so I wouldn't be surprised if this becomes the official recommendation for buckets with dots in their names. You already need to do this anyway if you want to use TLS with your S3-hosted static website, because you're probably not accessing it at a ((sub)sub)subdomain of s3.amazonaws.com.
Not having competition means they got a lot of users, but having long-term feature support means people didn't run away.
So if you don't want to change, you can continue using the old paths. Just might limit access to some new features coming later that are dependent on the virtual host sub domains.
Thank you for taking the time to write this up Jeff.
I think this should be:
"In this example, jbarr-public and jeffbarr-public are bucket names; /images/ritchie_and_thompson_pdp11.jpeg and /classic_amazon_door_desk.png are object keys."
I had thought the same about his background, from https://en.wikipedia.org/wiki/Jeff_Bezos looks like he was technical at college and the first couple of years (or less) of his career.
> He graduated from Princeton University in 1986 with degrees in electrical engineering and computer science.
> After Bezos graduated from Princeton in 1986,... He first worked at Fitel, a fintech telecommunications start-up, where he was tasked with building a network for international trade. Bezos was promoted to head of development and director of customer service thereafter. He transitioned into the banking industry when he became a product manager at Bankers Trust; he worked there from 1988 ...
2.) Only an actual programmer would know what malloc is and how it works (he knows what it is exactly since he has the idea "malloc for the web")
If your app for some arcane reason doesn't understand an HTTP status code that's been around for 20 years... your code is bad and you should feel bad.