My colleague wrote up a guide to pull-through caching dockerhub. Docker's pricing is a little bit funky, because a full image pull with all the layers costs the same as a request where you retrieve no layers at all.
If this AWS feature were out a couple months ago and had support for dockerhub, I'm not sure we would have set this up.
Not to single you out, but not too hard to imagine why companies start enabling rate limiting. I oftentimes wonder the obscene amount of bandwidth being consumed by this developer automation. This page indicates that PyPi serves 900 terabytes daily.
Seems obvious we need more turn-key solutions that enable caching for all of the popular package distributions (PyPi, NPM, Docker, etc). Too hard to configure and everyone will default to the path of least resistance (ie hammering the origin servers).
If you do a docker pull inside us-east-1, you’ll get a direct S3 link instead of a proxied download, and thus neither you nor docker need to pay AWS anything for bandwidth!
I thought cross-account data still cost something; is that not the case?
> Data transferred from an Amazon S3 bucket to any AWS service(s) within the same AWS Region as the S3 bucket (including to a different account in the same AWS Region).
That avoids paying NAT Gateway charges for traffic to S3 and in my testing also made a bit of a latency reduction, which came in handy once when I had a one-off data small file migration.
There are commercial products for doing this, e.g. https://jfrog.com/artifactory/
All the command line package management tools then need to be configured to point at the custom in-house package mirror, rather than connecting to the default public package server over the internet.
- ECR Public Gallery: https://gallery.ecr.aws/docker
- Launch blog: https://aws.amazon.com/blogs/containers/docker-official-imag...
Google supports this within GCP: https://cloud.google.com/container-registry/docs/pulling-cac... which makes it easy to avoid going all the way to Docker Hub for images.
Are there plans to add it in the future?
That sounds like a political
decision and not a technical one.
Looking at the gallery, the images I'm interested in seem to be built/uploaded by a different entity. Which is not great.
Is there any benefit to using pull-through caching with ECR Public images? Seems like it would just add extra storage costs.
If you don't build it, we might build it and open source it ourselves.
We have our images saved into private ECR. We are trying to deploy those using AWS lighthouse or copilot. Haven’t found any material on how to deploy an ECR image to either one of those services. Any references you can provide?
For Lightsail you use the command line to run `aws lightsail push-container-image`. This also automatically manages the ECR and container image push for you. You can read more about that here: https://lightsail.aws.amazon.com/ls/docs/en_us/articles/amaz...
Basically these higher level tools don't require you to start from a preexisting container image in ECR. Instead they help you push your container to the cloud automatically, and you don't even have to touch ECR directly.
Can't speak for every person at AWS, but I can say that I'm personally happy with my work/life balance working at Amazon. I work greater than 40 hours a week at times, but less than 40 hours a week other times. Overall it balances out between my needs and the needs of my broader team. I have the support of a great manager, who has never pushed me to do any extra overtime work that I didn't already want to, but has encouraged me and the team to take time for ourselves when we have been working really hard.
I also think work life balance has a lot to do with how much you enjoy the work. I've worked remote for AWS for nearly 5 years now, and my particular role at AWS gives me a lot of freedom in deciding what I do day to day, from writing blogs and technical content, to creating sample code, engaging with folks on social media, recording videos, travelling and giving talks at conferences, providing feedback on internal product specs, testing preproduction releases before they go live, etc. So the variety keeps me interested.
Hope this insider perspective is helpful!