Whoa, this seems to take for granted that devs targetting serverless infrastructure _must_ deploy up into the actual serverless infrastructure during development, unless I'm misreading? Why is there not a way to simulate the final infrastructure locally, so that development is possible without extremely inefficient pushes into a "dev" serverless deployment?
As CEO, I wanted to take a moment to update you on a change to our pricing model. Moving forward, there will be a cost increase for each new function that you write and deploy, as well as an additional fee for each variable declared in that function.
We believe this change will enable us to continue to provide you with the best services while also ensuring our business remains sustainable. Our team will provide a detailed quote for any new functions you request, outlining the costs and estimated timeframes.
We are committed to providing our services to everyone, regardless of their background or circumstances. This change will not affect the quality of the services that we provide, and we remain dedicated to providing you with the best possible experience.
Thank you for your continued support, and please do not hesitate to reach out to us with any questions or concerns.
Dagster is OSS and most folks do local development on their laptop which is quite fast. The speedups in this post are for production deployments (i.e. landing a commit in your master branch) and for branch deployments (the deployments that run on every pull request).
I hope it's straight line development on HEAD without branches. Continually rebasing what will eventually land now is cheaper than figuring out mega merges later.
Most of that issue isn't "serverless" itself, but the other things in the ecosystem it might talk to. A generic python AWS lambda is easy enough to simulate locally with nothing other than python and calling your lambda handler from main with a little boilerplate.
You can already do local development with dagster, if you set up a local environment. Some users may not want to set up a local environment with all dependencies, secrets and so on, so they can use the remote environment option. Remote environments are also easy to share with other developers and can be used in addition to local environments.
there are solutions for this. localstack comes to mind, but it is more expensive per user than GitHub enterprise and copilot combined. the localstack devs are very proud of localstack.
Localstack is brilliantly done but no great options for other cloud providers for non-AWS companies. It also can be a bit of a pain doing the initial set up, once done it definitely improves the development process/experience.
Majority of the time for me a docker compose with my dependencies does the job well enough and is much faster to get up and running with
> The key factor behind our decision was the realization that while Docker images are industry standard, moving around 100s of megabytes of images seems unnecessarily heavy-handed when we just need to synchronize a small change.
I think the culprit is more the GitHub Actions cache than Docker since it seems to be hard to get a clean cache management. I'm not sure about caching Docker image layers, but caching the Nix store with GitHub Actions is pretty complicated (not even sure it's possible): this means we have to download all required Nix store paths on each run, but i consider this is because of a GitHub Action cache limitation.
So, did you consider using another CI, which offers better caching mechanisms?
With a CI able to preserve the Nix store (Hydra[1] or Hercules[2] for instance), I think nix2container (author here) could also fit almost all of your requirements ("composability", reproducibility, isolation) and maybe provide better performances because it is able to split your application into several layers [2][3].
Note i'm pretty sure a lot of Docker CI also allows to efficiently build Docker images.
There's been a recent Launch HN of Depot.dev [1] - I've integrated it quickly into my GitHub Actions workflow and it's blazingly fast (13x speedups for me). It also was a drop-in replacement since I was using Docker Bake and Docker Action and Depot mimics that almost fully (except SBOM and provenance bits). It also works with Google Cloud Workload Identity Federation so image pushes to Artifact Registry didn't need any tweaking.
Thanks for the interesting links - I'll check them out! We would need not just another CI but also another container platform because launching a docker container is also slow.
Irrespective of the CI, I believe all cached Docker layers will need to be downloaded onto the build machine before it can be rebuilt.
Still, I believe it is possible to build and deploy faster even with a "docker image only" design and it's something we are still looking at. The question is what is the lower bound here - would be hard to beat "sync a file to a warm container and run it". Pex gives us a pretty good lower bound that is also container platform agnostic.
> would be hard to beat "sync a file to a warm container and run it"
It depends on the size of your Pex file (i don't think you mentioned it and sorry if i missed the info). With a Docker/OCI image, it would be possible to create a layer with only the Python files of your application (without deps and interpreter). (I say "would be possible" because that's currently not easy to achieve with Nix for instance.)
If I had to wait 45 seconds to see a single line of code change run, I would be looking for a new dev environment. Reminds me of compile times in the 1990’s. Serverless is “cool” and all but productivity is what really matters.
I agree it would be fantastic to have sub second deploys! Doing this for user specified Python environments is challenging in different ways than doing it for a JS SDK like Workers. Note that just provisioning a GitHub runner takes about 10s, before any deploy code even starts up. In theory we could rsync the code directly to the code server and reload it. But any of these options require bigger architectural changes.
Also you can already have a local environment setup for faster iteration and use this for the shared environments with other developers, where the speedup is still great to have.
dagster.io is an open source Python library for building data pipelines. When using Dagster your project may depend on other Python data analysis and warehouse libraries like pandas, snowflake, pyspark and so on. Each org's requirements for their data orchestration will be different. When executing the pipelines in Dagster Cloud, we need to replicate their Python environment.
CF Workers run straight on top of V8, IIRC? Which I think it was a great design decision. But I wonder how they achieve that w/ the other backends like rust.
I'm confused, this feels like a complicated way of creating a docker layer.
Building and hashing the dependencies is exactly what adding the requirements file/etc and building a layer does.
> moving around 100s of megabytes of images seems unnecessarily heavy-handed when we just need to synchronize a small change. Consider git – it only ships the diffs, yet it produces whole and consistent repositories.
Just ship the layers, that's literally what docker does, right?
Is this a whole build process just to get around the ephemeral nature of github actions?
Using --cache-from cuts down dramatically on the build times since (assuming your branch-master tag gets rebuilt nightly) at least the Dockerfile layer steps for downloading the base OS image and installing software can be automatically skipped.
But still, even swapping out the last step in the Docker build makes it necessary to pull the entire image first - assuming your average Java application, that's like 500+ MB for OS+Java JRE+OS dependencies. If you're sure that the change is only in the last COPY/ADD step, you still have to pull the image despite it not being needed technically.
So that's a problem with the builder being ephemeral then.
I am fairly sure a docker layer is just a tarfile. If you're just adding, say, python code and performing no execution I feel like you could create the tarfile and add it to the list of layers.
edit - This seemed dismissive of me. It isn't intended like that, I'm genuinely curious as to whether this is the right approach. The post feels intuitively like it's reinventing some things that exist, but perhaps for good reason. I think it'd be nice if everything else was "just docker". Perhaps I'm missing a key point - it feels like just having a build machine with storage would immediately solve the problem, there would be no remote pulling. Is that not a solution?
> If you're just adding, say, python code and performing no execution I feel like you could create the tarfile and add it to the list of layers.
Indeed. The problem is, creating the tarfile is "easy" but manipulating the image metadata and getting that pushed to the repository is not - you need to implement that all on your own as Docker tooling is useless without the full image.
> Perhaps I'm missing a key point - it feels like just having a build machine with storage would immediately solve the problem, there would be no remote pulling. Is that not a solution?
Hard to impossible to do in ephemeral or shared build services. IIRC, you can't add EBS or EFS volumes to Fargate tasks, and with EBS volumes you don't have concurrent access between multiple build runners.
I'm curious to try this now, manifests are just json and I'm pretty sure the registry lets you push these things directly - it's possible with bash alone.
> Hard to impossible to do in ephemeral or shared build services. IIRC, you can't add EBS or EFS volumes to Fargate tasks, and with EBS volumes you don't have concurrent access between multiple build runners.
You can attach EFS to fargate tasks. My more general point there was that it feels like a lot of complexity disappears if you don't have an ephemeral build service and can just use docker as docker.
Something I'd missed in my first readthrough was about using existing running containers - that makes me a little uneasy but "start from scratch" and "update a running system" are fair points to find tradeoffs between them. It all feels like trying to push the tools to work in a way they weren't really intended to.
There are two sides to the problem: the build and the deploy.
> it feels like just having a build machine with storage would immediately solve the problem, there would be no remote pulling. Is that not a solution?
Indeed, ephemeral builders are an issue and this would be an alternative solution. However doing this for an arbitrary numbers of users who want on-demand deploys for a variety of projects is non trivial. Most CI environments give you a "clean environment" at startup.
Even if the build takes 0 seconds, the other part is re-running this docker image (where 100s of MB is unchanged and say the final 1MB layer changed). You need to ship all these layers to a provisioned container and boot it up. Even this is possible to optimize by heavy caching and optimizing the container service - but that's an alternative solution with different trade-offs. With pex, we can update any docker image in-place so it is container service agnostic. It also works with our current service (Fargate) which has famously slow startup times.
> Most CI environments give you a "clean environment" at startup.
Yeah, it was partly that which made me wonder if just removing that would help a lot. Seems weird to keep a clean fresh CI setup and not have that in production
I understand handing off the scaling issues to someone else. Perhaps this is the difference between the "best outcome" and "best solution" as the latter takes into account what already exists.
K8s does some optimisation on image layers but my knowledge there is fuzzy.
Perhaps this is the best solution given the constraints, but it's a shame that the fully supported layering system in docker needs to be recreated essentially due to two services throwing away lots of useful data. A container runner that stops your container, pulls the layers and restarts it would only have the container startup time itself - which is extremely small - and keep everything separated.
K8s may have some more controls for the "incremental deployment" case but I'm less confident about the isolation between pods to run user provided code.
Haven't tested it in a couple years but start time used to correlate with image size (which makes sense). I think the Fargate VMs don't have very many IOPs so you're usually waiting on extracting tarballs for your container to start
With that in mind, it can sometimes be more efficient to ignore Docker later best practices and just try to smash everything down into the smallest single layer possible
It seems strange to use a highly managed deployment environment like Fargate, but then build another deployment tool on top of it to do things in a simpler way.
It feels like EC2 is being reconstructed on a platform meant to hide it.
Or (dare I say) look into EKS. Kubernetes can spin up containers faster than ECS in my experience (as of ~1 year ago). Seems like the ECS control plane just has more latency (even with EC2 instead of Fargate)
Is EKS safe for multi-tenant use? When we looked it appeared unsafe if we want to run our users code next to each other because of possible isolation issues.
I guess that depends on your use case and risk profile. Linux containers are a pretty well established isolation mechanism and you can potentially add some additional safety with per-tenant dedicated nodepools.
If pods have added privileges or there is a really low risk tolerance, maybe that's not enough isolation.
Sounds like you can change the container runtime with EKS (not sure if that impacts AWS support) so you could use gVisor or runvm
Is this approach not just Capistrano but using containers and specific to python? You could just use firecracker microVMs and ansistrano to implement this same workflow but get better isolation from firecracker.
Firecracker is definitely very interesting. Would require more ops work for us to run bare metal EC2 (we currently use Fargate). IIUC reusing pre-existing environments would require us to share ext4 filesystems across the VMs. Not sure if antistrano helps here but will look into it.
Quite cool to see PEX. I've used a similar package, shiv[0], with great results, and I always wondered why these are not used more. I think Python zipapps are really nice for bundling executables.
I'm curious about the remaining 20s to start the code on the container. What's the bottleneck there? It seems like you'd need to identify the container, tell it which S3 object has the new source.pex, the container would download it, and then when you run `PEX_PATH=deps.pex ./source.pex` you're up. All that feels like it should take less than 20s. If picking the container takes long, you could probably start that process as sources.pex is built.
Getting the message to the right container is one bottleneck. Currently this is routed through a couple of hops and includes some polling. This could all be optimized (if we had a direct line to the container) but the same messaging model is used in other contexts and would need architectural changes. Another bottleneck is running `source.pex` itself takes a few seconds to start up because it analyzes user code (and in some cases may do expensive computation.)
But you're right: if `source.pex` as a hello world program, just downloading and running it should be pretty fast - I'd expect around 1s.
It does make sense to avoid having servers that saturate their link with a zillion identical downloads on every deploy; but I wonder why they went for BitTorrent, rather than using multicast the way e.g. Windows enterprise installs do it.
> Consider git – it only ships the diffs, yet it produces whole and consistent repositories.
IIRC git does _not_ ship diffs. It copies whole files even for the tiniest change. The compression layer handles the de-duping, which is a different layer.
