Typically you need the intermediates to compute if you need the next one , so you cannot skip to final until you have the intermediaries.
The final asset/artifact is rarely small either. even best optimized artifacts can be few hundred MB docker image or more commonly multiple image layers running GBs in size .
each step is a network pull then recompute cache if stale and keep going till end .
We solve it 2 ways in the Bazel ecosystem: for the intermediate artifacts, we only fetch the digest (hash + size) of the blobs to calculate the merkle tree forward. The blob itself can stay on the remote cache server.
For the bigger final artifacts, we support using Content Defined Chunking (rolling gear hashing) to only fetch the missing chunks between incremental builds. Binaries executable with stable layout benefits from this quite a lot.
We are definitely not done with all of the improvements here. But since all the major AI labs are using Bazel, we know that the tools can support “Agent Scale”. https://webazel.dev/
> calculate the merkle tree forward. he blob itself can stay on the remote cache server.
Not sure how that would work with building say a docker image, reproducible builds are pretty hard problem to solve, and caching intermediate layers is not always simple or even doable, we typically still need to publish to a registry which is not the cache server.
The Bazel ecosystem builds container images not by using Dockerfile, which contains non-reproducible primitives such as RUN and others. We do it by actually constructing the file trees and tarballs manually, then using them to compose the JSON manifest and indices. This is done via smaller hermetic and reproducible Bazel actions and thus enables the ecosystem to scale way beyond what alternative BuildKit-based solutions can.
The final asset/artifact is rarely small either. even best optimized artifacts can be few hundred MB docker image or more commonly multiple image layers running GBs in size .
each step is a network pull then recompute cache if stale and keep going till end .