Repository navigation
bake cache misses with multiple targets #414
Description
I was hoping to speed up builds by using the parallelism provided by bake.
Therfore i'm running buildx bake -f docker-bake.hcl
group "default" {
targets = [ "s1","s2","s3","s4","s5","s6","s7","s8","s9","s10","s11","s12","s13","s14","s15","s16","s17","s18","s19","s20","s21"]
}
target "s1" {
context = "./s1"
}
target "s2" {
context = "./s2"
}
target "s3" {
context = "./s3"
}
...
target "s21" {
context = "./s21"
}but nearly all of the time one (or more) of the images gets rebuilt, even if they should hit the cache.
=> CACHED [s1 build-app.web 2/7] WORKDIR /app 0.0s
=> CACHED [s1 build-app.web 3/7] COPY package*.json /app/ 0.0s
=> CACHED [s1 build-app.web 4/7] RUN npm ci 0.0s
=> CACHED [s1 build-app.web 5/7] COPY ./*.json /app/ 0.0s
=> CACHED [s1 build-app.web 6/7] COPY ./src /app/src 0.0s
=> [s19 build 6/6] RUN npm run build 2.8s
If i specify single targets, the cache is used as i would expect:
for i in {1..21}; do buildx bake -f docker-bake.hcl s$i; done
What if found out so far:
- The more default targets i have, the less likely the cache is used for some of them.
- When i try to reproduce this with trivial Dockerfiles containing only a single
RUNthe problem doesn't occure
Are there any ideas why this is happening or how i could investigate this further?
Reacted by Adrien Nouvel, inickles-grapl, saruman9, Jean-Frédéric Gauron, Hari, Ruffin, Steffen Hanikel, Yotam Tanay, Nigel Banks, Leonardo Cencetti and 3 more
Activity
[-]bake missing cache with multiple targets[/-][+]bake cache misses with multiple targets[/+]weber-software commentedon Dec 21, 2020
Tested it again with 74f76cf but the problem still occures.
ruffsl commentedon May 2, 2024
I'm also encountering this issue. Baking one target at a time seems to prevent any erroneous cache misses.
At first, I had thought my issue was related to baking targets that spanned across multiple Dockerfile, and thus the multiple contexts was throwing buildkit for a loop. However, even when consolidating all my bake tarkets into a single Dockerfile, I can occasionally catch an instance where a stage that was cached for one target is somehow missed for another target, even when both targets share the same stages and context/inputs.
For example, when baking both
validatorandtoolertargets for the following:I'll occasionally the layer for
[tooler runner 4/4]miss the same layer cached for[validator runner 4/4]Note the difference between CACHED step
#28vs missed step#40. In particular, on step#31thevalidatortarget starts to fetch the cache for the layerrunner 3/4, which is then followed by (another?) step#31that starts to fetch the cache for layerrunner 3/4as well. On step#37we can see the layer is successfully concluded asCACHED, but then later we can see step#31finalize it'sextracting. The following step#40then subsequently misses it's cache, unline step#28that concludedrunner 4/4asCACHED.I realize some of the stdout can be out of sync due to line buffering, but I think this hints at some kind of race condition when download and extracting layers that can be used in caching for multiple targets.
@crazy-max or @tonistiigi , I can try including more telemetry if there is recommended method capturing traces, but from the log below you'll see this is using
docker.io/docker/dockerfile:1.7:Bake Log
Bake definition
{ "group": { "default": { "targets": [ "validator", "tooler" ] } }, "target": { "tooler": { "attest": [ "type=provenance,disabled=true" ], "context": ".", "dockerfile": "Dockerfile", "args": { "DOCKER_META_IMAGES": "<URL>/<REPO>", "DOCKER_META_VERSION": "<BRANCH>-tooler" }, "tags": [ "<URL>/<REPO>:<BRANCH>-tooler" ], "cache-from": [ "type=registry,ref=<URL>/<REPO>:main-tooler.cache", "type=s3,blobs_prefix=cache/<REPO>/,manifests_prefix=cache/<REPO>/,region=eu-west-2,bucket=<BUCKET>" ], "cache-to": [ "type=s3,blobs_prefix=cache/<REPO>/,manifests_prefix=cache/<REPO>/,region=eu-west-2,bucket=<BUCKET>,mode=max" ], "target": "tooler", "output": [ "type=image,push=true,push=true,push=true,push=true,push=true" ], "pull": true, "no-cache": false }, "validator": { "attest": [ "type=provenance,disabled=true" ], "context": ".", "dockerfile": "Dockerfile", "args": { "DOCKER_META_IMAGES": "<URL>/<REPO>", "DOCKER_META_VERSION": "<BRANCH>-validator" }, "tags": [ "<URL>/<REPO>:<BRANCH>-validator" ], "cache-from": [ "type=registry,ref=<URL>/<REPO>:main-tooler.cache", "type=s3,blobs_prefix=cache/<REPO>/,manifests_prefix=cache/<REPO>/,region=eu-west-2,bucket=<BUCKET>" ], "cache-to": [ "type=s3,blobs_prefix=cache/<REPO>/,manifests_prefix=cache/<REPO>/,region=eu-west-2,bucket=<BUCKET>,mode=max" ], "target": "validator", "output": [ "type=image,push=true,push=true,push=true,push=true" ], "pull": true, "no-cache": false } } }dud225 commentedon Jun 25, 2024
I've stumbled on this issue as well.
However when switching to a different cache backend I observe the following behaviour:
That makes me understand that multiple targets steps on each other's toes when not explicitely configured to use a separate storage area.
So I assume that the default internal cache stores all the data in the same storage area and is not suited for use with multiple targets.
ruffsl commentedon Sep 9, 2024
@dud225 or @weber-software , have you tried updating the dockerfile version used? For me, updating from v1.7 to v1.9 may have resolved the superfluous cache layer busting I've encountered!
From the change log, I suspect this may have helped in dealing with my more advanced Dockerfile staging DAG:
nigelgbanks commentedon Jan 29, 2025
Unfortunately, I see cache misses even when using a different storage location for each image.
I should say my setup is slightly different in that many of my targets, use other target as contexts. Something like the following.
c0m1c5an5 commentedon May 8, 2025
Indeed, I have the same setup, which also misses. By trial and error, I found that whenever there are at least two targets that use the same target as context, cache does not seem to work. I have two bake jsons attached, single dependant works properly, while having two or more breaks caching. It is for sure not the fault of the additional target between the jsons as using a different, or even more produces the same result. I will try to create a minimal reproduction repo.
bake-one-dependant.json
bake-two-dependants.json
c0m1c5an5 commentedon May 8, 2025
This is not as minimal as I wanted, but still pretty small.
This reproduction repo can be used to get the described issue, follow the instructions in the readme.
It is fully reproducible with my environment and GitLab registry.
https://github.com/c0m1c5an5/buildx-414-reproducer
eatdrinksleepcode commentedon Jul 23, 2025
I am also experiencing this problem. I am also using some targets as contexts for others, so running the targets one at a time doesn't even help: as soon as I run the top-level target, the lower-level targets miss their cache (even if they have just run).
This unfortunately kills most of the value proposition that I was looking to get from moving to bake.
tonistiigi commentedon Jul 24, 2025
@c0m1c5an5 Seem to work fine for me
I guess you are expecting the
(touch /var/cache/yum/**...)lines to not match cache. I did need to addmicrodnf-- refresh makecacheto the install commands as the pkg-cache seems to be stale, but (unless it is a pure timing thing) I doubt the actual command needs to be exact.rcwbr commentedon Nov 2, 2025
@c0m1c5an5 / @nigelgbanks I hit this issue as well, and boiled it down to a fairly minimal case before finding this thread, as well as a workaround (basically adding otherwise-unused
target:[target]contextspointing to each target that the selected target(s) depend on, including transitively). I hope this is helpful!@tonistiigi I also was able to verify that my example only manifests this issue when using remote registries (or at least does not reproduce using a local
distributionregistry). In your log, it looks like your registry is at least LAN-local -- so I'm curious what you'd see if using a remote registry... and if this all may come down to timing.nbanmp commentedon Dec 15, 2025
@rcwbr I unfortunately am also seeing this issue even when adding all the contexts you described. I also can confirm that I only see this with remote registries.
My workaround was different. I'm adding the base image as an explicit
cache-fromto the app target.c0m1c5an5 commentedon Jan 1, 2026
@rcwbr and @nbanmp Sadly (or maybe it is good actually), neither mine nor your examples are currently reproducible for me any more. It is either that a fix has been merged that unintentionally fixes this issue, or the GitLab registry, which I used for all experiments made some changes.
In any case, our current reproducer approach is incomplete, as we rely on an external registry. So the real reproducer should use a local registry started with
docker run -d -p 5000:5000 --restart always --name registry registry:3. It is also possible that the characteristics of the connection quality play a role here, which can be simulated bytcusing for example https://github.com/lukaszlach/docker-tc. So the proper reproducer should include the instructions on how to start a local registry and use it to store the cache.Currently the issue does not reproduce for me, but in case it still exists, please create a reproducer with a local registry. This will in addition allow us to cross check if, maybe, the issue is in the registry implementation of the provider.