[BUG] --no-cache is not respected for git sources

[willarmiros]

Description

I have a service in a docker compose that builds using a Dockerfile in a remote git repo, defined like so:

services:
  my-service:
      container_name: my-service
      build:
        context: [email protected]:protectai/my-repo.git#main
        dockerfile: Dockerfile
        ssh:
          - default

When re-building a service with the --no-cache flag, I would expect caching to be skipped for ALL parts of the build, including making a fresh call to retrieve the source code from git. However this is not the case, as I can see in the first line of the output:

=> CACHED [internal] load git source [email protected]:protectai/my-repo.git#main                                                       8.9s

I also confirmed that the contents of the container/service are an outdated version of content in that repo. When I change the git ref to be a commit hash or some branch other than main, it then actually re-pulls from the git source as I expect. For example:

=> [internal] load git source [email protected]:protectai/my-repo.git#75e9ec67c74dcddd8cd22c3f3c6ecb947ba55f2a                          2.3s

After changing the ref to the latest commit in the repo, I confirmed the service was actually updated with the latest source code. My ask is for the --no-cache flag to do this for me without needing to make manual changes.

Steps To Reproduce

Have a compose file like:

services:
  my-service:
      container_name: my-service
      build:
        context: [email protected]:protectai/my-repo.git#main
        dockerfile: Dockerfile
        ssh:
          - default

Run docker compose build --no-cache my-service

See in the first line of the output:

=> CACHED [internal] load git source [email protected]:protectai/my-repo.git#main                                                       8.9s

Compose Version

Docker Compose version v2.17.3

Docker Environment

Client:
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.4
    Path:     /Users/williamarmiros/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.17.3
    Path:     /Users/williamarmiros/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/williamarmiros/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.19
    Path:     /Users/williamarmiros/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.4
    Path:     /Users/williamarmiros/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/williamarmiros/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/williamarmiros/.docker/cli-plugins/docker-scan
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  v0.10.0
    Path:     /Users/williamarmiros/.docker/cli-plugins/docker-scout

Server:
 Containers: 7
  Running: 5
  Paused: 0
  Stopped: 2
 Images: 37
 Server Version: 23.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc version: v1.1.5-0-gf19387a
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.49-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 5
 Total Memory: 7.667GiB
 Name: docker-desktop
 ID: 1d550563-0ff9-47f6-a6eb-4014a3cd7bc2
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

[ndeloof]

Can you please double check issue isn't the same with a plain docker buildx build ... command ?

[willarmiros]

@ndeloof thanks for following up, this is in fact also a bug with regular docker build:

docker buildx build --ssh default --progress=plain --no-cache  [email protected]:protectai/my-repo.git#main

This command exhibits the same behavior of not respecting --no-cache when fetching git source. Would you be able to transfer this issue to the appropriate project?

[ndeloof]

@glours do you have permission to transfert issues to docker/buildx ? I don't

[glours]

I think so

[glours]

I don't 😭
@crazy-max I just gave your temp rights to move the issue to buildx, can you do it?

[crazy-max]

this is in fact also a bug with regular docker build:

docker buildx build --ssh default --progress=plain --no-cache  [email protected]:protectai/my-repo.git#main

I can't repro with Git HTTP source. I wonder if this is an SSH forwarding issue when checking out git repos in BuildKit 🤔

What's the output of:

docker buildx du --filter type=source.git.checkout --verbose

[thaJeztah]

Could this be a presentation issue, or did it actually fail to check? ISTR there were cases where it would perform the check, discovered the cache was still valid after checking git, and because of that printed "CACHED" (and there was some discussion if those could be presented as "CHECKED" (or something) to indicate that it did check again.

[crazy-max]

Could this be a presentation issue, or did it actually fail to check?

Looking at OP's report it looks like it:

I also confirmed that the contents of the container/service are an outdated version of content in that repo.

[thaJeztah]

Doh! Overlooked that from my phone; ignore me then 😅

[tonistiigi]

Seeing CACHED for a build step with --no-cache set is expected if that step can be verified with immutable checksum (like image layers, immutable blobs, git checkouts etc.).

If you are building from #main branch then you don't need to set --no-cache for that build to invalidate the cache if you have pushed more commits to the repo. BuildKit will automatically detect that the branch has changed. If you are seeing that this is not the case and your new commit files do not show up in build then this is a different issue, but nothing to do with --no-cache. In that case, we need reproduction steps for it.

[willarmiros]

@crazy-max here's the output:

ID:             52a71bdoqz74dwhjfz150me1z
Created at:     2025-01-10 22:25:02.756039464 +0000 UTC
Mutable:        true
Reclaimable:    true
Shared:         false
Size:           918.8kB
Description:    git snapshot for [email protected]:protectai/my-repo.git#75e9ec67c74dcddd8cd22c3f3c6ecb947ba55f2a
Usage count:    3
Last used:      3 days ago
Type:           source.git.checkout

Would this include activity done on behalf of docker compose? The commit hash here is the top of the main branch, but I'm not sure if this reflects what I originally observed when I was building it via docker compose and it appeared to be pulling in an old version

@tonistiigi thanks for clarifying this - unfortunately the repo in question doesn't get updated frequently, but I will report back next time it does get updated to see if this still happens.

[tonistiigi]

@willarmiros Are you saying that the main branch in this repo does not point to 75e9ec67c74dcddd8cd22c3f3c6ecb947ba55f2a but you still see

=> [internal] load git source [email protected]:protectai/my-repo.git#75e9ec67c74dcddd8cd22c3f3c6ecb947ba55f2a                          2.3s

The du also shows that you already have a local copy of that commit. Everything would be correct if main points to that commit(when you run the build), but if it doesn't then something quite fundamental is wrong.

[willarmiros]

@tonistiigi sorry for the confusion, the main branch does point to 75e9ec67c74dcddd8cd22c3f3c6ecb947ba55f2a

However previously when using docker compose, I observed it pulling an older version even though it was still pointing to main. I don't have any repro evidence of that tho, so it is possible I missed something on my end. So I will wait til there's a new commit to the downstream repo to test again if it properly pulls in those changes or not, and update if so.