[FEATURE]: Remove requirement for docker in garden publish

[salotz]

Feature Request

Background / Motivation

I am attempting to run garden publish in a CI pipeline which runs in kubernetes containers.

The main issue is the reliance on docker for performing/helping this. The documentation at least claims you need docker locally to make this work.

However, in container based CI having docker available is usually quite difficult and avoided. The gardendev/garden-gcloud images I use don't have it, and it wouldn't work in my CI system anyhow without massive tweaking.

What should the user be able to do?

garden publish should have more options available to set up authentication and credentials other than just "use docker".

Why do they want to do this? What problem does it solve?

Publishing images is currently pretty painful and under documented. In a previous attempt I just fell back to running my own CI jobs with Skopeo and outputting the garden build versions etc. Which was also ugly and requires a lot more CI steps and tooling. Which otherwise Garden usually helps eliminate a lot of.

Suggested Implementation(s)

I don't think this would be too hard to take away as a requirement as I was able to get it working with the assumption that I am using the gardendev/garden-gcloud container. With this I can:

$ gcloud --quiet auth configure-docker us-east1-docker.pkg.dev
# which just writes this out to ~/.docker/config.json
$ cat ~/.docker/config.json
{
  "credHelpers": {
    "us-east1-docker.pkg.dev": "gcloud"
  }
}

$ garden tools container.regctl -- registry set us-east1-docker.pkg.dev --cred-helper docker-credential-gcloud

$ garden publish my-container --tag=latest

So I don't have a proposal for the full system. But I think it should be doable to relax the hard requirement on docker fairly easily.

How important is this feature for you/your team?

🌵 Not having this feature makes using Garden painful

[twelvemo]

Thanks for the feature request. I worked on this in the past and i agree that it is not great to have a dependency on docker for publishing images. In #6208 i added regctl to publish multi-platform images and in #6681 i made sure that we use regctl for all cases where the image exists in a remote registry. The dependency on docker would only remain if the image was build and stored locally because regctl would not work in this scenario. The warning in the docs should reflect that though, oversight on our end.
It looks like the image you want to publish is already pushed to a registry by the build action, is that correct? Have you tried using the publish command from your CI without further workarounds?

[salotz]

It looks like the image you want to publish is already pushed to a registry by the build action, is that correct?

Yes, I have a previous build step that does an in-cluster build (kaniko) and pushes to the registry.

Have you tried using the publish command from your CI without further workarounds?

Yes I tried this first, but ran into regctl not having credentials. The in-cluster build uses the imagePullSecret in the cluster, whereas regctl needs credentials in the local CI runner. AFAIU

[twelvemo]

Yes I tried this first, but ran into regctl not having credentials. The in-cluster build uses the imagePullSecret in the cluster, whereas regctl needs credentials in the local CI runner. AFAIU

Gotcha. I guess we'd need to run regctl from the cluster. Which means you'd add imagePullSecrets for both the original registry and the one to publish to (in case they aren't the same and we are just publishing a different tag). This "publish from cluster" method could be configured both as a cli option or as part of the container provider if it is always the desired way. I am thinking out loud here, wondering what the most intuitive way to do this would be. Did you have any thoughts?

So aside from the incorrect documentation we are not really talking about the need to have docker on your CI runner for publish but more about the need to have the creds for one or more registries on the CI runner.

[salotz]

This issue is mostly for the documentation aspect. Cluster publishes are something I'm also interested in, but I guess would be a bigger issue. I can open another for that for ideas there.

What I think would be the easiest thing to improve the situation that wouldn't require a bunch of documentation explaining how to run garden tools container.regctl -- registry set us-east1-docker.pkg.dev --cred-helper docker-credential-gcloud in your runner, is to just run this automatically during a publish.

You should be able to figure out the credHelper from the docker config.json. Which using DOCKER_CONFIG from #6775 would make that easy to control.

You can document when and how docker is needed. The guide for "Garden in CI" could be updated with some patterns.

So aside from the incorrect documentation we are not really talking about the need to have docker on your CI runner for publish but more about the need to have the creds for one or more registries on the CI runner.

They are related. If I can load the credentials directly into regctl then you don't need docker. Because I don't have docker in the runner, I presume, that the credential helpers are not automatically fed into regctl as its documentation says. So you need to do it with the above command and flags directly on regctl.