[FEATURE]: Improving garden publish

[salotz]

Feature Request

Background / Motivation

From a discussion in: #6776

In general there is a lot more garden publish could do to make release pipelines much simpler to achieve with Garden.

What should the user be able to do?

I'm only really concerned with in-cluster builds personally as it is the most efficient and avoids running docker in a CI runner, or locally, which is a security risk, can be difficult to support cross-platform (e.g. Macs are on ARM, need Docker Desktop, etc.), and generally PITA.

Here are the current issues:

1) Credentials in multiple places

Needing to set up credentials in multiple places even for the same registry. One for the imagePullSecret for in-cluster builds and the same credential in the CI runner so garden can use regctl (and maybe docker) to achieve the image tag/movement.

See these issues for making local push easier to use: #6776 #6775 .

In-cluster pushing is a possible solution which would have the following benefits:

1. Reuse already established cluster secrets and RBAC and avoid the need to additionally inject all these into CI (which is truly a lot of time and PITA).
2. Not require additional tools to be in a local environment (e.g. Docker)

I've already done a bunch of in-cluster pushing manually with Garden using skopeo. Using a ConfigTemplate I can even make it semi-painless to write them. However, this could pretty easily something that garden has just built in and it could more easily reuse credentials via this method.

I can provide some more specific code on how I was using Skopeo if anyone is interested, but its nothing special. You just run skopeo in a pod.

2) Support for multiple publish targets

Currently you can only specify a single publish target:

spec:
  publishId: us-east1-docker.pkg.dev/my-reg/my-image

I often want to run workflows where I want to push an image to multiple registries. Say for instance I want to tag latest in the registry that holds my in-cluster build images. And then later push only the release builds to a public registry.

I can only specify one publishId so I will need to use a custom in-cluster push (like above with Skopeo) to achieve both.

Why do they want to do this? What problem does it solve?

Garden is great for doing a lot container build and tests over a large repo. It is still very difficult to actually control how these builds are published and made usable for downstream consumers outside of the same Garden stack.

I think this may be a little outside of the functionality of "ephemeral environments" goal of Garden which is why its not developed that much.

So I'll add my use case that Garden is an excellent tool for coordinating builds in general! I think mostly because builds then have great integration with the entire test infrastructure and aren't just "builds".

Suggested Implementation(s)

For in-cluster publishes I think this should be a feature of the provider or an additional flag in the spec.

For multiple publish targets I would expand a subsection that allows for more detailed configuration:

kind: Build
type: container
name: my-container
spec:
  publish:
    targets:
      # The same registry used for in-cluster builds
      - name: garden
         path: ${providers.kubernetes.deploymentRegistry.hostname}/${providers.kubernetes.deploymentRegistry.namespace}
      # Another registry for publishing release artifacts
      - name: release
         path: docker.io/my-org/my-container
         # K8s SA that has permissions (perhaps via Workload Identity) to push to the registry
         serviceAccount: release-publisher
         # OR an imagePullSecret
         # imagePullSecret: release-publish

How important is this feature for you/your team?

🌵 Not having this feature makes using Garden painful

[twelvemo]

Hi @salotz, sorry for the long wait for a reply here. I agree with you that the publish experience needs improvement. This is something that we have been aware of for quite a while and we really appreciate you taking time to describe your use-case. The in-cluster publish is a good idea, however it gets a bit more tricky when we also want to take local or cloud builds into account. I will bring this up for some more discussions internally.