Image build & promotion using gcb-docker-gcloud images is fragile because of image pruning
# Context
Related Slack thread: https://kubernetes.slack.com/archives/CCK68P2Q2/p1777389849034759
We had an issue with image promotion when cutting a CAPI patch release (xref: https://kubernetes.slack.com/archives/C08AA0FFWG3/p1776790418051959?thread_ts=1776786272.414509&cid=C08AA0FFWG3)
It looks to us like this might be a bigger issue with the image promotion process that also affects other projects.
The issue:
* Image promotion relies on images previously pushed to the staging registry
* These images are usually build and pushed via GCB (e.g. https://github.com/kubernetes/test-infra/blob/bf804659d4e3f11fee305d4ae2606abc1cedbeb2/config/jobs/image-pushing/k8s-staging-cluster-api.yaml#L25)
* The configuration for the GCB build is in the corresponding branches of the corresponding repo (e.g. release-1.12 in the cluster-api repository, https://github.com/kubernetes-sigs/cluster-api/blob/main/cloudbuild.yaml#L8)
* Cluster API (and probably other project as well) are using the [gcr.io/k8s-staging-test-infra/gcb-docker-gcloud](http://gcr.io/k8s-staging-test-infra/gcb-docker-gcloud) image in the GCB build
* The [gcr.io/k8s-staging-test-infra/gcb-docker-gcloud](http://gcr.io/k8s-staging-test-infra/gcb-docker-gcloud) image seems to be now deleted after 180 days
* EDIT: Clarification: "The latest tag will always be available and the last 2 versions."
# TL;DR
The image promotion in Cluster API (and probably other projects) is using `gcr.io/k8s-staging-test-infra/gcb-docker-gcloud`. The image is configured via `cloudbuild.yaml` files on various branches in the CAPI repository. As the `gcb-docker-gcloud` images are now not always retained anymore we have to keep bumping `gcb-docker-gcloud` on all branches. If we miss that a `gcb-docker-gcloud` image was deleted this will lead to a failed release (i.e. we have a Git tag for which we cannot produce a release anymore, so we have to bump the image and then create a new tag).
This wasn't an issue before because the `gcb-docker-gcloud` images were retained.
# Some ideas
1. Rollback the change to the `gcb-docker-gcloud` retention policy
2. Promote `gcb-docker-gcloud` images and then retain the promoted images
3. Use `gcb-docker-gcloud:latest` (not sure if this is okay as this makes the releases non-reproducible and it might be a supply chain security issue)
关闭于 17 天前 9 条评论