[BUG] registry-config not work
bug
<!--
BUG REPORT --- README
-> You're struggling with something and it may not be a bug afterall? Please open a discussion instead: https://github.com/k3d-io/k3d/discussions
-> Think it's for sure a bug? Please fill in as many of the fields below as you can so we can help you best!
@@@ Sponsors @@@
Need this fixed urgently for work? Consider sponsoring this issue via https://issuehunt.io/r/k3d-io/k3d
@@@ Sponsors @@@
-->
## What did you do
- How was the cluster created?
```bash
k3d cluster create big-data \
--image "srcr.proland.org.cn/sanrong/k3s:v1.34.1-k3s1-cuda-12.8.1-base-ubuntu22.04" \
--gpus all \
--agents 3 \
--volume ${CUR}/data:/var/lib/rancher/k3s/storage@all \
--volume ${CUR}/containerd.toml:/etc/containerd/config.d/registries.toml@all \
-p "13880:80@loadbalancer" \
-p "13443:443@loadbalancer" \
--api-port 6550
# --registry-config ${CUR}/registries.yaml \
```
- What did you do afterwards?
`kubectl cluster-info`
## What did you expect to happen
I want the cluster to work.
## Screenshots or terminal output
This is the log of server-0
[server.log](https://github.com/user-attachments/files/23577331/server.log)
## Which OS & Architecture
```
arch: x86_64
cgroupdriver: systemd
cgroupversion: "2"
endpoint: /var/run/docker.sock
filesystem: extfs
infoname: node6
name: docker
os: Ubuntu 22.04.5 LTS
ostype: linux
version: 24.0.7
```
## Which version of `k3d`
```
k3d version v5.8.3
k3s version v1.31.5-k3s1 (default)
```
## Which version of docker
```
Client: Docker Engine - Community
Version: 24.0.7
API version: 1.43
Go version: go1.20.10
Git commit: afdd53b
Built: Thu Oct 26 09:07:41 2023
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 24.0.7
API version: 1.43 (minimum version 1.12)
Go version: go1.20.10
Git commit: 311b9ff
Built: Thu Oct 26 09:07:41 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.26
GitCommit: 3dd1e886e55dd695541fdcd67420c2888645a495
runc:
Version: 1.1.10
GitCommit: v1.1.10-0-g18a0cb0
docker-init:
Version: 0.19.0
GitCommit: de40ad0
```
```
Client: Docker Engine - Community
Version: 24.0.7
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.11.2
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.21.0
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 35
Running: 35
Paused: 0
Stopped: 0
Images: 89
Server Version: 24.0.7
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 nvidia runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 3dd1e886e55dd695541fdcd67420c2888645a495
runc version: v1.1.10-0-g18a0cb0
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.8.0-85-generic
Operating System: Ubuntu 22.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 32
Total Memory: 124.9GiB
Name: node6
ID: 226fce35-5ef7-4318-8b16-741d82458f4f
Docker Root Dir: /data/docker
Debug Mode: false
Experimental: false
Insecure Registries:
localhost:5000
127.0.0.0/8
Registry Mirrors:
http://localhost:5000/
Live Restore Enabled: false
```
Here is registries.yaml
```yaml
x-auth: &auth
auth:
username: XXXX
password: YYYY
tls:
insecure_skip_verify: true
mirrors:
docker.io:
endpoint:
- https://ZZZZ
ghcr.io:
endpoint:
- https://ZZZZ
gcr.io:
endpoint:
- https://ZZZZ
k8s.gcr.io:
endpoint:
- https://ZZZZ
registry.k8s.io:
endpoint:
- https://ZZZZ
quay.io:
endpoint:
- https://ZZZZ
mcr.microsoft.com:
endpoint:
- https://ZZZZ
docker.elastic.co:
endpoint:
- https://ZZZZ
configs:
docker.io:
<<: *auth
ghcr.io:
<<: *auth
gcr.io:
<<: *auth
k8s.gcr.io:
<<: *auth
registry.k8s.io:
<<: *auth
quay.io:
<<: *auth
mcr.microsoft.com:
<<: *auth
docker.elastic.co:
<<: *auth
```
Here is the containerd.toml
```toml
version = 2
[plugins]
[plugins."io.containerd.grpc.v1.cri"]
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://ZZZZ"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."ghcr.io"]
endpoint = ["https://ZZZZ"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."gcr.io"]
endpoint = ["https://ZZZZ"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."k8s.gcr.io"]
endpoint = ["https://ZZZZ"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."registry.k8s.io"]
endpoint = ["https://ZZZZ"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."quay.io"]
endpoint = ["https://ZZZZ"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."mcr.microsoft.com"]
endpoint = ["https://ZZZZ"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.elastic.co"]
endpoint = ["https://ZZZZ"]
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."docker.io".auth]
username = "XXX"
password = "YYY"
[plugins."io.containerd.grpc.v1.cri".registry.configs."ghcr.io".auth]
username = "XXX"
password = "YYY"
[plugins."io.containerd.grpc.v1.cri".registry.configs."gcr.io".auth]
username = "XXX"
password = "YYY"
[plugins."io.containerd.grpc.v1.cri".registry.configs."k8s.gcr.io".auth]
username = "XXX"
password = "YYY"
[plugins."io.containerd.grpc.v1.cri".registry.configs."registry.k8s.io".auth]
username = "XXX"
password = "YYY"
[plugins."io.containerd.grpc.v1.cri".registry.configs."quay.io".auth]
username = "XXX"
password = "YYY"
[plugins."io.containerd.grpc.v1.cri".registry.configs."mcr.microsoft.com".auth]
username = "XXX"
password = "YYY"
[plugins."io.containerd.grpc.v1.cri".registry.configs."docker.elastic.co".auth]
username = "XXX"
password = "YYY"
```
At first, I use registries.yaml, but it not work, I found
```
pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"nvidia-device-plugin-daemonset-b5vzq_kube-system(dc2c100e-2481-4b85-bb55-849c00f4e470)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"nvidia-device-plugin-daemonset-b5vzq_kube-system(dc2c100e-2481-4b85-bb55-849c00f4e470)\\\": rpc error: code = DeadlineExceeded desc = failed to start sandbox \\\"ae2d3b423f893d70189dd94ba4f18ecc3a13bb17d621f403916bc1c2f849d9b0\\\": failed to get sandbox image \\\"rancher/mirrored-pause:3.6\\\": failed to pull image \\\"rancher/mirrored-pause:3.6\\\": failed to pull and unpack image \\\"docker.io/rancher/mirrored-pause:3.6\\\": failed to resolve reference \\\"docker.io/rancher/mirrored-pause:3.6\\\": failed to do request: Head \\\"https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/3.6\\\": dial tcp 199.59.150.45:443: i/o timeout\"" pod="kube-system/nvidia-device-plugin-daemonset-b5vzq" podUID="dc2c100e-2481-4b85-bb55-849c00f4e470"
```
in logs.
Then I changed to use containerd's config.toml, but still not work.
I also exec into the server-0, and typed
```
ctr -n k8s.io image pull docker.io/rancher/mirrored-pause:3.6
```
and output
```
ctr: failed to resolve image: failed to do request: Head "https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/3.6": dial tcp 104.244.46.186:443: i/o timeout: context deadline exceeded
```
However, I can pull `docker.io/rancher/mirrored-pause:3.6` in the host machine with the same mirror config.
0 条评论