`SECRETS_postgres_password` not documented but required
I am in the process of upgrading from old `zulip/docker-zulip:11.6-0` to the new `ghcr.io/zulip/zulip-server:11.6-0` image.
I managed to find the docs at <https://zulip.readthedocs.io/projects/docker/en/latest/how-to/compose-upgrading-from-legacy.html>.
I translated my docker compose file to the new "format" with the secrets and all that. (Since my docker compose file is an Ansible template, I did not copy the file from the repository, but adapted my old Ansible template. This was less work.)
```yaml
services:
{{zulip_postgres_database_host}}:
image: "{{zulip_postgres_database_docker_image}}"
container_name: "{{zulip_postgres_database_host}}"
restart: unless-stopped
environment:
POSTGRES_DB: "{{zulip_postgres_database_name}}"
POSTGRES_USER: "{{zulip_postgres_database_user}}"
PGPORT: "{{zulip_postgres_database_port_internal}}"
POSTGRES_PASSWORD_FILE: "/run/secrets/zulip__postgres_password"
secrets:
# Note that you need to do a manual `ALTER ROLE` query if you
# change this on a system after booting the postgres container
# the first time on a host. Instructions are available in README.md.
- zulip__postgres_password
volumes:
- "{{zulip_postgres_database_docker_volume}}:{{zulip_postgres_database_docker_volume_mount_point}}:{{zulip_postgres_database_docker_volume_read_write_permission}}"
networks:
- "{{zulip_docker_network}}"
# See: https://docs.docker.com/engine/logging/configure/
logging:
driver: "json-file"
options:
max-size: "16m"
max-file: "4"
compress: "false"
{{zulip_memcached_host}}:
image: "{{zulip_memcached_docker_image}}"
container_name: "{{zulip_memcached_host}}"
restart: unless-stopped
environment:
SASL_CONF_PATH: "{{zulip_memcached_sasl_conf_path}}"
MEMCACHED_SASL_PWDB: "{{zulip_memcached_sasl_pwdb}}"
MEMCACHED_PASSWORD_FILE: "/run/secrets/zulip__memcached_password"
secrets:
- zulip__memcached_password
command:
- "sh"
- "-euc"
- |
echo 'mech_list: plain' > "$$SASL_CONF_PATH"
echo "zulip@$$HOSTNAME:$$(cat $$MEMCACHED_PASSWORD_FILE)" > "$$MEMCACHED_SASL_PWDB"
echo "zulip@localhost:$$(cat $$MEMCACHED_PASSWORD_FILE)" >> "$$MEMCACHED_SASL_PWDB"
exec memcached -S
networks:
- "{{zulip_docker_network}}"
# See: https://docs.docker.com/engine/logging/configure/
logging:
driver: "json-file"
options:
max-size: "8m"
max-file: "4"
compress: "false"
{{zulip_rabbitmq_host}}:
image: "{{zulip_rabbitmq_docker_image}}"
container_name: "{{zulip_rabbitmq_host}}"
restart: unless-stopped
environment:
RABBITMQ_DEFAULT_USER: "{{zulip_rabbitmq_default_user}}"
RABBITMQ_PASSWORD_FILE: "/run/secrets/zulip__rabbitmq_password"
secrets:
- zulip__rabbitmq_password
command:
- "sh"
- "-euc"
- |
export RABBITMQ_DEFAULT_PASS="$$(cat $$RABBITMQ_PASSWORD_FILE)"
echo 'default_user = $$(RABBITMQ_DEFAULT_USER)' >> /etc/rabbitmq/rabbitmq.conf
echo 'default_pass = $$(RABBITMQ_DEFAULT_PASS)' >> /etc/rabbitmq/rabbitmq.conf
exec docker-entrypoint.sh rabbitmq-server
volumes:
- "{{zulip_rabbitmq_volume}}:{{zulip_rabbitmq_volume_mount_point}}:rw"
networks:
- "{{zulip_docker_network}}"
# See: https://docs.docker.com/engine/logging/configure/
logging:
driver: "json-file"
options:
max-size: "8m"
max-file: "4"
compress: "false"
{{zulip_redis_host}}:
image: "{{zulip_redis_docker_image}}"
container_name: "{{zulip_redis_host}}"
restart: unless-stopped
environment:
REDIS_PASSWORD_FILE: "/run/secrets/zulip__redis_password"
secrets:
- zulip__redis_password
command:
- "sh"
- "-euc"
- '/usr/local/bin/docker-entrypoint.sh --requirepass "$$(cat $$REDIS_PASSWORD_FILE)"'
volumes:
- "{{zulip_redis_volume}}:{{zulip_redis_volume_mount_point}}:rw"
networks:
- "{{zulip_docker_network}}"
# See: https://docs.docker.com/engine/logging/configure/
logging:
driver: "json-file"
options:
max-size: "8m"
max-file: "4"
compress: "false"
{{zulip_service_name}}:
image: "{{zulip_docker_image}}"
container_name: "{{zulip_service_name}}"
restart: unless-stopped
build:
context: .
args:
# Change these if you want to build zulip from a different repo/branch
ZULIP_GIT_URL: "{{zulip_git_repository_url}}"
ZULIP_GIT_REF: "{{zulip_git_repository_ref}}"
# Set this up if you plan to use your own CA certificate bundle for building
# CUSTOM_CA_CERTIFICATES:
depends_on:
- {{zulip_postgres_database_host}}
- {{zulip_memcached_host}}
- {{zulip_rabbitmq_host}}
- {{zulip_redis_host}}
ports:
- "{{zulip_http_port_external}}:{{zulip_http_port_internal}}"
- "{{zulip_https_port_external}}:{{zulip_https_port_internal}}"
environment:
DB_USER: "{{zulip_postgres_database_user}}"
CONFIG_postgresql__database_user: "{{zulip_postgres_database_user}}"
CONFIG_postgresql__database_name: "{{zulip_postgres_database_name}}"
SETTING_REMOTE_POSTGRES_HOST: "{{zulip_postgres_database_host}}:{{zulip_postgres_database_port_external}}"
SETTING_MEMCACHED_LOCATION: "{{zulip_memcached_host}}:{{zulip_memcached_port}}"
SETTING_RABBITMQ_HOST: "{{zulip_rabbitmq_host}}"
SETTING_REDIS_HOST: "{{zulip_redis_host}}"
SETTING_EXTERNAL_HOST: "{{zulip_fqdn}}"
# This will become ZULIP_ADMINISTRATOR in /etc/zulip/settings.py
# See: https://zulip.readthedocs.io/en/latest/production/email.html#how-to-configure
SETTING_ZULIP_ADMINISTRATOR: "{{zulip_admin_email_address}}"
# AUTOMATIC E-MAIL SENDING
# See: https://zulip.readthedocs.io/en/latest/production/email.html#how-to-configure
# THEORY: Probably used to send e-mails to users upon registration.
SETTING_NOREPLY_EMAIL_ADDRESS: "{{zulip_no_reply_email_address}}"
SETTING_EMAIL_HOST: "{{smtp_host}}"
SETTING_EMAIL_HOST_USER: "{{zulip_no_reply_email_address}}"
SETTING_EMAIL_PORT: "{{smtp_port}}"
# It seems that the email server needs to use ssl or tls and
# can't be used without it.
SETTING_EMAIL_USE_SSL: "False"
SETTING_EMAIL_USE_TLS: "True"
ZULIP_AUTH_BACKENDS: "EmailAuthBackend"
# Uncomment this when configuring the mobile push notifications service
# SETTING_PUSH_NOTIFICATION_BOUNCER_URL: 'https://push.zulipchat.com'
######################
# NOT SURE IF NEEDED #
######################
# CONFIG_http_proxy__allow_addresses: "NOT-YET-SET"
# CONFIG_http_proxy__allow_ranges: "NOT-YET-SET"
# CONFIG_application_server__queue_workers_multiprocess: "NOT-YET-SET"
# SETTING_REMOTE_POSTGRES_SSLMODE: "NOT-YET-SET"
# LOADBALANCER_IPS: "{{proxy_ip_address.stdout}}"
# Alternately, you can default to trusting the Docker gateway IP:
# TRUST_GATEWAY_IP: True
secrets:
- zulip__postgres_password
- zulip__memcached_password
- zulip__rabbitmq_password
- zulip__redis_password
- zulip__secret_key
- zulip__email_password
volumes:
- "{{zulip_docker_volume}}:{{zulip_docker_volume_mount_point}}:{{zulip_docker_volume_read_write_permission}}"
# NGINX configuration
- type: bind
source: "{{zulip_nginx_access_logs}}"
target: "{{zulip_nginx_access_logs_mount_point}}"
- type: bind
source: "{{zulip_nginx_error_logs}}"
target: "{{zulip_nginx_error_logs_mount_point}}"
# Zulip's own configuration file
- type: bind
source: "{{zulip_configuration_file}}"
target: "{{zulip_configuration_file_mount_point}}"
- type: bind
source: "{{zulip_error_log_file}}"
target: "{{zulip_error_log_file_mount_point}}"
- type: bind
source: "{{zulip_server_log_file}}"
target: "{{zulip_server_log_file_mount_point}}"
- type: bind
source: "{{zulip_email_logs}}"
target: "{{zulip_email_logs_mount_point}}"
networks:
- "{{zulip_docker_network}}"
- "{{proxy_network}}"
ulimits:
nofile:
soft: 1000000
hard: 1048576
# container resource limits
# CPU in microseconds
# every 1s
cpu_period: 1000000
# the container may use 0.5s of a whole CPU core, switching between cores as the OS sees fit
cpu_quota: 500000
# RAM limits
# use at max 2GB of memory and 2GB of swap --> 4GB in total
# mem_reservation: "10M"
mem_limit: "3072M"
memswap_limit: "6144M"
# See: https://docs.docker.com/engine/logging/configure/
logging:
driver: "json-file"
options:
max-size: "32m"
max-file: "4"
compress: "false"
volumes:
{{zulip_docker_volume}}:
name: "{{zulip_docker_volume}}"
{{zulip_postgres_database_docker_volume}}:
name: "{{zulip_postgres_database_docker_volume}}"
{{zulip_rabbitmq_volume}}:
name: "{{zulip_rabbitmq_volume}}"
{{zulip_redis_volume}}:
name: "{{zulip_redis_volume}}"
networks:
{{proxy_network}}:
name: "{{proxy_network}}"
external: true
{{zulip_docker_network}}:
name: "{{zulip_docker_network}}"
external: true
secrets:
zulip__postgres_password:
environment: "ZULIP__POSTGRES_PASSWORD"
zulip__memcached_password:
environment: "ZULIP__MEMCACHED_PASSWORD"
zulip__rabbitmq_password:
environment: "ZULIP__RABBITMQ_PASSWORD"
zulip__redis_password:
environment: "ZULIP__REDIS_PASSWORD"
zulip__secret_key:
environment: "ZULIP__SECRET_KEY"
zulip__email_password:
environment: "ZULIP__EMAIL_PASSWORD"
# To add secrets check:
# https://zulip.readthedocs.io/projects/docker/en/latest/how-to/compose-secrets.html#additional-secrets.
```
In my `.env` file the structure looks like this:
```env
ZULIP__POSTGRES_PASSWORD={{zulip_postgres_database_password}}
ZULIP__MEMCACHED_PASSWORD={{zulip_memcached_password}}
ZULIP__RABBITMQ_PASSWORD={{zulip_rabbitmq_default_password}}
ZULIP__REDIS_PASSWORD={{zulip_redis_password}}
ZULIP__SECRET_KEY={{zulip_secret_key}}
ZULIP__EMAIL_PASSWORD={{zulip_no_reply_email_password}}
# To add secrets check:
# https://zulip.readthedocs.io/projects/docker/en/latest/how-to/compose-secrets.html#additional-secrets.
```
Note, that following the upgrade guide I named the postgres password `ZULIP__POSTGRES_PASSWORD` in the env file, which is referenced in the `secrets` section of the docker compose file.
I have managed to get my containers up, but the `zulip` container keeps crashing. Some way down in the logs, I see:
```
zulip | Executing Zulip configuration ...
zulip | Setting key "EMAIL_HOST", type "string" in file "/etc/zulip/settings.py".
zulip | Setting key "EMAIL_HOST_USER", type "string" in file "/etc/zulip/settings.py".
zulip | Setting key "EMAIL_PORT", type "integer" in file "/etc/zulip/settings.py".
zulip | Setting key "EMAIL_USE_SSL", type "bool" in file "/etc/zulip/settings.py".
zulip | Setting key "EMAIL_USE_TLS", type "bool" in file "/etc/zulip/settings.py".
zulip | Setting key "EXTERNAL_HOST", type "string" in file "/etc/zulip/settings.py".
zulip | Setting key "MEMCACHED_LOCATION", type "string" in file "/etc/zulip/settings.py".
zulip | Setting key "NOREPLY_EMAIL_ADDRESS", type "string" in file "/etc/zulip/settings.py".
zulip | Setting key "RABBITMQ_HOST", type "string" in file "/etc/zulip/settings.py".
zulip | Setting key "RABBITMQ_USER", type "string" in file "/etc/zulip/settings.py".
zulip | Setting key "RATE_LIMITING", type "bool" in file "/etc/zulip/settings.py".
zulip | Setting key "REDIS_HOST", type "string" in file "/etc/zulip/settings.py".
zulip | Setting key "REDIS_PORT", type "integer" in file "/etc/zulip/settings.py".
zulip | Setting key "REMOTE_POSTGRES_HOST", type "string" in file "/etc/zulip/settings.py".
zulip | Setting key "ZULIP_ADMINISTRATOR", type "string" in file "/etc/zulip/settings.py".
zulip | Zulip configuration succeeded.
zulip | Auto backup enabled.
zulip | === End Initial Configuration Phase ===
zulip | === Begin Bootstrap Phase ===
zulip | Waiting for database server to allow connections ...
zulip | /sbin/entrypoint.sh: line 338: SECRETS_postgres_password: parameter not set
zulip exited with code 1 (restarting)
```
I see that `SECRETS_postgres_password` conforms to the naming scheme, but it is not documented anywhere. I looked at the following places:
* <https://github.com/zulip/docker-zulip/blob/ead7d71f42a86b2c1079c9caeac75bfda95fc02f/README.md>
* https://zulip.readthedocs.io/projects/docker/en/latest/how-to/compose-secrets.html
* https://zulip.readthedocs.io/projects/docker/en/latest/how-to/compose-upgrading-from-legacy.html
The only place I can find it is in issues, where people have the same error message, but the solutions are not always stated very clearly. My confusion now is:
1. Are `SECRETS_postgres_password` and the env var `ZULIP__POSTGRES_PASSWORD` something different? For different purposes?
2. If they are not something different, then why have 2 names? Or is the name I have just outdated?
3. What is the correct setup?
Some comments that are not clear:
* https://github.com/zulip/docker-zulip/issues/272#issuecomment-804045300 What? The secret is now in a file and no longer an environment variable in the container. And the secrets file is referenced in the database container description that I have: `POSTGRES_PASSWORD_FILE: "/run/secrets/zulip__postgres_password"`. Does the _filename_ matter??
* https://github.com/zulip/docker-zulip/issues/272#issuecomment-1017197718 This comment seems to contradict the upgrade guide. The upgrade guide talks about:
```yaml
secrets:
zulip__postgres_password:
environment: "ZULIP__POSTGRES_PASSWORD"
zulip__memcached_password:
environment: "ZULIP__MEMCACHED_PASSWORD"
zulip__rabbitmq_password:
environment: "ZULIP__RABBITMQ_PASSWORD"
zulip__redis_password:
environment: "ZULIP__REDIS_PASSWORD"
zulip__secret_key:
environment: "ZULIP__SECRET_KEY"
zulip__email_password:
environment: "ZULIP__EMAIL_PASSWORD"
services:
zulip:
environment:
# Include all settings starting with SETTING_ from your old
# docker-compose.yml, plus any of the renamed settings above.
```
So all is prefixed with "zulip". But those are just _names of the secrets for referencing in docker compose_ not the actual variable names, so I guess it doesn't matter, as long as I pass the service the correct filename.
* The diff at https://github.com/zulip/docker-zulip/pull/527/changes#diff-394168cbf7a0067ccc93452f3678b3b0d5bf76184567e5dacbea9324eb0c9c72 seems to be just what I have:
1. Setting the env var `POSTGRES_PASSWORD_FILE: /run/secrets/zulip__postgres_password` for the database container.
2. Adding the `secrets` list item `zulip__postgres_password` for the database container. (Note the prefix `zulip_` not `SECRET_`, but since that is just the name of the file it should not matter?)
3. Adding the `secrets` list item `zulip__postgres_password` for the zulip container. Of course they need to share that secret. Makes sense.
4. `env` file with `ZULIP__POSTGRES_PASSWORD=postgres_password`.
There is one difference in my setup, that I can see: I don't want to use a compose override separate file. It would be useless for me, and just another additional redirection. The upgrade guide has it: <https://zulip.readthedocs.io/projects/docker/en/latest/how-to/compose-secrets.html#store-secrets-in-files-in-a-directory> but I don't see why that should cause any issue, since it's just merged, overriding what one has in ones `docker-compose.yaml` file at the end of the day. So I should be fine to have the `secrets` key in my normal `docker-compose.yaml` file.
I could set the environment variable `SECRETS_postgres_password`, but then I think it would defeat the purpose of the whole switch to docker secrets, again exposing it in environment variables.
For example https://github.com/zulip/docker-zulip/issues/272#issuecomment-1326628502 seems to suggest to `export` the secrets as env vars. Dubious solution?
2 条评论