Plugin broker drops notifications on rapid shutdown (race between Dying channel and pending alerts)

### What happened? The chance the notification is sent is about 50%, literally a roll of the dice. ### What did you expect to happen? Notifications should work. ### How can we reproduce it (as minimally and precisely as possible)? Try to send notification with `cscli notifications test [whatever]_default`. ### Anything else we need to know? When `cscli notifications test` or any rapid shutdown triggers the plugin tomb's `Kill()`, the broker's `select` loop can pick the `Dying()` channel before reading `PluginChannel`, causing the enqueued alert to be silently dropped. Additionally, after `watcher.Kill()`, the inner loop races between `Dead()` and `PluginEvents`, alerts queued by `addProfileAlert` via the watcher ticker may be lost if `Dead()` wins. Also I have ready a branch with a fix and if there is interest I can make a pull request. ### Crowdsec version <details> ```console $ cscli version version: Codename: BuildDate: GoVersion: 1.26.1 Platform: linux libre2: WebAssembly User-Agent: crowdsec/-linux Constraint_parser: >= 1.0, <= 3.0 Constraint_scenario: >= 1.0, <= 3.0 Constraint_api: v1 Constraint_acquis: >= 1.0, < 2.0 Built-in optional components: cscli_setup, datasource_appsec, datasource_cloudwatch, datasource_docker, datasource_file, datasource_http, datasource_journalctl, datasource_k8s-audit, datasource_kafka, datasource_kinesis, datasource_loki, datasource_s3, datasource_syslog, datasource_victorialogs, datasource_wineventlog, db_mysql, db_postgres, db_sqlite ``` </details> ### OS version <details> ```console # On Linux: $ cat /etc/os-release PRETTY_NAME="Armbian 26.8.0-trunk.42 trixie" NAME="Debian GNU/Linux" VERSION_ID="13" VERSION="13 (trixie)" VERSION_CODENAME=trixie DEBIAN_VERSION_FULL=13.5 ID=debian HOME_URL="https://www.armbian.com" SUPPORT_URL="https://forum.armbian.com" BUG_REPORT_URL="https://www.armbian.com/bugs" ARMBIAN_PRETTY_NAME="Armbian 26.8.0-trunk.42 trixie" $ uname -a Linux orion 6.18.32-current-rockchip64 #1 SMP PREEMPT Sun May 17 15:15:37 UTC 2026 aarch64 GNU/Linux ``` </details> ### Enabled collections and parsers <details> ```console $ cscli hub list -o raw Loaded: 165 parsers, 12 postoverflows, 781 scenarios, 9 contexts, 5 appsec-configs, 213 appsec-rules, 163 collections Unmanaged items: 3 local, 0 tainted name,status,version,description,type crowdsecurity/dateparse-enrich,enabled,0.2,,parsers crowdsecurity/dovecot-logs,enabled,0.9,Parse dovecot logs,parsers crowdsecurity/geoip-enrich,enabled,0.5,"Populate event with geoloc info : as, country, coords, source range.",parsers crowdsecurity/http-logs,enabled,1.3,"Parse more Specifically HTTP logs, such as HTTP Code, HTTP path, HTTP args and if its a static ressource",parsers crowdsecurity/postfix-logs,enabled,0.9,Parse postfix logs,parsers crowdsecurity/postscreen-logs,enabled,0.3,Parse postscreen logs,parsers crowdsecurity/public-dns-allowlist,enabled,0.1,Allow events from public DNS servers,parsers crowdsecurity/sshd-logs,enabled,3.1,Parse openSSH logs,parsers crowdsecurity/sshd-success-logs,enabled,0.1,Parse successful ssh logins,parsers crowdsecurity/syslog-logs,enabled,1.0,,parsers crowdsecurity/traefik-logs,enabled,1.5,Parse Traefik access logs,parsers crowdsecurity/whitelists,enabled,0.3,Whitelist events from private ipv4 addresses,parsers lan-whitelist,"enabled,local",,,parsers stalwart/stalwart-logs,"enabled,local",,,parsers crowdsecurity/cdn-whitelist,enabled,0.5,Whitelist CDN providers,postoverflows crowdsecurity/google-special-crawlers-whitelist,enabled,0.1,"Whitelist events from Google special crawlers (e.g. Google-InspectionTool, GoogleOther)",postoverflows crowdsecurity/rdns,enabled,0.3,Lookup the DNS associated to the source IP only for overflows,postoverflows crowdsecurity/seo-bots-whitelist,enabled,0.5,Whitelist good search engine crawlers,postoverflows crowdsecurity/apache_log4j2_cve-2021-44228,enabled,0.7,Detect cve-2021-44228 exploitation attempts,scenarios crowdsecurity/CVE-2017-9841,enabled,0.2,Detect CVE-2017-9841 exploits,scenarios crowdsecurity/CVE-2019-18935,enabled,0.2,Detect Telerik CVE-2019-18935 exploitation attempts,scenarios crowdsecurity/CVE-2022-26134,enabled,0.4,Confluence - RCE (CVE-2022-26134),scenarios crowdsecurity/CVE-2022-35914,enabled,0.2,Detect CVE-2022-35914 exploits,scenarios crowdsecurity/CVE-2022-37042,enabled,0.2,Detect CVE-2022-37042 exploits,scenarios crowdsecurity/CVE-2022-40684,enabled,0.3,Detect cve-2022-40684 exploitation attempts,scenarios crowdsecurity/CVE-2022-41082,enabled,0.4,Detect CVE-2022-41082 exploits,scenarios crowdsecurity/CVE-2022-41697,enabled,0.2,Detect CVE-2022-41697 enumeration,scenarios crowdsecurity/CVE-2022-42889,enabled,0.3,Detect CVE-2022-42889 exploits (Text4Shell),scenarios crowdsecurity/CVE-2022-44877,enabled,0.3,Detect CVE-2022-44877 exploits,scenarios crowdsecurity/CVE-2022-46169,enabled,0.2,Detect CVE-2022-46169 brute forcing,scenarios crowdsecurity/CVE-2023-22515,enabled,0.1,Detect CVE-2023-22515 exploitation,scenarios crowdsecurity/CVE-2023-22518,enabled,0.3,Detect CVE-2023-22518 exploits,scenarios crowdsecurity/CVE-2023-49103,enabled,0.3,Detect owncloud CVE-2023-49103 exploitation attempts,scenarios crowdsecurity/CVE-2024-0012,enabled,0.1,Detect CVE-2024-0012 exploitation attempts,scenarios crowdsecurity/CVE-2024-38475,enabled,0.1,Detect CVE-2024-38475 exploitation attempts,scenarios crowdsecurity/CVE-2024-9474,enabled,0.1,Detect CVE-2024-9474 exploitation attempts,scenarios crowdsecurity/dovecot-spam,enabled,0.6,Detect Dovecot bruteforce,scenarios crowdsecurity/f5-big-ip-cve-2020-5902,enabled,0.3,F5 BIG-IP TMUI - RCE (CVE-2020-5902),scenarios crowdsecurity/fortinet-cve-2018-13379,enabled,0.4,Detect cve-2018-13379 exploitation attempts,scenarios crowdsecurity/grafana-cve-2021-43798,enabled,0.3,Grafana - Arbitrary File Read (CVE-2021-43798),scenarios crowdsecurity/http-admin-interface-probing,enabled,0.5,Detect generic HTTP admin interface probing,scenarios crowdsecurity/http-backdoors-attempts,enabled,0.6,Detect attempt to common backdoors,scenarios crowdsecurity/http-bad-user-agent,enabled,1.2,Detect usage of bad User Agent,scenarios crowdsecurity/http-crawl-non_statics,enabled,0.7,Detect aggressive crawl on non static resources,scenarios crowdsecurity/http-cve-2021-41773,enabled,0.3,Apache - Path Traversal (CVE-2021-41773),scenarios crowdsecurity/http-cve-2021-42013,enabled,0.3,Apache - Path Traversal (CVE-2021-42013),scenarios crowdsecurity/http-cve-probing,enabled,0.6,Detect generic HTTP cve probing,scenarios crowdsecurity/http-generic-bf,enabled,0.9,Detect generic http brute force,scenarios crowdsecurity/http-generic-test,enabled,0.2,Crowdsec Generic Test Scenario: basic HTTP trigger,scenarios crowdsecurity/http-open-proxy,enabled,0.5,Detect scan for open proxy,scenarios crowdsecurity/http-path-traversal-probing,enabled,0.4,Detect path traversal attempt,scenarios crowdsecurity/http-probing,enabled,0.4,Detect site scanning/probing from a single ip,scenarios crowdsecurity/http-sap-interface-probing,enabled,0.1,Detect generic HTTP SAP interface probing,scenarios crowdsecurity/http-sensitive-files,enabled,0.4,"Detect attempt to access to sensitive files (.log, .db ..) or folders (.git)",scenarios crowdsecurity/http-sqli-probing,enabled,0.4,A scenario that detects SQL injection probing with minimal false positives,scenarios crowdsecurity/http-technology-probing,enabled,0.1,Detect HTTP technology/vendor probing,scenarios crowdsecurity/http-wordpress-scan,enabled,0.4,Detect exploitation attempts against common WordPress endpoints,scenarios crowdsecurity/http-xss-probing,enabled,0.4,A scenario that detects XSS probing with minimal false positives,scenarios crowdsecurity/jira_cve-2021-26086,enabled,0.4,Detect Atlassian Jira CVE-2021-26086 exploitation attempts,scenarios crowdsecurity/netgear_rce,enabled,0.4,Detect Netgear RCE DGN1000/DGN220 exploitation attempts,scenarios crowdsecurity/postfix-helo-rejected,enabled,0.1,Detect HELO rejections,scenarios crowdsecurity/postfix-non-smtp-command,enabled,0.1,Detect scanning of postfix service through non-SMTP commands,scenarios crowdsecurity/postfix-relay-denied,enabled,0.1,Detect multiple open relay attempts,scenarios crowdsecurity/postfix-spam,enabled,0.4,Detect spammers,scenarios crowdsecurity/pulse-secure-sslvpn-cve-2019-11510,enabled,0.4,Detect cve-2019-11510 exploitation attempts,scenarios crowdsecurity/spring4shell_cve-2022-22965,enabled,0.3,Detect cve-2022-22965 probing,scenarios crowdsecurity/ssh-bf,enabled,0.3,Detect ssh bruteforce,scenarios crowdsecurity/ssh-cve-2024-6387,enabled,0.2,Detect exploitation attempt of CVE-2024-6387,scenarios crowdsecurity/ssh-generic-test,enabled,0.2,Crowdsec Generic Test Scenario: SSH brute force trigger,scenarios crowdsecurity/ssh-refused-conn,enabled,0.1,Detect sshd refused connections,scenarios crowdsecurity/ssh-slow-bf,enabled,0.4,Detect slow ssh bruteforce,scenarios crowdsecurity/ssh-time-based-bf,enabled,0.2,Detect time-based ssh bruteforce attempts that evade rate limiting (with false positive reduction),scenarios crowdsecurity/thinkphp-cve-2018-20062,enabled,0.7,Detect ThinkPHP CVE-2018-20062 exploitation attempts,scenarios crowdsecurity/vmware-cve-2022-22954,enabled,0.3,Detect Vmware CVE-2022-22954 exploitation attempts,scenarios crowdsecurity/vmware-vcenter-vmsa-2021-0027,enabled,0.3,Detect VMSA-2021-0027 exploitation attempts,scenarios ltsich/http-w00tw00t,enabled,0.3,detect w00tw00t,scenarios stalwart/stalwart-smtp-auth-bf,"enabled,local",,,scenarios crowdsecurity/bf_base,enabled,0.1,,contexts crowdsecurity/http_base,enabled,0.3,,contexts crowdsecurity/base-http-scenarios,enabled,1.4,http common : scanners detection,collections crowdsecurity/dovecot,enabled,0.2,dovecot support : parser and spammer detection,collections crowdsecurity/http-cve,enabled,3.0,Detect CVE exploitation in http logs,collections crowdsecurity/linux,enabled,0.4,core linux support : syslog+geoip+ssh,collections crowdsecurity/postfix,enabled,0.5,postfix support : parser and spammer detection,collections crowdsecurity/sshd,enabled,0.9,sshd support : parser and brute-force detection,collections crowdsecurity/traefik,enabled,0.2,traefik support: parser and generic http scenarios,collections crowdsecurity/whitelist-good-actors,enabled,0.4,Good actors whitelists,collections ``` </details> ### Acquisition config <details> ```console # On Linux: $ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/* cat: /etc/crowdsec/acquis.yaml: No such file or directory source: file filenames: - /srv/mailserver/logs/mail.log labels: type: syslog # # Configuration generated by "cscli setup". # Please check your non-generated configuration files to make sure # the log sources are not acquired twice. This includes # the file acquis.yaml created by crowdsec <= 1.7.0. # # cscli-checksum: a380e4b3fc5e43e510e8287a3fcbe374 filenames: - /var/log/messages - /var/log/syslog - /var/log/kern.log labels: type: syslog source: file # # Configuration generated by "cscli setup". # Please check your non-generated configuration files to make sure # the log sources are not acquired twice. This includes # the file acquis.yaml created by crowdsec <= 1.7.0. # # cscli-checksum: 70986510f568531d62e879aeeea05989 filenames: - /var/log/auth.log - /var/log/secure labels: type: syslog source: file source: file filenames: - /srv/stalwart/logs/stalwart.* labels: type: stalwart source: file filenames: - /srv/traefik/logs/access.log labels: type: traefik ``` </details> ### Config show <details> ```console $ cscli config show Global: - Configuration Folder : /etc/crowdsec - Data Folder : /var/lib/crowdsec/data - Hub Folder : /etc/crowdsec/hub - Notification Folder : /etc/crowdsec/notifications - Simulation File : /etc/crowdsec/simulation.yaml - Log Folder : /var/log - Log level : info - Log Media : file Crowdsec: - Acquisition File : /etc/crowdsec/acquis.yaml - Parsers routines : 1 - Acquisition Folder : /etc/crowdsec/acquis.d cscli: - Output : human - Hub Branch : API Client: - URL : censored - Login : censored - Credentials File : censored Local API Server: - Listen URL : censored - Listen Socket : - Profile File : /etc/crowdsec/profiles.yaml - Trusted IPs: - 127.0.0.1 - ::1 - Database: - Type : sqlite - Path : /var/lib/crowdsec/data/crowdsec.db - Flush age : 168h0m0s - Flush size : 5000 ``` </details> ### Prometheus metrics <details> ```console $ cscli metrics censored, irrelevant ``` </details> ### Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc. <details> </details>