ITADN

extend "topology change coordinator fiber got error std::runtime_error" for disrupt_terminate_and_replace_node nemesis

#13382Closedtimtimb0t 创建于 2026-01-28
T
timtimb0tcommented
[Argus](https://argus.scylladb.com/test/b188a1f4-47cc-4fb3-b6b4-7c6428910c5f/runs?additionalRuns[]=62e9ade2-6235-4cd4-a8d9-d0b0eabf88ae) Scylla version: `2026.1.0~rc0-20260125.f94296e0ae43` with build-id `9680213fda6f301234c43da8ca27e47953987cd8` Now **ignore_raft_topology_cmd_failing** only wraps **_terminate_cluster_node** and stays active for just 30s after that call finishes. But then nemesis does sleep(300) and continues with the replace process. During this window SCT reports non filtered errors: ``` 2026-01-26 00:27:09.876 <2026-01-26 00:27:09.176>: (DatabaseLogEvent Severity.ERROR) period_type=one-time event_id=96190725-8934-4f4a-b2ab-7df4ff65152d: type=RUNTIME_ERROR regex=std::runtime_error line_number=30415 node=alternator-3h-2026-1-db-node-62e9ade2-1 2026-01-26T00:27:09.176 alternator-3h-2026-1-db-node-62e9ade2-1 !ERR | scylla[6193] [shard 0: gms] raft_topology - topology change coordinator fiber got error std::runtime_error (raft topology: exec_global_command(barrier) failed with seastar::rpc::closed_error (got error from node 56ee97ae-2307-492b-8e78-ca1a24741faa/10.12.8.89: connection is closed)) ``` Kernel Version: `6.14.0-1018-aws` <details> <summary> Extra information </summary> ## Installation details Cluster size: 6 nodes (i4i.4xlarge) Scylla Nodes used in this run: - alternator-3h-2026-1-db-node-62e9ade2-1 (54.91.48.70 | 10.12.10.144) (shards: -1) - alternator-3h-2026-1-db-node-62e9ade2-2 (34.228.23.243 | 10.12.11.42) (shards: 14) - alternator-3h-2026-1-db-node-62e9ade2-3 (100.27.213.2 | 10.12.8.89) (shards: 14) - alternator-3h-2026-1-db-node-62e9ade2-4 (3.91.67.218 | 10.12.9.113) (shards: 14) - alternator-3h-2026-1-db-node-62e9ade2-5 (54.211.97.89 | 10.12.9.72) (shards: 14) - alternator-3h-2026-1-db-node-62e9ade2-6 (54.235.3.142 | 10.12.8.239) (shards: 14) - alternator-3h-2026-1-db-node-62e9ade2-7 (3.87.235.240 | 10.12.10.184) (shards: 14) - alternator-3h-2026-1-db-node-62e9ade2-8 (54.167.51.111 | 10.12.10.126) (shards: 14) OS / Image: `ami-041ecb6271ecc1499` (aws: N/A) Test: `longevity-alternator-3h-test` Test id: `62e9ade2-6235-4cd4-a8d9-d0b0eabf88ae` Test name: `scylla-2026.1/alternator/longevity-alternator-3h-test` Test method: `longevity_test.LongevityTest.test_custom_time` Test config file(s): - [longevity-alternator-3h.yaml](https://github.com/scylladb/scylla-cluster-tests/blob/d26ba86342b75cfc21ce38dbe73db685e979dab4/test-cases/longevity/longevity-alternator-3h.yaml) ## Logs: - **[alternator-3h-2026-1-db-node-62e9ade2-3](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/62e9ade2-6235-4cd4-a8d9-d0b0eabf88ae/log/alternator-3h-2026-1-db-node-62e9ade2-3/download)** - **[alternator-3h-2026-1-db-node-62e9ade2-8](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/62e9ade2-6235-4cd4-a8d9-d0b0eabf88ae/log/alternator-3h-2026-1-db-node-62e9ade2-8/download)** - **[db-cluster-62e9ade2.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/62e9ade2-6235-4cd4-a8d9-d0b0eabf88ae/log/db-cluster-62e9ade2.tar.zst/download)** - **[schema-logs-62e9ade2.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/62e9ade2-6235-4cd4-a8d9-d0b0eabf88ae/log/schema-logs-62e9ade2.tar.zst/download)** - **[failure-statistics-62e9ade2.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/62e9ade2-6235-4cd4-a8d9-d0b0eabf88ae/log/failure-statistics-62e9ade2.tar.zst/download)** - **[sct-runner-events-62e9ade2.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/62e9ade2-6235-4cd4-a8d9-d0b0eabf88ae/log/sct-runner-events-62e9ade2.tar.zst/download)** - **[sct-62e9ade2.log.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/62e9ade2-6235-4cd4-a8d9-d0b0eabf88ae/log/sct-62e9ade2.log.tar.zst/download)** - **[loader-set-62e9ade2.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/62e9ade2-6235-4cd4-a8d9-d0b0eabf88ae/log/loader-set-62e9ade2.tar.zst/download)** - **[monitor-set-62e9ade2.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/62e9ade2-6235-4cd4-a8d9-d0b0eabf88ae/log/monitor-set-62e9ade2.tar.zst/download)** - **[parallel-timelines-report-62e9ade2.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/62e9ade2-6235-4cd4-a8d9-d0b0eabf88ae/log/parallel-timelines-report-62e9ade2.tar.zst/download)** - **[builder-62e9ade2.log.tar.gz](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/62e9ade2-6235-4cd4-a8d9-d0b0eabf88ae/log/builder-62e9ade2.log.tar.gz/download)** [Jenkins job URL](https://jenkins.scylladb.com/job/scylla-2026.1/job/alternator/job/longevity-alternator-3h-test/2/) </details>
关闭于 2026-01-29 1 条评论