Make Group0 membership read strict and retry only expected CQL exceptions
[Argus](https://argus.scylladb.com/test/0dff5ae0-9a6c-48c3-8561-d95a09795905/runs?additionalRuns[]=ae55afa4-98cc-434a-8cfb-5d7738aba978)
Scylla version: `2026.1.0~rc0-20260125.f94296e0ae43` with build-id `9680213fda6f301234c43da8ca27e47953987cd8`
Nemesis failure:
```
Traceback (most recent call last):
File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 6360, in wrapper
result = method(*args, **kwargs)
File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 5907, in disrupt_bootstrap_streaming_error
bootstrapabortmanager.clean_unbootstrapped_node()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/home/ubuntu/scylla-cluster-tests/sdcm/sct_events/group_common_events.py", line 634, in inner_func
return func(*args, **kwargs)
File "/home/ubuntu/scylla-cluster-tests/sdcm/utils/raft/common.py", line 171, in clean_unbootstrapped_node
assert self.verification_node.raft.is_cluster_topology_consistent(), (
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
AssertionError: Group0, Token Ring and number of node in cluster are differs. Check logs
```
SCT code:
```
def is_cluster_topology_consistent(self) -> bool:
group0_ids = [member["host_id"] for member in self.get_group0_members()]
LOGGER.debug("Group0 member ids %s", group0_ids)
token_ring_ids = [member["host_id"] for member in self._node.get_token_ring_members()]
LOGGER.debug("Token ring member ids: %s", token_ring_ids)
diff = set(group0_ids) - set(token_ring_ids) or set(token_ring_ids) - set(group0_ids)
LOGGER.debug("Difference between group0 and token ring: %s", diff)
num_of_nodes = len(self._node.parent_cluster.nodes)
LOGGER.debug("Number of nodes in sct cluster %s", num_of_nodes)
non_voters_ids = self.search_inconsistent_host_ids()
return not diff and not non_voters_ids and len(group0_ids) == len(token_ring_ids) == num_of_nodes
```
```
def get_group0_members(self) -> list[Group0Member]:
LOGGER.debug("Get group0 members")
group0_members = []
try:
with self._node.parent_cluster.cql_connection_patient_exclusive(node=self._node) as session:
raft_group0_id = self.get_group0_id(session)
assert raft_group0_id, "Group0 id was not found"
rows = session.execute(
f"select server_id, can_vote from system.raft_state \
where group_id = {raft_group0_id} and disposition = 'CURRENT'"
).all()
for row in rows:
group0_members.append(Group0Member(host_id=str(row.server_id), voter=row.can_vote))
except Exception as exc: # noqa: BLE001
err_msg = f"Get group0 members failed with error: {exc}"
LOGGER.error(err_msg)
LOGGER.debug("Group0 members: %s", group0_members)
return group0_members
```
in this particular case `except Exception as exc` masked the underlying driver error:
```
< t:2026-01-26 02:17:37,459 f:cluster.py l:4103 c:cassandra.cluster p:WARNING > cassandra.connection.ConnectionShutdown: [Errno 9] Bad file descriptor
```
and returned empty list that led to nemesis failure. I don’t think a catch-all exception handler here is the right approach. If this driver issue hadn’t already been known, we could have easily missed it. A blind retry isn’t a good solution either. From my point of view, we should only retry on a small set of expected exceptions (maybe, timeouts, NoHostAvailable etc.). Anything else should be logged as-is and stop current nemesis
Kernel Version: `6.14.0-1018-aws`
<details>
<summary>
Extra information
</summary>
## Installation details
Cluster size: 6 nodes (i4i.4xlarge)
Scylla Nodes used in this run:
- longevity-100gb-4h-2026-1-db-node-ae55afa4-1 (18.214.100.191 | 10.12.8.254) (shards: 14)
- longevity-100gb-4h-2026-1-db-node-ae55afa4-2 (98.93.132.101 | 10.12.11.251) (shards: 14)
- longevity-100gb-4h-2026-1-db-node-ae55afa4-3 (100.31.91.66 | 10.12.9.119) (shards: 14)
- longevity-100gb-4h-2026-1-db-node-ae55afa4-4 (54.196.137.234 | 10.12.8.236) (shards: 14)
- longevity-100gb-4h-2026-1-db-node-ae55afa4-5 (54.90.78.63 | 10.12.10.173) (shards: 14)
- longevity-100gb-4h-2026-1-db-node-ae55afa4-6 (13.220.180.103 | 10.12.10.163) (shards: 14)
- longevity-100gb-4h-2026-1-db-node-ae55afa4-7 (34.224.86.8 | 10.12.8.121) (shards: 14)
- longevity-100gb-4h-2026-1-db-node-ae55afa4-8 (54.242.91.143 | 10.12.11.54) (shards: -1)
OS / Image: `ami-041ecb6271ecc1499` (aws: N/A)
Test: `longevity-100gb-4h-test`
Test id: `ae55afa4-98cc-434a-8cfb-5d7738aba978`
Test name: `scylla-2026.1/longevity/longevity-100gb-4h-test`
Test method: `longevity_test.LongevityTest.test_custom_time`
Test config file(s):
- [longevity-100gb-4h-cql-stress.yaml](https://github.com/scylladb/scylla-cluster-tests/blob/d26ba86342b75cfc21ce38dbe73db685e979dab4/test-cases/longevity/longevity-100gb-4h-cql-stress.yaml)
## Logs:
- **[longevity-100gb-4h-2026-1-db-node-ae55afa4-5](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/ae55afa4-98cc-434a-8cfb-5d7738aba978/log/longevity-100gb-4h-2026-1-db-node-ae55afa4-5/download)**
- **[longevity-100gb-4h-2026-1-db-node-ae55afa4-8](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/ae55afa4-98cc-434a-8cfb-5d7738aba978/log/longevity-100gb-4h-2026-1-db-node-ae55afa4-8/download)**
- **[db-cluster-ae55afa4.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/ae55afa4-98cc-434a-8cfb-5d7738aba978/log/db-cluster-ae55afa4.tar.zst/download)**
- **[schema-logs-ae55afa4.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/ae55afa4-98cc-434a-8cfb-5d7738aba978/log/schema-logs-ae55afa4.tar.zst/download)**
- **[failure-statistics-ae55afa4.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/ae55afa4-98cc-434a-8cfb-5d7738aba978/log/failure-statistics-ae55afa4.tar.zst/download)**
- **[sct-runner-events-ae55afa4.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/ae55afa4-98cc-434a-8cfb-5d7738aba978/log/sct-runner-events-ae55afa4.tar.zst/download)**
- **[sct-ae55afa4.log.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/ae55afa4-98cc-434a-8cfb-5d7738aba978/log/sct-ae55afa4.log.tar.zst/download)**
- **[loader-set-ae55afa4.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/ae55afa4-98cc-434a-8cfb-5d7738aba978/log/loader-set-ae55afa4.tar.zst/download)**
- **[monitor-set-ae55afa4.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/ae55afa4-98cc-434a-8cfb-5d7738aba978/log/monitor-set-ae55afa4.tar.zst/download)**
- **[parallel-timelines-report-ae55afa4.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/ae55afa4-98cc-434a-8cfb-5d7738aba978/log/parallel-timelines-report-ae55afa4.tar.zst/download)**
- **[ssl-conf-ae55afa4.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/ae55afa4-98cc-434a-8cfb-5d7738aba978/log/ssl-conf-ae55afa4.tar.zst/download)**
- **[builder-ae55afa4.log.tar.gz](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/ae55afa4-98cc-434a-8cfb-5d7738aba978/log/builder-ae55afa4.log.tar.gz/download)**
[Jenkins job URL](https://jenkins.scylladb.com/job/scylla-2026.1/job/longevity/job/longevity-100gb-4h-test/2/)
</details>
1 条评论