ITADN

Make Group0 membership read strict and retry only expected CQL exceptions

#13371Opentimtimb0t 创建于 2026-01-27
T
timtimb0tcommented
[Argus](https://argus.scylladb.com/test/0dff5ae0-9a6c-48c3-8561-d95a09795905/runs?additionalRuns[]=ae55afa4-98cc-434a-8cfb-5d7738aba978) Scylla version: `2026.1.0~rc0-20260125.f94296e0ae43` with build-id `9680213fda6f301234c43da8ca27e47953987cd8` Nemesis failure: ``` Traceback (most recent call last): File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 6360, in wrapper result = method(*args, **kwargs) File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 5907, in disrupt_bootstrap_streaming_error bootstrapabortmanager.clean_unbootstrapped_node() ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/ubuntu/scylla-cluster-tests/sdcm/sct_events/group_common_events.py", line 634, in inner_func return func(*args, **kwargs) File "/home/ubuntu/scylla-cluster-tests/sdcm/utils/raft/common.py", line 171, in clean_unbootstrapped_node assert self.verification_node.raft.is_cluster_topology_consistent(), ( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ AssertionError: Group0, Token Ring and number of node in cluster are differs. Check logs ``` SCT code: ``` def is_cluster_topology_consistent(self) -> bool: group0_ids = [member["host_id"] for member in self.get_group0_members()] LOGGER.debug("Group0 member ids %s", group0_ids) token_ring_ids = [member["host_id"] for member in self._node.get_token_ring_members()] LOGGER.debug("Token ring member ids: %s", token_ring_ids) diff = set(group0_ids) - set(token_ring_ids) or set(token_ring_ids) - set(group0_ids) LOGGER.debug("Difference between group0 and token ring: %s", diff) num_of_nodes = len(self._node.parent_cluster.nodes) LOGGER.debug("Number of nodes in sct cluster %s", num_of_nodes) non_voters_ids = self.search_inconsistent_host_ids() return not diff and not non_voters_ids and len(group0_ids) == len(token_ring_ids) == num_of_nodes ``` ``` def get_group0_members(self) -> list[Group0Member]: LOGGER.debug("Get group0 members") group0_members = [] try: with self._node.parent_cluster.cql_connection_patient_exclusive(node=self._node) as session: raft_group0_id = self.get_group0_id(session) assert raft_group0_id, "Group0 id was not found" rows = session.execute( f"select server_id, can_vote from system.raft_state \ where group_id = {raft_group0_id} and disposition = 'CURRENT'" ).all() for row in rows: group0_members.append(Group0Member(host_id=str(row.server_id), voter=row.can_vote)) except Exception as exc: # noqa: BLE001 err_msg = f"Get group0 members failed with error: {exc}" LOGGER.error(err_msg) LOGGER.debug("Group0 members: %s", group0_members) return group0_members ``` in this particular case `except Exception as exc` masked the underlying driver error: ``` < t:2026-01-26 02:17:37,459 f:cluster.py l:4103 c:cassandra.cluster p:WARNING > cassandra.connection.ConnectionShutdown: [Errno 9] Bad file descriptor ``` and returned empty list that led to nemesis failure. I don’t think a catch-all exception handler here is the right approach. If this driver issue hadn’t already been known, we could have easily missed it. A blind retry isn’t a good solution either. From my point of view, we should only retry on a small set of expected exceptions (maybe, timeouts, NoHostAvailable etc.). Anything else should be logged as-is and stop current nemesis Kernel Version: `6.14.0-1018-aws` <details> <summary> Extra information </summary> ## Installation details Cluster size: 6 nodes (i4i.4xlarge) Scylla Nodes used in this run: - longevity-100gb-4h-2026-1-db-node-ae55afa4-1 (18.214.100.191 | 10.12.8.254) (shards: 14) - longevity-100gb-4h-2026-1-db-node-ae55afa4-2 (98.93.132.101 | 10.12.11.251) (shards: 14) - longevity-100gb-4h-2026-1-db-node-ae55afa4-3 (100.31.91.66 | 10.12.9.119) (shards: 14) - longevity-100gb-4h-2026-1-db-node-ae55afa4-4 (54.196.137.234 | 10.12.8.236) (shards: 14) - longevity-100gb-4h-2026-1-db-node-ae55afa4-5 (54.90.78.63 | 10.12.10.173) (shards: 14) - longevity-100gb-4h-2026-1-db-node-ae55afa4-6 (13.220.180.103 | 10.12.10.163) (shards: 14) - longevity-100gb-4h-2026-1-db-node-ae55afa4-7 (34.224.86.8 | 10.12.8.121) (shards: 14) - longevity-100gb-4h-2026-1-db-node-ae55afa4-8 (54.242.91.143 | 10.12.11.54) (shards: -1) OS / Image: `ami-041ecb6271ecc1499` (aws: N/A) Test: `longevity-100gb-4h-test` Test id: `ae55afa4-98cc-434a-8cfb-5d7738aba978` Test name: `scylla-2026.1/longevity/longevity-100gb-4h-test` Test method: `longevity_test.LongevityTest.test_custom_time` Test config file(s): - [longevity-100gb-4h-cql-stress.yaml](https://github.com/scylladb/scylla-cluster-tests/blob/d26ba86342b75cfc21ce38dbe73db685e979dab4/test-cases/longevity/longevity-100gb-4h-cql-stress.yaml) ## Logs: - **[longevity-100gb-4h-2026-1-db-node-ae55afa4-5](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/ae55afa4-98cc-434a-8cfb-5d7738aba978/log/longevity-100gb-4h-2026-1-db-node-ae55afa4-5/download)** - **[longevity-100gb-4h-2026-1-db-node-ae55afa4-8](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/ae55afa4-98cc-434a-8cfb-5d7738aba978/log/longevity-100gb-4h-2026-1-db-node-ae55afa4-8/download)** - **[db-cluster-ae55afa4.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/ae55afa4-98cc-434a-8cfb-5d7738aba978/log/db-cluster-ae55afa4.tar.zst/download)** - **[schema-logs-ae55afa4.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/ae55afa4-98cc-434a-8cfb-5d7738aba978/log/schema-logs-ae55afa4.tar.zst/download)** - **[failure-statistics-ae55afa4.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/ae55afa4-98cc-434a-8cfb-5d7738aba978/log/failure-statistics-ae55afa4.tar.zst/download)** - **[sct-runner-events-ae55afa4.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/ae55afa4-98cc-434a-8cfb-5d7738aba978/log/sct-runner-events-ae55afa4.tar.zst/download)** - **[sct-ae55afa4.log.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/ae55afa4-98cc-434a-8cfb-5d7738aba978/log/sct-ae55afa4.log.tar.zst/download)** - **[loader-set-ae55afa4.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/ae55afa4-98cc-434a-8cfb-5d7738aba978/log/loader-set-ae55afa4.tar.zst/download)** - **[monitor-set-ae55afa4.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/ae55afa4-98cc-434a-8cfb-5d7738aba978/log/monitor-set-ae55afa4.tar.zst/download)** - **[parallel-timelines-report-ae55afa4.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/ae55afa4-98cc-434a-8cfb-5d7738aba978/log/parallel-timelines-report-ae55afa4.tar.zst/download)** - **[ssl-conf-ae55afa4.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/ae55afa4-98cc-434a-8cfb-5d7738aba978/log/ssl-conf-ae55afa4.tar.zst/download)** - **[builder-ae55afa4.log.tar.gz](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/ae55afa4-98cc-434a-8cfb-5d7738aba978/log/builder-ae55afa4.log.tar.gz/download)** [Jenkins job URL](https://jenkins.scylladb.com/job/scylla-2026.1/job/longevity/job/longevity-100gb-4h-test/2/) </details>
1 条评论