disrupt_no_corrupt_repair finished with "Exit code: 1" due to "Can't find a column family standard1 in keyspace drop_table_during_repair_ks_0"
[Argus](https://argus.scylladb.com/test/6e39841d-5ed5-4f83-aa70-f51949559234/runs?additionalRuns[]=60838ffe-e0da-4e03-ab38-f893dfa49fa3)
Scylla version: `2026.2.0~dev-20260125.55422593a7f7` with build-id `b115844afd5e535d95cdd0fd30d691dd3d753f9e`
SCT reported error:
```
Traceback (most recent call last):
File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 6360, in wrapper
result = method(*args, **kwargs)
File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 1731, in disrupt_no_corrupt_repair
thread.result()
~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.14/concurrent/futures/_base.py", line 443, in result
return self.__get_result()
~~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.14/concurrent/futures/_base.py", line 395, in __get_result
raise self._exception
File "/usr/local/lib/python3.14/concurrent/futures/thread.py", line 86, in run
result = ctx.run(self.task)
File "/usr/local/lib/python3.14/concurrent/futures/thread.py", line 73, in run
return fn(*args, **kwargs)
File "/home/ubuntu/scylla-cluster-tests/sdcm/utils/decorators.py", line 238, in wrapped
res = func(*args, **kwargs)
File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 2074, in run_repair_nodetool
target_node.run_nodetool(sub_cmd="cluster repair", publish_event=publish_event)
~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 3071, in run_nodetool
result = runner(cmd, timeout=timeout, ignore_status=ignore_status, verbose=verbose, retry=retry)
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/remote_base.py", line 746, in run
result = _run()
File "/home/ubuntu/scylla-cluster-tests/sdcm/utils/decorators.py", line 79, in inner
return func(*args, **kwargs)
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/remote_base.py", line 737, in _run
return self._run_execute(cmd, timeout, ignore_status, verbose, new_session, watchers)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/remote_base.py", line 670, in _run_execute
result = connection.run(**command_kwargs)
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/libssh2_client/__init__.py", line 690, in run
return self._complete_run(channel, exception, timeout_reached, timeout, result, warn, stdout, stderr)
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/libssh2_client/__init__.py", line 732, in _complete_run
raise UnexpectedExit(result)
sdcm.remote.libssh2_client.exceptions.UnexpectedExit: Encountered a bad command exit code!
Command: '/usr/bin/nodetool cluster repair '
Exit code: 1
Stdout:
[2026-01-26 09:04:46,275] Starting repair with task_id=0f8d1cf0-fa96-11f0-8ca4-a7b38e2f13b5 keyspace=drop_table_during_repair_ks_4 table=standard1
[2026-01-26 09:04:56,593] Repair with task_id=0f8d1cf0-fa96-11f0-8ca4-a7b38e2f13b5 finished
[2026-01-26 09:04:56,887] Starting repair with task_id=15c7cf20-fa96-11f0-8ca4-a7b38e2f13b5 keyspace=drop_table_during_repair_ks_7 table=standard1
[2026-01-26 09:05:05,315] Repair with task_id=15c7cf20-fa96-11f0-8ca4-a7b38e2f13b5 finished
[2026-01-26 09:05:05,620] Starting repair with task_id=1afaae40-fa96-11f0-8ca4-a7b38e2f13b5 keyspace=drop_table_during_repair_ks_2 table=standard1
[2026-01-26 09:05:17,485] Repair with task_id=1afaae40-fa96-11f0-8ca4-a7b38e2f13b5 finished
[2026-01-26 09:05:17,614] Starting repair with task_id=223b85d0-fa96-11f0-8ca4-a7b38e2f13b5 keyspace=drop_table_during_repair_ks_3 table=standard1
[2026-01-26 09:05:29,471] Repair with task_id=223b85d0-fa96-11f0-8ca4-a7b38e2f13b5 finished
[2026-01-26 09:05:29,632] Starting repair with task_id=29609800-fa96-11f0-8ca4-a7b38e2f13b5 keyspace=keyspace1 table=standard1
[2026-01-26 09:06:07,732] Repair with task_id=29609800-fa96-11f0-8ca4-a7b38e2f13b5 finished
Stderr:
```
Scylla reported:
```
< t:2026-01-26 09:06:08,094 f:base.py l:276 c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.184>: [2026-01-26 09:04:33,716] Starting repair with task_id=07ffd310-fa96-11f0-8ca4-a7b38e2f13b5 keyspace=drop_table_during_repair_ks_5 table=standard1
< t:2026-01-26 09:06:08,094 f:base.py l:276 c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.184>: [2026-01-26 09:04:46,138] Repair with task_id=07ffd310-fa96-11f0-8ca4-a7b38e2f13b5 finished
< t:2026-01-26 09:06:08,094 f:base.py l:276 c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.184>: [2026-01-26 09:04:46,143] ERROR: Repair request for keyspace=drop_table_during_repair_ks_0 table=standard1 failed with error executing POST request to http://localhost:10000/storage_service/tablets/repair with parameters {"tokens": ["all"], "ks": ["drop_table_during_repair_ks_0"], "table": ["standard1"]}: remote replied with status code 400 Bad Request:
< t:2026-01-26 09:06:08,094 f:base.py l:276 c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.184>: Can't find a column family standard1 in keyspace drop_table_during_repair_ks_0
< t:2026-01-26 09:06:08,094 f:db_log_reader.py l:152 c:sdcm.db_log_reader p:DEBUG > 2026-01-26T09:04:40.415 longevity-10gb-3h-master-db-node-60838ffe-0-5 !INFO | scylla[7599] [shard 2:strm] repair - repair[6466ffdf-40ba-4bd2-a145-9dec65f9d989]: Started to repair 1 out of 1 tables in keyspace=drop_table_during_repair_ks_5, table=standard1, table_id=ac80a060-fa94-11f0-9ba3-06c78dee1a5c, repair_reason=repair
< t:2026-01-26 09:06:08,094 f:base.py l:276 c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.184>: [2026-01-26 09:04:46,275] Starting repair with task_id=0f8d1cf0-fa96-11f0-8ca4-a7b38e2f13b5 keyspace=drop_table_during_repair_ks_4 table=standard1
< t:2026-01-26 09:06:08,095 f:base.py l:276 c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.184>: [2026-01-26 09:04:56,593] Repair with task_id=0f8d1cf0-fa96-11f0-8ca4-a7b38e2f13b5 finished
< t:2026-01-26 09:06:08,095 f:base.py l:276 c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.184>: [2026-01-26 09:04:56,887] Starting repair with task_id=15c7cf20-fa96-11f0-8ca4-a7b38e2f13b5 keyspace=drop_table_during_repair_ks_7 table=standard1
< t:2026-01-26 09:06:08,095 f:base.py l:276 c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.184>: [2026-01-26 09:05:05,315] Repair with task_id=15c7cf20-fa96-11f0-8ca4-a7b38e2f13b5 finished
< t:2026-01-26 09:06:08,095 f:base.py l:276 c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.184>: [2026-01-26 09:05:05,620] Starting repair with task_id=1afaae40-fa96-11f0-8ca4-a7b38e2f13b5 keyspace=drop_table_during_repair_ks_2 table=standard1
< t:2026-01-26 09:06:08,095 f:base.py l:276 c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.184>: [2026-01-26 09:05:17,485] Repair with task_id=1afaae40-fa96-11f0-8ca4-a7b38e2f13b5 finished
< t:2026-01-26 09:06:08,095 f:base.py l:276 c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.184>: [2026-01-26 09:05:17,614] Starting repair with task_id=223b85d0-fa96-11f0-8ca4-a7b38e2f13b5 keyspace=drop_table_during_repair_ks_3 table=standard1
< t:2026-01-26 09:06:08,095 f:base.py l:276 c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.184>: [2026-01-26 09:05:29,471] Repair with task_id=223b85d0-fa96-11f0-8ca4-a7b38e2f13b5 finished
< t:2026-01-26 09:06:08,095 f:base.py l:276 c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.184>: [2026-01-26 09:05:29,632] Starting repair with task_id=29609800-fa96-11f0-8ca4-a7b38e2f13b5 keyspace=keyspace1 table=standard1
< t:2026-01-26 09:06:08,095 f:base.py l:276 c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.184>: [2026-01-26 09:06:07,732] Repair with task_id=29609800-fa96-11f0-8ca4-a7b38e2f13b5 finished
< t:2026-01-26 09:06:08,095 f:base.py l:178 c:RemoteLibSSH2CmdRunner p:ERROR > <10.142.0.184>: Error executing command: "/usr/bin/nodetool cluster repair "; Exit status: 1
< t:2026-01-26 09:06:08,095 f:base.py l:182 c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.184>: STDOUT: 7b38e2f13b5 finished
```
But seems like it's possible expected behavior, nemesis code:
```
self.log.debug("Start repair target_node in background")
with ThreadPoolExecutor(max_workers=1, thread_name_prefix="NodeToolRepairThread") as thread_pool:
thread = thread_pool.submit(partial(self.run_repair_nodetool, nodes=[self.target_node]))
try:
# drop test tables one by one during repair
for i in range(10):
time.sleep(random.randint(0, 300))
with self.cluster.cql_connection_patient(self.target_node, connect_timeout=600) as session:
self.actions_log.info(f"Dropping table drop_table_during_repair_ks_{i}.standard1")
session.execute(
SimpleStatement(f"DROP TABLE drop_table_during_repair_ks_{i}.standard1"), timeout=300
)
finally:
thread.result()
```
test intentionally drops the tables
Kernel Version: `6.14.0-1021-gcp`
<details>
<summary>
Extra information
</summary>
## Installation details
Cluster size: 6 nodes (n2-highmem-16)
Scylla Nodes used in this run:
- longevity-10gb-3h-master-db-node-60838ffe-0-1 (35.231.129.186 | 10.142.0.170) (shards: 14)
- longevity-10gb-3h-master-db-node-60838ffe-0-2 (34.148.71.27 | 10.142.0.176) (shards: 14)
- longevity-10gb-3h-master-db-node-60838ffe-0-3 (35.196.122.157 | 10.142.0.181) (shards: 14)
- longevity-10gb-3h-master-db-node-60838ffe-0-4 (35.196.63.133 | 10.142.0.162) (shards: 14)
- longevity-10gb-3h-master-db-node-60838ffe-0-5 (34.148.150.161 | 10.142.0.143) (shards: 14)
- longevity-10gb-3h-master-db-node-60838ffe-0-6 (35.231.12.123 | 10.142.0.184) (shards: 14)
- longevity-10gb-3h-master-db-node-60838ffe-0-7 (35.231.128.197 | 10.142.0.7) (shards: 14)
- longevity-10gb-3h-master-db-node-60838ffe-0-8 (35.231.215.75 | 10.142.0.15) (shards: 14)
OS / Image: `https://www.googleapis.com/compute/v1/projects/scylla-images/global/images/3885115548033007141` (gce: N/A)
Test: `longevity-10gb-3h-gce-test`
Test id: `60838ffe-e0da-4e03-ab38-f893dfa49fa3`
Test name: `scylla-master/longevity/longevity-10gb-3h-gce-test`
Test method: `longevity_test.LongevityTest.test_custom_time`
Test config file(s):
- [longevity-10gb-3h.yaml](https://github.com/scylladb/scylla-cluster-tests/blob/b6d2009d9aa6520a80c62e130bd58c56bc1091fd/test-cases/longevity/longevity-10gb-3h.yaml)
## Logs:
- **[longevity-10gb-3h-master-db-node-60838ffe-0-4](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/60838ffe-e0da-4e03-ab38-f893dfa49fa3/log/longevity-10gb-3h-master-db-node-60838ffe-0-4/download)**
- **[longevity-10gb-3h-master-db-node-60838ffe-0-1](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/60838ffe-e0da-4e03-ab38-f893dfa49fa3/log/longevity-10gb-3h-master-db-node-60838ffe-0-1/download)**
- **[db-cluster-60838ffe.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/60838ffe-e0da-4e03-ab38-f893dfa49fa3/log/db-cluster-60838ffe.tar.zst/download)**
- **[schema-logs-60838ffe.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/60838ffe-e0da-4e03-ab38-f893dfa49fa3/log/schema-logs-60838ffe.tar.zst/download)**
- **[failure-statistics-60838ffe.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/60838ffe-e0da-4e03-ab38-f893dfa49fa3/log/failure-statistics-60838ffe.tar.zst/download)**
- **[sct-runner-events-60838ffe.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/60838ffe-e0da-4e03-ab38-f893dfa49fa3/log/sct-runner-events-60838ffe.tar.zst/download)**
- **[sct-60838ffe.log.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/60838ffe-e0da-4e03-ab38-f893dfa49fa3/log/sct-60838ffe.log.tar.zst/download)**
- **[loader-set-60838ffe.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/60838ffe-e0da-4e03-ab38-f893dfa49fa3/log/loader-set-60838ffe.tar.zst/download)**
- **[monitor-set-60838ffe.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/60838ffe-e0da-4e03-ab38-f893dfa49fa3/log/monitor-set-60838ffe.tar.zst/download)**
- **[parallel-timelines-report-60838ffe.tar.zst](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/60838ffe-e0da-4e03-ab38-f893dfa49fa3/log/parallel-timelines-report-60838ffe.tar.zst/download)**
- **[builder-60838ffe.log.tar.gz](https://argus.scylladb.com/api/v1/tests/scylla-cluster-tests/60838ffe-e0da-4e03-ab38-f893dfa49fa3/log/builder-60838ffe.log.tar.gz/download)**
[Jenkins job URL](https://jenkins.scylladb.com/job/scylla-master/job/longevity/job/longevity-10gb-3h-gce-test/1036/)
</details>
关闭于 2026-02-05 11 条评论