ITADN

sql: SHOW PARTITIONS persistently fails with unknown schema "[id]" for tables created during concurrent schema drops

#173660Closedviragtripathi 创建于 5 天前
C-enhancementT-sql-foundationsC-autosolvetarget-release-26.4.0
V
viragtripathicommented
## Summary On v26.2.5, `SHOW PARTITIONS FROM TABLE t` fails with `ERROR: unknown schema "[<descriptor id>]"` (SQLSTATE 3F000) when table `t` was created while another session was concurrently dropping schemas. The referenced descriptor id is one of the concurrently dropped schemas, not anything related to table `t`. The failure is persistent, not transient: it survives new connections and kept failing for minutes on a single node and, on a 5 node cluster, for up to the dropped tables' `gc.ttlseconds` window (an hour in our case) on every gateway node. Tables created before the churn are unaffected. Version bracket, same harness on stock single nodes: v25.4.14 and v26.1.8 do not reproduce (40 to 60 probes each); v26.2.5 and v26.3.0 reproduce on the first probe. The regression was introduced in the v26.2 release line and is still present in the latest innovation release. Searching the tracker for `unknown schema` combined with SHOW PARTITIONS / crdb_internal.partitions found no existing report. ## Reproduction Stock single node, insecure, no zone configs, no multi-region, no enterprise features. Two concurrent SQL sessions: Session A, in a loop: ```sql CREATE SCHEMA sc_i; CREATE TABLE sc_i.t (id INT PRIMARY KEY); DROP SCHEMA sc_i CASCADE; ``` Session B, in a loop: ```sql CREATE SCHEMA pr_i; CREATE TABLE pr_i.t (id INT PRIMARY KEY); SHOW PARTITIONS FROM TABLE pr_i.t; ``` Session B hits the error within the first one or two iterations, reliably: ``` ERROR: unknown schema "[106]" SQLSTATE: 3F000 ``` A self-contained script (starts a throwaway docker node, runs both loops, prints the result) is attached; it reproduced on the first probe on every run we tried against v26.2.5 and never reproduced in 40 to 60 probes against v25.4.14. ## Observed state while broken At failure time, `system.descriptor` holds the churned table in state DROP with `unexposedParentSchemaId` pointing at the schema id in the error message, and that schema descriptor is already deleted. That orphaned shape also exists in healthy databases after any schema drop with tables awaiting GC, so it is not sufficient on its own; the concurrency window during creation of the probed table is what triggers it. Plan for the failing delegate query shows a virtual table lookup join into `table_indexes@table_indexes_descriptor_id_idx`; individually, full scans of `crdb_internal.partitions`, `tables`, `table_indexes` and `zones` (including the config SQL columns) all succeed while the delegate query fails. ## Impact Found via PeerDB's CockroachDB connector, whose mirror validation runs `SHOW PARTITIONS` per replicated table to compute the effective `gc.ttlseconds` across partition zone configs. Any workload that creates a table and inspects its partitions while unrelated schema drops are running (test suites, multi-tenant provisioning, demo harnesses) can hit this, and the affected table's SHOW PARTITIONS stays broken long after the drops finish. ## Environment - Reproduced: v26.2.5 (single node docker and a 5 node cluster behind haproxy, all gateways affected) and v26.3.0 (single node docker), aarch64 linux - Not reproduced: v25.4.14 and v26.1.8, same harness - Introduced in the v26.2 release line per the bracket above ## Self-contained repro script ```bash #!/bin/bash # Minimal reproduction: SHOW PARTITIONS fails persistently with # 'unknown schema "[<id>]"' (SQLSTATE 3F000) for tables created while another # session concurrently drops schemas. Reproduces on stock v26.2.5 single node # within one or two probe iterations; does not reproduce on v25.4.14. # # Usage: ./crdb-show-partitions-repro.sh [host:port] # Defaults to a fresh throwaway docker single node on port 26264. set -u HOSTPORT="${1:-}" if [ -z "$HOSTPORT" ]; then docker rm -f crdb-sp-repro >/dev/null 2>&1 docker run -d --name crdb-sp-repro -p 26264:26257 cockroachdb/cockroach:v26.2.5 start-single-node --insecure >/dev/null sleep 10 HOSTPORT="localhost:26264" echo "started throwaway single node cockroachdb/cockroach:v26.2.5 on $HOSTPORT" fi C() { cockroach sql --insecure --host "$HOSTPORT" -e "$1" 2>&1; } C "SELECT version()" | tail -1 # session A: schema churn, plain tables, no zone configs, no MR, nothing fancy (for i in $(seq 1 60); do C "CREATE SCHEMA sc_$i; CREATE TABLE sc_$i.t (id INT PRIMARY KEY); DROP SCHEMA sc_$i CASCADE" >/dev/null 2>&1 done) & CHURN=$! # session B: create a fresh table and immediately SHOW PARTITIONS on it for i in $(seq 1 60); do s="pr_$i" C "CREATE SCHEMA $s; CREATE TABLE $s.t (id INT PRIMARY KEY)" >/dev/null 2>&1 err=$(C "SHOW PARTITIONS FROM TABLE $s.t" | grep -oE 'unknown schema "\[[0-9]+\]"' | head -1) if [ -n "$err" ]; then kill $CHURN 2>/dev/null echo "REPRODUCED at probe $i: SHOW PARTITIONS FROM TABLE $s.t -> ERROR: $err" echo "the referenced id is one of session A's dropped schemas" echo "note: the error persists for this table across new connections;" echo " rerun 'SHOW PARTITIONS FROM TABLE $s.t' minutes later to confirm" exit 0 fi done wait $CHURN 2>/dev/null echo "no reproduction in 60 probes" exit 1 ``` Jira issue: CRDB-66970 Epic CRDB-65516
关闭于 5 天前 3 条评论