bug: FTS on a JSON subfield — path-scoped inverted index is silently inert, and non-triple queries panic

## Summary FTS over a string nested inside a JSON column *is* possible and gives accurate, field-scoped answers — but only through a direct `INVERTED` index queried with an undocumented `field,type,value` triple syntax. The two things a user would naturally try both fail: 1. **A JSON-path inverted index is accepted, listed, and completely inert.** `IndexConfig(index_type="json", parameters={"target_index_type": "inverted", "path": "$.title"})` builds without error and shows up in `list_indices()`, but the planner never uses it — the plan is `FlatMatchQuery`, byte-for-byte identical to having no index at all. Queries fall back to a brute-force scan of the raw JSON *text*, so they match JSON key names and values from other fields, i.e. they ignore the `path` the index was declared on. 2. **Any non-triple query against a whole-column `INVERTED` index panics.** `flatten_triplet(query_text, ...).unwrap()` at `rust/lance-index/src/scalar/inverted/tokenizer/document_tokenizer.rs:147` unwraps a `Result` whose error is driven purely by user input, so the ordinary FTS query `"brown"` panics a background thread and surfaces as `RuntimeError: Task was aborted`. ## Repro pylance 10.0.0, Linux x86_64. ```python import os, shutil, tempfile, json, logging import pyarrow as pa, lance from lance.indices import IndexConfig from lance.query import MatchQuery logging.disable(logging.WARNING) DOCS = [{"title": "quick brown fox", "author": "alice"}, {"title": "lazy dog sleeps", "author": "bob"}, {"title": "brown bear roars", "author": "carol"}, {"title": "fox and hound", "author": "alice"}] docs = [json.dumps(d) for d in DOCS] jf = pa.field("doc", pa.string(), metadata={b"ARROW:extension:name": b"arrow.json"}) schema = pa.schema([pa.field("id", pa.int32()), jf]) def mk(cfg=None, name="fts"): uri = os.path.join(tempfile.mkdtemp(), "f.lance"); shutil.rmtree(uri, ignore_errors=True) ds = lance.write_dataset(pa.table({"id": pa.array(range(4), pa.int32()), "doc": pa.array(docs, pa.string())}, schema=schema), uri) if cfg is not None: ds.create_scalar_index("doc", cfg, name=name) return lance.dataset(uri) def search(ds, q): try: return sorted(ds.scanner(columns=["id"], full_text_query=MatchQuery(q, column="doc") ).to_table().column("id").to_pylist()) except Exception as e: return f"ERR {type(e).__name__}: {str(e)[:55]}" def plan(ds, q="brown"): return next((l.strip() for l in ds.scanner(columns=["id"], full_text_query=MatchQuery(q, column="doc")).explain_plan(True).splitlines() if "MatchQuery" in l), "?") NONE = mk() PATH = mk(IndexConfig(index_type="json", parameters={"target_index_type": "inverted", "path": "$.title"}), "fts_title") print("Defect 1: JSON-path inverted index (path=$.title)") print(" created:", [(i["name"], i["type"]) for i in PATH.list_indices()]) for term, want in [("brown", [0,2]), ("fox", [0,3]), ("alice", []), ("title", []), ("author", [])]: print(f" {term:<8} {str(search(NONE, term)):<14} {str(search(PATH, term)):<16} {want}") print(" plan, no index :", plan(NONE)) print(" plan, path index:", plan(PATH)) WHOLE = mk("INVERTED", "fts_all") print("\nDefect 2: direct INVERTED on the JSON column") print(" plan:", plan(WHOLE)) for q in ["brown", "title,string,brown", "title,str,brown"]: print(f" {q!r:<24} -> {search(WHOLE, q)}") for q, want in [("title,str,brown", [0,2]), ("title,str,alice", []), ("author,str,alice", [0,3])]: got = search(WHOLE, q) print(f" {q!r:<24} -> {got} want {want} {got == want}") ``` ## Actual ``` Defect 1: JSON-path inverted index (path=$.title) created: [('fts_title', 'Json')] term no index with path index want (for $.title) brown [0, 2] [0, 2] [0, 2] fox [0, 3] [0, 3] [0, 3] alice [0, 3] [0, 3] [] title [0, 1, 2, 3] [0, 1, 2, 3] [] author [0, 1, 2, 3] [0, 1, 2, 3] [] plan, no index : FlatMatchQuery: column=doc, query=brown plan, path index: FlatMatchQuery: column=doc, query=brown Defect 2: direct INVERTED on the JSON column plan: MatchQuery: column=doc, query=[brown] 'brown' -> ERR ArrowInvalid: External error: RuntimeError: Task was aborted 'title,string,brown' -> ERR ArrowInvalid: External error: RuntimeError: Task was aborted 'title,str,brown' -> [0, 2] 'title,str,alice' -> [] want [] True 'author,str,alice' -> [0, 3] want [0, 3] True ``` ### Defect 1 detail `alice` appears only in `$.author`, yet matches rows 0 and 3. `title` and `author` are JSON *key names*, not content, yet each matches all four rows. Every column is identical to the no-index run, and both plans are `FlatMatchQuery`, so the index contributes nothing — the answers come from a flat scan over the serialized JSON text. So a user who builds this index gets no acceleration and no path scoping, with no indication that either is missing. Note also that `list_indices()` reports the index type as `Json` rather than `Inverted`, which matches the existing `TODO` on `JsonIndex::index_type()`. ### Defect 2 detail Both panics come from the same `.unwrap()`: ```rust fn token_stream_for_search<'a>(&'a mut self, query_text: &'a str) -> BoxTokenStream<'a> { let tokens = flatten_triplet(query_text, &mut self.tokenizer).unwrap(); // :147 ``` - `"brown"` → `InvalidInput { source: "Invalid triple format: brown" }` (`:180`) - `"title,string,brown"` → `InvalidInput { source: "Invalid triple type: string" }` (`:214`) The second is worth calling out: the type token is `str`, not `string`, and getting it wrong panics rather than erroring. Accepted tokens are `str`, `number`, `bool`, `null`; field names are dotted paths built by `flatten_json` (so `meta.tag,str,nature` works for a nested object). ## Expected - A JSON-path inverted index should either be used by the planner and scoped to its `path`, or be rejected at creation time with a clear "not supported" error. Silently creating an index that is never consulted, and answering with whole-document semantics under a `$.title` declaration, is the worst outcome. - A malformed FTS query should return an `InvalidInput` error to the caller, not panic a background thread. The error type is already there; it just needs to propagate instead of being unwrapped. ## Notes - Verified against a control: a plain `string` column with `INVERTED` plans as `MatchQuery: column=title` and returns correct results, so the `FlatMatchQuery` fallback is specific to the JSON path index. - The triple syntax appears to be the intended query interface for the JSON inverted index, but I could not find it documented anywhere. Given #7445 is already reworking flattened JSON sub-doc indexing, it may be worth settling the user-facing query surface there — ideally so that `MatchQuery("brown", column="doc.title")` or similar works instead of requiring callers to hand-assemble `title,str,brown`. - Related: #4749 (add full text json index), #7445 (flattened JSON sub-doc indexing), #8806 (JSON path scalar index returns incorrect results for `json_extract` filters).