services/sqlite: stat() reads the entire value to compute content_length (get_length() exists but is unused)
bugreleases-note/fix
### What happens
In the sqlite service, `stat()` computes `content_length` by fetching the **entire value**:
https://github.com/apache/opendal/blob/5d84f2b34261d38f6c59254fc9e561f5941fb6ee/core/services/sqlite/src/backend.rs#L225-L235
`self.core.get(&p)` runs `SELECT value FROM table WHERE key = ?`, materializes the whole blob into memory, and then only uses `bs.len()`.
The efficient query already exists right next to it — `SqliteCore::get_length` does `SELECT LENGTH(value) ...`, which SQLite answers from the record header without loading the blob:
https://github.com/apache/opendal/blob/5d84f2b34261d38f6c59254fc9e561f5941fb6ee/core/services/sqlite/src/core.rs#L63-L83
but `stat()` does not use it. (The range-read path was already fixed to push down via `SUBSTR` in #5701 / `get_range`; `stat` is the remaining whole-value read.)
### Why it's a footgun
Any workflow that stats before reading pays a full-value read per stat. In our application (a glTF asset server using the sqlite service as blob storage), an HTTP range-download handler performed one `op.stat()` (integrity/length check) + one `reader.read(range)` per request. With a 66 MB stored value and ~3 MiB ranges, every request silently re-read the entire 66 MB out of SQLite just to learn its length — invisible in code review because `stat()` reads like a cheap metadata call, which it is on fs/s3.
### Measured impact
Methodology: 40 sequential HTTP range requests (3 MiB each, ascending offsets) against the same 66 MB value in a single-file sqlite backend, warm OS page cache, release build, timed at the HTTP client. Before/after removing the per-request `stat` (length taken from our own DB row instead; the first `read(range)` doubles as the existence check):
| | median / request | p90 | 40-request total |
|---|---|---|---|
| with per-request `stat` | 99 ms | 110 ms | 4.36 s |
| without | 30 ms | 33 ms | 1.49 s |
~3.3x per request. (Caveat: our patch also stopped rebuilding the `Operator` per request, so a small share of the delta is pool reuse — but the dominant cost was the 66 MB `SELECT` per stat, ~2.6 GB of redundant reads across those 40 requests.)
### Suggested fix
Have `stat()` call the existing `get_length()` instead of `get()`:
```rust
let length = self.core.get_length(&p).await?;
match length {
Some(len) => Ok(RpStat::new(
Metadata::new(EntryMode::from_path(&p)).with_content_length(len as u64),
)),
None => { /* existing directory-prefix fallback */ }
}
```
The `None` / directory-prefix fallback is unaffected. Alternatively (or additionally), size and other metadata could be persisted as columns alongside the value at write time — that generalizes to richer metadata (`last_modified`, etag) — but for `content_length` alone the `LENGTH()` one-liner seems sufficient and schema-compatible.
Happy to send a PR if the `get_length` approach sounds right.
### Disclosure
This investigation, the measurements, and this issue text were produced by an AI assistant (Claude, by Anthropic) working under my direction on our downstream project; I reviewed the findings before filing.
1 条评论