KV-cache / long-context: smallest canonical repro boundary + metric (7-day receipts eval)
### Feature request
Request: add an official “smallest long-context / KV-cache perf boundary” recipe for TGI.
What I mean by “boundary”:
- a single runnable command/config (model, context length, batch/concurrency, backend),
- the metric to report (tokens/s and p99 latency at minimum; OOM threshold optional),
- a short note on the expected bottleneck (KV-cache memory traffic) so users don’t mis-measure.
Why this matters:
KV-cache / long-context discussions often stall without a canonical minimal repro. A maintained boundary makes perf work comparable, and lets contributors return receipts instead of opinions.
Reference: vLLM boundary request already posted here:
https://github.com/vllm-project/vllm/issues/32038
### Motivation
I’m frustrated that long-context / KV-cache perf threads are hard to reproduce across machines because every report uses a different model/context/concurrency/backend.
A canonical minimal boundary would:
- make regressions obvious,
- make improvements measurable,
- reduce triage time for maintainers.
### Your contribution
If maintainers confirm the canonical boundary (command/config + acceptance metric), I can run it and return within 7 days:
- receipts-backed before/after on the same boundary (p95/p99/p99.9 deltas),
- optional energy/cost proxy,
- filename-only SHA256 index for internal forwarding.
I’m not asking anyone to believe a claim; I’m asking for the smallest boundary to measure.
0 条评论