ITADN

KV-cache / long-context: smallest canonical repro boundary + metric (7-day receipts eval)

#3351OpenStanByriukov02 创建于 2026-01-09
S
StanByriukov02commented
### Feature request Request: add an official “smallest long-context / KV-cache perf boundary” recipe for TGI. What I mean by “boundary”: - a single runnable command/config (model, context length, batch/concurrency, backend), - the metric to report (tokens/s and p99 latency at minimum; OOM threshold optional), - a short note on the expected bottleneck (KV-cache memory traffic) so users don’t mis-measure. Why this matters: KV-cache / long-context discussions often stall without a canonical minimal repro. A maintained boundary makes perf work comparable, and lets contributors return receipts instead of opinions. Reference: vLLM boundary request already posted here: https://github.com/vllm-project/vllm/issues/32038 ### Motivation I’m frustrated that long-context / KV-cache perf threads are hard to reproduce across machines because every report uses a different model/context/concurrency/backend. A canonical minimal boundary would: - make regressions obvious, - make improvements measurable, - reduce triage time for maintainers. ### Your contribution If maintainers confirm the canonical boundary (command/config + acceptance metric), I can run it and return within 7 days: - receipts-backed before/after on the same boundary (p95/p99/p99.9 deltas), - optional energy/cost proxy, - filename-only SHA256 index for internal forwarding. I’m not asking anyone to believe a claim; I’m asking for the smallest boundary to measure.
0 条评论