ITADN

Lots of exemplars seems to be dropped

#1254Openjonatan-ivanov 创建于 2026-04-09
J
jonatan-ivanovcommented
I'm testing OTLP Exemplars using [otel-lgtm](https://hub.docker.com/r/grafana/otel-lgtm) and I noticed that only a few exemplars are kept in Grafana even if I send much more per time-series: - 1 exemplar per minute per time series for Sums (I send ~100 exemplars per minute for a single Sum) - 2 exemplars per minute per time series for Histograms (I send ~50 exemplars per minute for a single Histogram) <img width="1920" height="530" alt="Image" src="https://github.com/user-attachments/assets/1c8506f3-4e41-4e74-93ca-556e9198e09b" /> I tried pushing with higher frequency: it does not matter if I send 100 exemplars once per minute (100/min) or 100 exemplars 6 times per minute (600/min, 100 per every 10s). I think I should see more than one exemplar per minute for sums and 2 exemplars per minute for histograms even if I only push every 60s, shouldn't I? If I query Prometheus inside of the lgtm stack: ``` http://localhost:9090/api/v1/query_exemplars?query=my_counter_total ``` I see only ~5 exemplars per minute per time series which I might be able to explain if the Prometheus export is happening ~5 times per minute since Prometheus can only have one exemplar per time series. But ~5 is still more than the single one that Grafana shows. If I don't use the otel-lgtm image but the grafana image and set it up with a Prometheus datasource, I can see more exemplars.
0 条评论