Model Selection

Select Model {{ id }}

Request level analysis

Request throughput: {{ allRequestProperties[selectedIndex]?.throughput }} requests/sec
Request average Latency: {{ allRequestProperties[selectedIndex]?.averageLatencyMs }} ms

All the time related columns displayed in this table are in milliseconds.

Request details table, percentile is calculated based on column:

{{ column }}

Batch level analysis

Batch throughput: {{ allBatchProperties[selectedIndex]?.throughput }} batches/sec
Batch average Latency: {{ allBatchProperties[selectedIndex]?.averageLatencyMs }} ms
This model is controlled by the following batching parameters:
  • Number of batching threads: {{ allBatchProperties[selectedIndex]?.batchingParamNumBatchThreads }}
  • Batch timeout in microseconds: {{ allBatchProperties[selectedIndex]?.batchingParamBatchTimeoutMicros }}
  • Maximum size of a batch: {{ allBatchProperties[selectedIndex]?.batchingParamMaxBatchSize }}
  • Maximum number of enqueued batches: {{ allBatchProperties[selectedIndex]?.batchingParamMaxEnqueuedBatches }}
  • Sizes that are allowed to form a batch: {{ allBatchProperties[selectedIndex]?.batchingParamAllowedBatchSizes }}

Batch details table, percentile is calculated based on column:

{{ column }}

Tensor transfer analysis