triton concurrency settings

Hi @whoisj, are there any triton inference server settings to address the following based on @nvpohanh question? > This does not look like a TensorRT issue, but a Triton inference server issue. It looks like when TensorRT is running computations on GPU, Triton inference server just waits for the current request to complete without taking a new request and schedule/enqueue the new request. Could you check if Triton inference server has some configs which allows Triton to establish more than one TensorRT execution context so that Triton inference server can schedule the next request even before the current request has completed? Thanks! see: https://github.com/NVIDIA/TensorRT/issues/4680#issuecomment-3851620224 in my benchmark the client sends a request with 8 prompts inside (client side batch) + the concurrency of that client is set to 1. my current config.pbtxt settings: ``` name: classification platform: "tensorrt_plan" backend: "tensorrt" max_batch_size: 12 input: [ { name: "input_ids" data_type: TYPE_INT64 dims: [-1] }, { name: "attention_mask" data_type: TYPE_INT64 dims: [-1] } ] output: [ { name: "logits" data_type: TYPE_FP32 dims: [10] } ] instance_group [ { cout: 1 kind: KIND_GPU } ] dynamic_batching { } ``` additional experiments which didnt help: - experiment 1: i increased instance_group count from 1 to 3 - experiment 2: i also set in dynamic_batching preferred_batch_size: [4, 8] max_queue_delay_microseconds: 100