Handling Concurrent Client Requests with Faster Whisper

I am currently using the Faster Whisper (medium) model running on a GPU to handle requests from multiple clients. However, when multiple clients send concurrent requests, the inference latency increases significantly. Could you please suggest a solution to scale this setup so that it can handle a higher number of concurrent client requests while maintaining manageable latency?