Event handler leaks session streams on consumption error
bugevent-handler
Expected behavior:
If event handler session consumption encounters an error, it should cancel context and cause the session stream to be torn down. The Auth should not spike in memory usage (and cause OOM) during regular event handler operation due to continuous failed session consumption.
Current behavior:
The event handler can cause stream leakage in the Auth Service and spike Auth memory usage due to not canceling stream contexts correctly.
The event handler currently opens session streams with the long-lived app context. If an error occurs in session consumption (`consumeSession`), the early return doesn't cancel the stream context and the stream isn't released on the Auth.
```go
// integrations/event-handler/session_events_job.go
// consumeSession ingests session
func (j *SessionEventsJob) consumeSession(ctx context.Context, s session) (bool, error) {
// ...
chEvt, chErr := j.app.client.StreamUnstructuredSessionEvents(ctx, s.ID, s.Index)
```
Event handler retry behavior will continue opening new streams, so old streams that are not released will spike memory usage in the Auth. This happens regardless of setting the concurrency option `FDFWD_CONCURRENCY` (eg. 5), and the open concurrent streams will continue accumulating until the limit of 1000 (`GRPCMaxConcurrentStreams = 1000` in `lib/defaults/defaults.go`).
Bug details:
- Teleport version: 18.7.4
- Teleport event handler version: 18.7.4
0 条评论