Exponential memory use when chaining seqgen
I know this isn't a useful command, but to showcase the issue:
```console
$ time -v mlr seqgen -f one --stop 5000 then seqgen -f two --stop 5000 | wc -l
Command being timed: "mlr seqgen -f one --stop 5000 then seqgen -f two --stop 5000"
User time (seconds): 5.20
System time (seconds): 0.51
Percent of CPU this job got: 139%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:04.10
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 14361336
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 111376
Voluntary context switches: 1380
Involuntary context switches: 2394
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
5000
```
That's ~14 GB max RSS. If you add another `seqgen --stop 5000` it should run out of memory on any computer available today.
It seems to be because the first `seqgen` outputs a single batch with all records, and then the second one outputs its own batch for every record of the input. I think the final slice that the output writer gets looks likes this: `[1, 2, 3, ..., 5000, end-of-stream, 1, 2, ..., 5000, end-of-stream, 1, ...]`. It will have 5000 end of stream markers, and it only output until the first marker - it's the correct behavior AFAICT, just wasteful.
关闭于 2026-06-19 1 条评论