ITADN

Exponential memory use when chaining seqgen

#2072Closedfarnoy 创建于 2026-06-09
F
farnoycommented
I know this isn't a useful command, but to showcase the issue: ```console $ time -v mlr seqgen -f one --stop 5000 then seqgen -f two --stop 5000 | wc -l Command being timed: "mlr seqgen -f one --stop 5000 then seqgen -f two --stop 5000" User time (seconds): 5.20 System time (seconds): 0.51 Percent of CPU this job got: 139% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:04.10 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 14361336 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 111376 Voluntary context switches: 1380 Involuntary context switches: 2394 Swaps: 0 File system inputs: 0 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0 5000 ``` That's ~14 GB max RSS. If you add another `seqgen --stop 5000` it should run out of memory on any computer available today. It seems to be because the first `seqgen` outputs a single batch with all records, and then the second one outputs its own batch for every record of the input. I think the final slice that the output writer gets looks likes this: `[1, 2, 3, ..., 5000, end-of-stream, 1, 2, ..., 5000, end-of-stream, 1, ...]`. It will have 5000 end of stream markers, and it only output until the first marker - it's the correct behavior AFAICT, just wasteful.
关闭于 2026-06-19 1 条评论