ITADN

can mounted file reading respect monty memory limits?

#318Opensathish-t 创建于 2026-04-11
S
sathish-tcommented
Hi developers, In the latest versions of monty, the program can read and write to a file system which is very nice for me. The coding agent tells me that when a read file operation is performed, the whole file is read into memory and then if its too big, then monty says "I can't fit it within my memory limits" and throws an error. This leaves open the possibility that a large file could exhaust the host memory before monty can catch and throw an error. I am wondering if you could say examine the file metadata before reading the file and say "Monty only has so much (approximate) memory left, so refuse to read if file is larger than this". I am asking because in earlier versions of Monty, I supplied a read_file function myself, where I instituted a max file size manually. Now it would be more convenient for me to leverage Monty's existing file system access. The background for me is that I am developing agent based approaches to read large genomic datasets where sometimes text files are several GB large. If monty errors out, I can alert the agent that the file is too large and cannot be read. But if there's an OOM error, the agent may not know what went wrong. I may have gotten this wrong as I am reading the codebase using an agent, so please forgive any errors! The agent gave some suggestions for a code change, I am just pasting them here to give some more context. I don't know the codebase well enough to judge this. ```` Yep — here’s the call sketch I’d put in an issue. Current call flow ```text Python sandbox code pathlib.Path.read_text() / read_bytes() ↓ crates/monty/src/types/path.rs Path::py_call_attr() -> detects OsFunction::ReadText / ReadBytes -> returns CallResult::OsCall(...) ↓ VM FrameExit::OsCall -> RunProgress::OsCall ↓ Host binding crates/monty-python/src/monty_cls.rs or async_dispatch.rs -> handle_mount_os_call(...) -> MountTable::handle_os_call(...) ↓ FS dispatch crates/monty/src/fs/dispatch.rs -> direct::execute(...) or overlay::execute(...) ↓ Read helpers crates/monty/src/fs/common.rs -> read_text_fs(...) / read_bytes_fs(...) (today: eager fs::read(...) or clone overlay buffer) ↓ Return to VM MontyObject::String / MontyObject::Bytes -> py_to_monty / MontyObject::to_value -> Heap allocation is counted by LimitedTracker ``` What we want instead The key change is: check the memory budget before the host materializes the file contents. ```text ... same up to direct::execute / overlay::execute ... ↓ [NEW] preflight read budget check - inspect current Monty memory usage - inspect max memory limit - estimate result size from file size / overlay buffer length - add safety margin for object overhead / UTF-8 conversion - if estimate > remaining budget: return MemoryError-style failure immediately ↓ bounded/streaming read - do not use unbounded fs::read for large files - stop before host-side buffering can explode ↓ materialize MontyObject::String / Bytes ↓ resume VM ``` Where the hook should live The most natural place is the mount layer, not types/path.rs. Concretely: - crates/monty/src/fs/direct.rs - crates/monty/src/fs/overlay.rs Those are the points where we already know: - the virtual path - the resolved host path - whether the file is coming from the real FS or the overlay memory layer So the preflight would sit right before the current calls into: - read_text_fs(...) - read_bytes_fs(...) What needs to be threaded through Today MountContext only carries mount identity + write quota: - mount_virtual - mount_host - write_bytes_used - write_bytes_limit For this, it would need some kind of read-memory budget handle too. Because the memory tracker is not currently exposed as a simple object-safe API, the likely design is one of: - a small internal MemoryBudget helper threaded into MountContext, or - a generic path from OsCall<T> into mount dispatch so the tracker’s current usage can be consulted I would not expose “remaining memory” to sandbox code; this should stay internal. ````
1 条评论