rattler.package_streaming efficiency
### Checklist
- [x] I added a descriptive title
- [x] I searched open requests and couldn't find a duplicate
### What is the idea?
Hello, I was looking at the new `rattler.package_streaming` module and it looks great.
While testing it, I noticed that it differs from `conda-package-streaming` by not allowing to retrieve multiple files from the same stream.
In conda-package-streaming, we can do stuff like
```python
import contextlib
from conda_package_streaming.url import conda_reader_for_url
from conda_package_streaming.package_streaming import stream_conda_component
filename, conda = conda_reader_for_url(url)
data = {}
with contextlib.closing(conda):
for tar, member in stream_conda_component(filename, conda, component="pkg"):
if member.name.endswith(".so"):
data[member.name] = tar.extractfile(member).read()
```
which allows us to read multiple files from within with a single connection and I believe fewer HTTP calls (though I have no verified this).
With rattler, I see the `fetch_raw_package_file_from_url` function and from what I see it works. But from what I see, it seems a little bit slower (which is hard to explain). And it is definitely slower if I try the equivalent, which look like:
```python
import json
import rattler
import rattler.package_streaming
client = rattler.Client.default_client()
paths = json.loads(await rattler.package_streaming.fetch_raw_package_file_from_url(client, url, "info/paths.json"))
data = {}
for entry in paths:
if entry["_path"].endswith(".so"):
data[entry["_path"]] = await rattler.package_streaming.fetch_raw_package_file_from_url(client, url, entry["_path"])
```
Should rattler provide a similar API to conda-package-streaming?
### Why is this needed?
I think it could be more efficient to have an API like conda-package-streaming provides. But I have not fully verified this yet.
### What should happen?
_No response_
### Additional Context
_No response_
2 条评论