AWS mirrored SRA file download
<!--
# nf-core/fetchngs pull request
Many thanks for contributing to nf-core/fetchngs!
Please fill in the appropriate checklist below (delete whatever is not relevant).
These are the most common things requested on pull requests (PRs).
Remember that PRs should be made against the dev branch, unless you're preparing a pipeline release.
Learn more about contributing: [CONTRIBUTING.md](https://github.com/nf-core/fetchngs/tree/master/.github/CONTRIBUTING.md)
-->
Per Issue #354, I've taken a stab at adapting my direct AWS download method for S3-mirrored SRA files. For ease of integration, I've so far just specified environment options with aws cli installed, but perhaps it would be more appropriate to try and instead use existing nextflow AWS integrations?
I've tested this on a couple datasets of various sizes I'm working with right now and it seems to be working quite reliably.
I did some very basic benchmarking of performance differences.
<img width="451" height="431" alt="Screenshot 2025-08-15 at 1 31 49 AM" src="https://github.com/user-attachments/assets/02a83196-d50e-4c75-a012-63e30121cc4a" />
<img width="349" height="386" alt="Screenshot 2025-08-15 at 1 31 58 AM" src="https://github.com/user-attachments/assets/e2da1220-3255-48d0-a43a-ff77b0e05cbe" />
On a study of 28 samples with around 10 Gbp sequences, the AWS method was around 5 times faster for the download step than the SRA Prefetch approach. The bulk of the time was still taken up by unpacking, though.
## PR checklist
- [x] This comment contains a description of changes (with reason).
Changes:
- addition of aws cli download method for mirrored SRA files
- [x] If you've fixed a bug or added code that should be tested, add tests!
Tests added (plus nf-test snapshots) for the revised workflow, the new subworkflow, and the added module
- [x] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/nf-core/fetchngs/tree/master/.github/CONTRIBUTING.md)
- [ ] If necessary, also make a PR on the nf-core/fetchngs _branch_ on the [nf-core/test-datasets](https://github.com/nf-core/test-datasets) repository.
Uses existing test data and approaches from sratools downloads.
- [ ] Make sure your code lints (`nf-core lint`).
There are linting errors in the dev branch, but no new linting errors introduced.
- [x] Ensure the test suite passes (`nextflow run . -profile test,docker --outdir <OUTDIR>`).
Tests passing.
- [x] Check for unexpected warnings in debug mode (`nextflow run . -profile debug,test,docker --outdir <OUTDIR>`).
- [x] Usage Documentation in `docs/usage.md` is updated.
- [ ] Output Documentation in `docs/output.md` is updated.
- [ ] `CHANGELOG.md` is updated.
- [ ] `README.md` is updated (including new tool citations and authors/contributors).
合并状态:未合并 0 条评论