ITADN

remove partially failed samples from output

#340Opendmalzl 创建于 2025-03-19
enhancement
D
dmalzlcommented
### Description of feature Hi and thanks for providing this nice pipeline it really saves me a lot of time and headache when downloading large amounts of published data. However, recently I found some hiccups which left me with a bit of wondering if one could maybe include this in the pipeline. So here is the problem: When downloading files with `sratools` it sometime seems to happen that the datastream fetched by `prefetch` is corrupt but `prefetch` just continues as if nothing happened and the process completes as usual. Unfortunately, `fasterq-dump` later complains and fails with "SRA ID not found" ("or another one was "qualities empty" or something). Anyway, I guess this could easily be solved by just retrying another time or using the FTP or Aspera for download of these particular files and does not really concern much. However, when using SRA experiment IDs (SRX) for download this can lead to a situation where the sample is only partially downloaded i.e. if an SRX contains multiple SRRs and one of them fails I am left with an incomplete data download, which requires me to manually check the failed processes and see if I got some finished ones for the same SRX. Not only is it additional work to parse the `.nextflow.log` file but it is also cumbersome for downstream processing (in my case via `rnaseq`) if one is not aware of this behaviour. Possible solution: Connected to #270, I would suggest to somehow check for completeness based on the retrieved retrieved metadata i.e. simple check if number of SRRs is consistent with the number listed in the ENA table and simply fail if not. This way partial download can happen without polluting the results directory.
0 条评论