deploy local avian-flu with "nextstrain build . --configfile config/gisaid.yaml -f deploy_all" failed
bug
### Current Behavior
When I want to deploy local avian-flu with `nextstrain build . --configfile config/gisaid.yaml -f deploy_all`, but some files seemed can't be download, the log as fellow:
### How to reproduce
Steps to reproduce the current behavior:
```console
$ nextstrain build . --configfile config/gisaid.yaml -f deploy_all
Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 48
Rules claiming more threads will be scaled down.
Job counts:
count jobs
2 add_h5_clade
48 align
48 ancestral
6 cleavage_site
1 deploy_all
1 download_s3_metadata
8 download_s3_sequences
48 export
48 filter
4 filter_metadata_by_subtype
32 filter_sequences_by_subtype
48 refine
48 rename_auspice_datasets
48 traits
48 translate
48 tree
486
[Wed Feb 26 04:57:42 2025]
rule download_s3_sequences:
output: data/gisaid/sequences_pb2.fasta
jobid: 571
wildcards: input_name=gisaid, segment=pb2
aws s3 cp s3://nextstrain-data-private/files/workflows/avian-flu/pb2/sequences.fasta.zst - | zstd -d > data/gisaid/sequences_pb2.fasta
[Wed Feb 26 04:57:42 2025]
rule download_s3_metadata:
output: data/gisaid/metadata.tsv
jobid: 572
wildcards: input_name=gisaid
aws s3 cp s3://nextstrain-data-private/files/workflows/avian-flu/metadata.tsv.zst - | zstd -d > data/gisaid/metadata.tsv
[Wed Feb 26 04:57:42 2025]
rule download_s3_sequences:
output: data/gisaid/sequences_na.fasta
jobid: 578
wildcards: input_name=gisaid, segment=na
aws s3 cp s3://nextstrain-data-private/files/workflows/avian-flu/na/sequences.fasta.zst - | zstd -d > data/gisaid/sequences_na.fasta
[Wed Feb 26 04:57:42 2025]
rule download_s3_sequences:
output: data/gisaid/sequences_ha.fasta
jobid: 576
wildcards: input_name=gisaid, segment=ha
aws s3 cp s3://nextstrain-data-private/files/workflows/avian-flu/ha/sequences.fasta.zst - | zstd -d > data/gisaid/sequences_ha.fasta
[Wed Feb 26 04:57:42 2025]
rule download_s3_sequences:
output: data/gisaid/sequences_ns.fasta
jobid: 580
wildcards: input_name=gisaid, segment=ns
aws s3 cp s3://nextstrain-data-private/files/workflows/avian-flu/ns/sequences.fasta.zst - | zstd -d > data/gisaid/sequences_ns.fasta
[Wed Feb 26 04:57:42 2025]
rule download_s3_sequences:
output: data/gisaid/sequences_pa.fasta
jobid: 575
wildcards: input_name=gisaid, segment=pa
aws s3 cp s3://nextstrain-data-private/files/workflows/avian-flu/pa/sequences.fasta.zst - | zstd -d > data/gisaid/sequences_pa.fasta
[Wed Feb 26 04:57:42 2025]
rule download_s3_sequences:
output: data/gisaid/sequences_pb1.fasta
jobid: 574
wildcards: input_name=gisaid, segment=pb1
aws s3 cp s3://nextstrain-data-private/files/workflows/avian-flu/pb1/sequences.fasta.zst - | zstd -d > data/gisaid/sequences_pb1.fasta
[Wed Feb 26 04:57:42 2025]
rule download_s3_sequences:
output: data/gisaid/sequences_np.fasta
jobid: 577
wildcards: input_name=gisaid, segment=np
aws s3 cp s3://nextstrain-data-private/files/workflows/avian-flu/np/sequences.fasta.zst - | zstd -d > data/gisaid/sequences_np.fasta
[Wed Feb 26 04:57:42 2025]
rule download_s3_sequences:
output: data/gisaid/sequences_mp.fasta
jobid: 579
wildcards: input_name=gisaid, segment=mp
aws s3 cp s3://nextstrain-data-private/files/workflows/avian-flu/mp/sequences.fasta.zst - | zstd -d > data/gisaid/sequences_mp.fasta
download failed: s3://nextstrain-data-private/files/workflows/avian-flu/pa/sequences.fasta.zst to - Unable to locate credentials
download failed: s3://nextstrain-data-private/files/workflows/avian-flu/mp/sequences.fasta.zst to - Unable to locate credentials
download failed: s3://nextstrain-data-private/files/workflows/avian-flu/ha/sequences.fasta.zst to - Unable to locate credentials
download failed: s3://nextstrain-data-private/files/workflows/avian-flu/ns/sequences.fasta.zst to - Unable to locate credentials
download failed: s3://nextstrain-data-private/files/workflows/avian-flu/pb2/sequences.fasta.zst to - Unable to locate credentials
download failed: s3://nextstrain-data-private/files/workflows/avian-flu/na/sequences.fasta.zst to - Unable to locate credentials
download failed: s3://nextstrain-data-private/files/workflows/avian-flu/pb1/sequences.fasta.zst to - Unable to locate credentials
download failed: s3://nextstrain-data-private/files/workflows/avian-flu/np/sequences.fasta.zst to - Unable to locate credentials
download failed: s3://nextstrain-data-private/files/workflows/avian-flu/metadata.tsv.zst to - Unable to locate credentials
zstd: /*stdin*\: unexpected end of file
zstd: /*stdin*\: unexpected end of file
zstd: /*stdin*\: unexpected end of file
[Wed Feb 26 04:57:50 2025]
Error in rule download_s3_sequences:
jobid: 580
[Wed Feb 26 04:57:50 2025]
[Wed Feb 26 04:57:50 2025]
output: data/gisaid/sequences_ns.fasta
Error in rule download_s3_sequences:
jobid: 577
shell:
aws s3 cp s3://nextstrain-data-private/files/workflows/avian-flu/ns/sequences.fasta.zst - | zstd -d > data/gisaid/sequences_ns.fasta
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
Removing output files of failed job download_s3_sequences since they might be corrupted:
data/gisaid/sequences_ns.fasta
output: data/gisaid/sequences_np.fasta
shell:
aws s3 cp s3://nextstrain-data-private/files/workflows/avian-flu/np/sequences.fasta.zst - | zstd -d > data/gisaid/sequences_np.fasta
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
Removing output files of failed job download_s3_sequences since they might be corrupted:
data/gisaid/sequences_np.fasta
Error in rule download_s3_sequences:
jobid: 571
output: data/gisaid/sequences_pb2.fasta
shell:
aws s3 cp s3://nextstrain-data-private/files/workflows/avian-flu/pb2/sequences.fasta.zst - | zstd -d > data/gisaid/sequences_pb2.fasta
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
Removing output files of failed job download_s3_sequences since they might be corrupted:
data/gisaid/sequences_pb2.fasta
zstd: /*stdin*\: unexpected end of file
[Wed Feb 26 04:57:50 2025]
Error in rule download_s3_sequences:
jobid: 575
output: data/gisaid/sequences_pa.fasta
shell:
aws s3 cp s3://nextstrain-data-private/files/workflows/avian-flu/pa/sequences.fasta.zst - | zstd -d > data/gisaid/sequences_pa.fasta
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
Removing output files of failed job download_s3_sequences since they might be corrupted:
data/gisaid/sequences_pa.fasta
zstd: /*stdin*\: unexpected end of file
zstd: /*stdin*\: unexpected end of file
[Wed Feb 26 04:57:50 2025]
[Wed Feb 26 04:57:50 2025]
Error in rule download_s3_sequences:
Error in rule download_s3_sequences:
jobid: 576
jobid: 579
output: data/gisaid/sequences_ha.fasta
shell:
aws s3 cp s3://nextstrain-data-private/files/workflows/avian-flu/ha/sequences.fasta.zst - | zstd -d > data/gisaid/sequences_ha.fasta
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
Removing output files of failed job download_s3_sequences since they might be corrupted:
data/gisaid/sequences_ha.fasta
output: data/gisaid/sequences_mp.fasta
shell:
aws s3 cp s3://nextstrain-data-private/files/workflows/avian-flu/mp/sequences.fasta.zst - | zstd -d > data/gisaid/sequences_mp.fasta
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
Removing output files of failed job download_s3_sequences since they might be corrupted:
data/gisaid/sequences_mp.fasta
zstd: /*stdin*\: unexpected end of file
zstd: /*stdin*\: unexpected end of file
zstd: /*stdin*\: unexpected end of file
[Wed Feb 26 04:57:50 2025]
[Wed Feb 26 04:57:50 2025]
Error in rule download_s3_sequences:
Error in rule download_s3_metadata:
jobid: 578
jobid: 572
output: data/gisaid/sequences_na.fasta
[Wed Feb 26 04:57:50 2025]
Error in rule download_s3_sequences:
jobid: 574
output: data/gisaid/sequences_pb1.fasta
shell:
aws s3 cp s3://nextstrain-data-private/files/workflows/avian-flu/pb1/sequences.fasta.zst - | zstd -d > data/gisaid/sequences_pb1.fasta
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
output: data/gisaid/metadata.tsv
shell:
aws s3 cp s3://nextstrain-data-private/files/workflows/avian-flu/metadata.tsv.zst - | zstd -d > data/gisaid/metadata.tsv
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
Removing output files of failed job download_s3_sequences since they might be corrupted:
data/gisaid/sequences_pb1.fasta
shell:
aws s3 cp s3://nextstrain-data-private/files/workflows/avian-flu/na/sequences.fasta.zst - | zstd -d > data/gisaid/sequences_na.fasta
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
Removing output files of failed job download_s3_sequences since they might be corrupted:
data/gisaid/sequences_na.fasta
Removing output files of failed job download_s3_metadata since they might be corrupted:
data/gisaid/metadata.tsv
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /nextstrain/build/.snakemake/log/2025-02-26T045741.789585.snakemake.log
```
### Possible solution
(optional)
### Your environment: if browsing Nextstrain online
- Operating system:
- Browser:
### Your environment: if running Nextstrain locally
- Operating system: ubuntu 16.04
- Browser: Chrome
- Version (e.g. `auspice 2.59.1`):
关闭于 2025-02-27 1 条评论