BUG: FileNotFoundError during resubmit
After successfully running a dpgen2 job, I modified the input json to envolve more stages in explore. I want to reuse the previous jobs, but it raised FileNotFound error. I also tried no modifications on dpgen.json it is the same. It cannot resubmit the job.
Through the `-l` command, we can see the previous jobs finished.
```shell
dpgen2 resubmit dpgen_wsl.json job-name-lryu0 -l
0 : init--scheduler
1 : init--id
2 : iter-000000--prep-train
3 -> 3 : iter-000000--run-train-0000 -> iter-000000--run-train-0000
4 : iter-000000--prep-lmp
5 -> 5 : iter-000000--run-lmp-000000 -> iter-000000--run-lmp-000000
6 : iter-000000--select-confs
7 : iter-000000--prep-fp
8 -> 8 : iter-000000--run-fp-000000 -> iter-000000--run-fp-000000
9 : iter-000000--collect-data
10 : iter-000000--scheduler
11 : iter-000000--id
12 : iter-000001--prep-train
13 -> 13 : iter-000001--run-train-0000 -> iter-000001--run-train-0000
14 : iter-000001--prep-lmp
15 -> 15 : iter-000001--run-lmp-000000 -> iter-000001--run-lmp-000000
16 : iter-000001--select-confs
17 : iter-000001--prep-fp
18 -> 18 : iter-000001--run-fp-000000 -> iter-000001--run-fp-000000
19 : iter-000001--collect-data
20 : iter-000001--scheduler
21 : iter-000001--id
```
But resubmit raise errors:
```shell
dpgen2 resubmit dpgen_wsl.json job-name-lryu0 -u 0-21
Workflow is running locally (ID: job-name-smbtm)
CP2KDATA| step (key: init--scheduler) skipped
CP2KDATA| step (key: init--id) skipped
CP2KDATA| step (key: iter-000000--prep-train) skipped
Traceback (most recent call last):
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1666, in run
self.exec(scope, parameters, None, context, order)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1858, in exec
self.exec_pod(scope, parameters, item, order)
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 2189, in exec_pod
os.makedirs(save_path, exist_ok=True)
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen os>", line 228, in makedirs
FileNotFoundError: [Errno 2] No such file or directory: '/home/outis/Research/dp_train/dpgen2/test/dpgen2_job/job-name-smbtm/.././job-name-smbtm/iter-000000--run-train-group/model'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1949, in exec_steps
steps.run(scope.workflow_id, context, stepdir)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/steps.py", line 292, in run
step.run(self, context, "%s-0" % ii)
~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1670, in run
raise RuntimeError("Step %s failed" % self)
RuntimeError: Step run-train-init-artifact failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1666, in run
self.exec(scope, parameters, None, context, order)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1856, in exec
self.exec_steps(scope, parameters, item, context, order)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1954, in exec_steps
raise RuntimeError("Step %s failed" % self)
RuntimeError: Step concurrent-learning-block-prep-run-dp-train failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1949, in exec_steps
steps.run(scope.workflow_id, context, stepdir)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/steps.py", line 292, in run
step.run(self, context, "%s-0" % ii)
~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1670, in run
raise RuntimeError("Step %s failed" % self)
RuntimeError: Step concurrent-learning-block-prep-run-dp-train failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1666, in run
self.exec(scope, parameters, None, context, order)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1856, in exec
self.exec_steps(scope, parameters, item, context, order)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1954, in exec_steps
raise RuntimeError("Step %s failed" % self)
RuntimeError: Step concurrent-learning-loop-block failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1949, in exec_steps
steps.run(scope.workflow_id, context, stepdir)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/steps.py", line 292, in run
step.run(self, context, "%s-0" % ii)
~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1670, in run
raise RuntimeError("Step %s failed" % self)
RuntimeError: Step concurrent-learning-loop-block failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1666, in run
self.exec(scope, parameters, None, context, order)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1856, in exec
self.exec_steps(scope, parameters, item, context, order)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1954, in exec_steps
raise RuntimeError("Step %s failed" % self)
RuntimeError: Step concurrent-learning-loop failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1949, in exec_steps
steps.run(scope.workflow_id, context, stepdir)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/steps.py", line 292, in run
step.run(self, context, "%s-0" % ii)
~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1670, in run
raise RuntimeError("Step %s failed" % self)
RuntimeError: Step concurrent-learning-loop failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1666, in run
self.exec(scope, parameters, None, context, order)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1856, in exec
self.exec_steps(scope, parameters, item, context, order)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1954, in exec_steps
raise RuntimeError("Step %s failed" % self)
RuntimeError: Step dpgen-step failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/workflow.py", line 345, in submit
entrypoint.run(self.id, self.context, wfdir)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/steps.py", line 292, in run
step.run(self, context, "%s-0" % ii)
~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1670, in run
raise RuntimeError("Step %s failed" % self)
RuntimeError: Step dpgen-step failed
```
关闭于 2025-08-25 1 条评论