ITADN

BUG: FileNotFoundError during resubmit

#305ClosedOutisLi 创建于 2025-08-23
O
OutisLicommented
After successfully running a dpgen2 job, I modified the input json to envolve more stages in explore. I want to reuse the previous jobs, but it raised FileNotFound error. I also tried no modifications on dpgen.json it is the same. It cannot resubmit the job. Through the `-l` command, we can see the previous jobs finished. ```shell dpgen2 resubmit dpgen_wsl.json job-name-lryu0 -l 0 : init--scheduler 1 : init--id 2 : iter-000000--prep-train 3 -> 3 : iter-000000--run-train-0000 -> iter-000000--run-train-0000 4 : iter-000000--prep-lmp 5 -> 5 : iter-000000--run-lmp-000000 -> iter-000000--run-lmp-000000 6 : iter-000000--select-confs 7 : iter-000000--prep-fp 8 -> 8 : iter-000000--run-fp-000000 -> iter-000000--run-fp-000000 9 : iter-000000--collect-data 10 : iter-000000--scheduler 11 : iter-000000--id 12 : iter-000001--prep-train 13 -> 13 : iter-000001--run-train-0000 -> iter-000001--run-train-0000 14 : iter-000001--prep-lmp 15 -> 15 : iter-000001--run-lmp-000000 -> iter-000001--run-lmp-000000 16 : iter-000001--select-confs 17 : iter-000001--prep-fp 18 -> 18 : iter-000001--run-fp-000000 -> iter-000001--run-fp-000000 19 : iter-000001--collect-data 20 : iter-000001--scheduler 21 : iter-000001--id ``` But resubmit raise errors: ```shell dpgen2 resubmit dpgen_wsl.json job-name-lryu0 -u 0-21 Workflow is running locally (ID: job-name-smbtm) CP2KDATA| step (key: init--scheduler) skipped CP2KDATA| step (key: init--id) skipped CP2KDATA| step (key: iter-000000--prep-train) skipped Traceback (most recent call last): File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1666, in run self.exec(scope, parameters, None, context, order) ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1858, in exec self.exec_pod(scope, parameters, item, order) ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 2189, in exec_pod os.makedirs(save_path, exist_ok=True) ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<frozen os>", line 228, in makedirs FileNotFoundError: [Errno 2] No such file or directory: '/home/outis/Research/dp_train/dpgen2/test/dpgen2_job/job-name-smbtm/.././job-name-smbtm/iter-000000--run-train-group/model' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1949, in exec_steps steps.run(scope.workflow_id, context, stepdir) ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/steps.py", line 292, in run step.run(self, context, "%s-0" % ii) ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1670, in run raise RuntimeError("Step %s failed" % self) RuntimeError: Step run-train-init-artifact failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1666, in run self.exec(scope, parameters, None, context, order) ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1856, in exec self.exec_steps(scope, parameters, item, context, order) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1954, in exec_steps raise RuntimeError("Step %s failed" % self) RuntimeError: Step concurrent-learning-block-prep-run-dp-train failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1949, in exec_steps steps.run(scope.workflow_id, context, stepdir) ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/steps.py", line 292, in run step.run(self, context, "%s-0" % ii) ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1670, in run raise RuntimeError("Step %s failed" % self) RuntimeError: Step concurrent-learning-block-prep-run-dp-train failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1666, in run self.exec(scope, parameters, None, context, order) ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1856, in exec self.exec_steps(scope, parameters, item, context, order) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1954, in exec_steps raise RuntimeError("Step %s failed" % self) RuntimeError: Step concurrent-learning-loop-block failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1949, in exec_steps steps.run(scope.workflow_id, context, stepdir) ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/steps.py", line 292, in run step.run(self, context, "%s-0" % ii) ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1670, in run raise RuntimeError("Step %s failed" % self) RuntimeError: Step concurrent-learning-loop-block failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1666, in run self.exec(scope, parameters, None, context, order) ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1856, in exec self.exec_steps(scope, parameters, item, context, order) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1954, in exec_steps raise RuntimeError("Step %s failed" % self) RuntimeError: Step concurrent-learning-loop failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1949, in exec_steps steps.run(scope.workflow_id, context, stepdir) ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/steps.py", line 292, in run step.run(self, context, "%s-0" % ii) ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1670, in run raise RuntimeError("Step %s failed" % self) RuntimeError: Step concurrent-learning-loop failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1666, in run self.exec(scope, parameters, None, context, order) ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1856, in exec self.exec_steps(scope, parameters, item, context, order) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1954, in exec_steps raise RuntimeError("Step %s failed" % self) RuntimeError: Step dpgen-step failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/workflow.py", line 345, in submit entrypoint.run(self.id, self.context, wfdir) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/steps.py", line 292, in run step.run(self, context, "%s-0" % ii) ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/outis/miniforge3/envs/dpmd/lib/python3.13/site-packages/dflow/step.py", line 1670, in run raise RuntimeError("Step %s failed" % self) RuntimeError: Step dpgen-step failed ```
关闭于 2025-08-25 1 条评论