ITADN

There is no supported evaluation like swebench as mentioned in the official docs

#55OpenZoneTwelve 创建于 2026-04-15
Z
ZoneTwelvecommented
## Bug Report ### Description The [official documentation / website](https://docs.coralxyz.com/getting-started/installation#optional-dependencies) mentions support for several evaluation benchmarks, specifically **SWE-bench**. However, when attempting to install the required dependencies or use the benchmark, it appears to be completely missing from the current repository. Specifically, trying to sync the `swebench` extra via `uv` fails because it is not defined in [pyproject.toml](cci:7://file:///private/tmp/CORAL/pyproject.toml:0:0-0:0). ### Steps to Reproduce ```bash # SWE-bench uv sync --extra swebench # Terminal-Bench uv sync --extra terminalbench # Erdős Problems uv sync --extra erdos ``` ### Expected Behavior Dependencies for the documented benchmarks (like SWE-bench, Terminal-bench) are available and can be installed via the package manager. ### Actual Behavior ``` Resolved 119 packages in 4ms error: Extra `swebench` is not defined in the project's `optional-dependencies` table ``` ### Environment - CORAL version: [9ccabab](https://github.com/Human-Agent-Society/CORAL/tree/9ccabab42853715c8d1bd97598302b473a374d26) - Python version: `Python 3.12.3` - Astral UV: `uv 0.9.15` ### Additional Context A search of the repository shows that while the `swebench` grader type is referenced in [coral/template/coral_md.py](cci:7://CORAL/coral/template/coral_md.py:0:0-0:0) (for agent instructions), the actual grader implementation and its associated Python dependency extras are missing from [pyproject.toml](cci:7://CORAL/pyproject.toml:0:0-0:0) and the source tree.
0 条评论