There is no supported evaluation like swebench as mentioned in the official docs
## Bug Report
### Description
The [official documentation / website](https://docs.coralxyz.com/getting-started/installation#optional-dependencies) mentions support for several evaluation benchmarks, specifically **SWE-bench**. However, when attempting to install the required dependencies or use the benchmark, it appears to be completely missing from the current repository.
Specifically, trying to sync the `swebench` extra via `uv` fails because it is not defined in [pyproject.toml](cci:7://file:///private/tmp/CORAL/pyproject.toml:0:0-0:0).
### Steps to Reproduce
```bash
# SWE-bench
uv sync --extra swebench
# Terminal-Bench
uv sync --extra terminalbench
# Erdős Problems
uv sync --extra erdos
```
### Expected Behavior
Dependencies for the documented benchmarks (like SWE-bench, Terminal-bench) are available and can be installed via the package manager.
### Actual Behavior
```
Resolved 119 packages in 4ms
error: Extra `swebench` is not defined in the project's `optional-dependencies` table
```
### Environment
- CORAL version: [9ccabab](https://github.com/Human-Agent-Society/CORAL/tree/9ccabab42853715c8d1bd97598302b473a374d26)
- Python version: `Python 3.12.3`
- Astral UV: `uv 0.9.15`
### Additional Context
A search of the repository shows that while the `swebench` grader type is referenced in [coral/template/coral_md.py](cci:7://CORAL/coral/template/coral_md.py:0:0-0:0) (for agent instructions), the actual grader implementation and its associated Python dependency extras are missing from [pyproject.toml](cci:7://CORAL/pyproject.toml:0:0-0:0) and the source tree.
0 条评论