[Enhancement] Add "Winner.BothBad" state to pairwise comparison outcomes
Hi, @dustalov @shenxiangzhuang This library provides excellent metrics for pairwise model comparison. Currently, the comparison outcomes only include three states:
```
WINNERS = [
Winner.X,
Winner.Y,
Winner.Draw
]
```
This classification system lacks a way to distinguish between true draws (where both models perform adequately but equally well) and cases where both models fail to generate useful output. I recommend adding a new enumeration called `Winner.BothBad` or similar. This enhancement would:
- Allow tracking the failure rate of evaluated models separately from draws
- Provide more nuanced evaluation insights for software engineering tasks
- Make the comparison framework more comprehensive by distinguishing between "**equally good**" and "**equally bad**" outcomes
This addition would significantly enhance the analytical capabilities of your metrics calculation library when used in evaluation platforms such as [SE Arena](https://huggingface.co/spaces/SE-Arena/Software-Engineering-Arena).
关闭于 2025-05-13 13 条评论