ITADN

[Enhancement] Add "Winner.BothBad" state to pairwise comparison outcomes

#41Closedzhimin-z 创建于 2025-05-04
Z
zhimin-zcommented
Hi, @dustalov @shenxiangzhuang This library provides excellent metrics for pairwise model comparison. Currently, the comparison outcomes only include three states: ``` WINNERS = [ Winner.X, Winner.Y, Winner.Draw ] ``` This classification system lacks a way to distinguish between true draws (where both models perform adequately but equally well) and cases where both models fail to generate useful output. I recommend adding a new enumeration called `Winner.BothBad` or similar. This enhancement would: - Allow tracking the failure rate of evaluated models separately from draws - Provide more nuanced evaluation insights for software engineering tasks - Make the comparison framework more comprehensive by distinguishing between "**equally good**" and "**equally bad**" outcomes This addition would significantly enhance the analytical capabilities of your metrics calculation library when used in evaluation platforms such as [SE Arena](https://huggingface.co/spaces/SE-Arena/Software-Engineering-Arena).
关闭于 2025-05-13 13 条评论