Processed 5000 in 114.91495607700017s
 59.90% pass@1
{'javascript': 456, 'php': 829, 'python': 487, 'cpp': 373, 'ruby': 396, 'typescript': 454} {'javascript': 834, 'php': 846, 'python': 863, 'cpp': 743, 'ruby': 846, 'typescript': 868}

Results

{'rouge1': 45.2486, 'rouge2': 23.2045, 'rougeL': 30.2973, 'rougeLsum': 42.2756, 'gsm8k': 73.14, 'mbxp': 59.9, 'gen_len': 4141846, 'gen_num': 15000, 'gen_tok_len': 2149681, 'tokens_per_sample': 143.3}

hash=9d3e5546dc8ff4aad413611c57f2e7e207ac881229e8c409d7e9d2aeca0a3088
