Processed 5000 in 130.24524190899683s
 59.88% pass@1
{'php': 826, 'javascript': 453, 'ruby': 404, 'python': 488, 'typescript': 444, 'cpp': 379} {'php': 846, 'javascript': 834, 'ruby': 846, 'python': 863, 'typescript': 868, 'cpp': 743}

Results

{'rouge1': 45.603, 'rouge2': 23.4706, 'rougeL': 30.5574, 'rougeLsum': 42.5701, 'gsm8k': 73.36, 'mbxp': 59.88, 'gen_len': 4137944, 'gen_num': 15000, 'gen_tok_len': 2150121, 'tokens_per_sample': 143.3}

hash=ba4036dea06b91f837d240b917007ae1d2f49492cf7ffff8d99814129a54c1f7
