Qwen2.5-Coder 32B Instruct

accuracy • Self-reported

accuracy • Self-reported

accuracy • Self-reported

accuracy • Self-reported

pass@1 Method verifies, capable whether model solve task with first attempts. Task is considered if answer correct at If answer incorrect, task is considered additional question or attempts solutions on several examples not Model or handles with task with first attempts, or not handles. Metric not accounts for possible improvements at additional attempts and reflects base ability model solve tasks without approach • Self-reported

pass@1 This metric evaluates probability that, that model correct answer with first attempts, then is in its answer. For each tasks model receives one if her/its first answer contains correct solution, and in case. score represents itself average value by all tasks. In difference from other metrics, such how pass@k, which evaluate probability correct answer among several attempts, pass@1 evaluates ability model find correct solution with first times. This strict since he not allows no/none errors in process solutions. pass@1 is metric for evaluation reliability and accuracy model in tasks, where important correctness, for example, in or where at users can not be capabilities or resources for verification several answers • Self-reported

accuracy • Self-reported

accuracy • Self-reported

accuracy • Self-reported

accuracy • Self-reported

accuracy • Self-reported

pass@1 — this way measurement performance model, when it is provided only one attempt. This standard score, for evaluation model on tasks, which require exact answer. For example, if model answers on 75 from 100 questions correctly with first attempts, then score pass@1 will 75%. In difference from other metrics, such how pass@k (where model generates k different answers and is considered if although would one correct), pass@1 ability model find correct answer with first attempts. This more strict measure, since model not receives several This score especially important for applications, where usually is required one specific answer, and not several options • Self-reported

accuracy • Self-reported

accuracy • Self-reported

accuracy • Self-reported