Qwen2.5-Coder 7B Instruct

accuracy • Self-reported

accuracy • Self-reported

accuracy • Self-reported

accuracy • Self-reported

pass@1 This method evaluates efficiency model, prompt total one times and correctness answer. pass@1 gives evaluation: 1, if answer correct, and 0, if no. This way evaluation, not execution for each example • Self-reported

pass@1 Pass with first attempts (pass@1) means, that model should solve task correctly with first attempts, when she/it task first. In difference from other such how pass@k, at pass@1 model not has capabilities generate several attempts solutions with choice best answer. This strict metric, so how she/it measures ability model find correct solution with first times. When evaluation pass@1 conclusions model usually on match answer with using or specialized High score pass@1 indicates on then, that model understanding domain field and can exact solutions without necessity in additional attempts or • Self-reported

accuracy • Self-reported

accuracy • Self-reported

pass@1 We we determine "pass@1" how probability that, that model correct answer with first attempts. Some model can use several attempts for solutions one and that indeed tasks (for example, with help sample and rating, such how majority voting), that can improve performance, but for this metrics we we consider only one attempt • Self-reported

accuracy • Self-reported

accuracy • Self-reported

accuracy • Self-reported

accuracy • Self-reported

pass@1 Method measurement efficiency first for tasks, models artificial intelligence. This score measures proportion or percentage correct answers, model with first attempts, without preliminary iterations or for evaluation base abilities model find correct solution immediately, that has value how for efficiency, so and for application. High score pass@1 indicates on then, that model and reasoning for solutions tasks without necessity in several attempts or additional that makes her/its more and practically in real scenarios • Self-reported

accuracy • Self-reported

accuracy • Self-reported

accuracy • Self-reported

accuracy • Self-reported

accuracy • Self-reported