DeepSeek R1 Distill Qwen 7B

Diamond, Pass@1 Methodology evaluation, on abilities model generate although would one correct answer for several attempts. For each tasks model generates several independent solutions (usually 5-200), and then solution from total set. Pass@1 evaluates probability that, that model correct answer with first attempts. This metric especially useful for tasks, where exists correct answer, for example, for mathematical tasks, programming or Methodology accounts for data large language models and measures their performance at solving complex tasks • Self-reported

Cons@64 Cons@64 — this method improvement accuracy computations in LLM, which consists in intermediate tokens output for obtaining more correct answers. In situations, when model performs computation, complex logical tasks or numerical computation, answers often from-for errors in chain reasoning. Cons@64 solves this problem by means of limitations output model to 64 tokens, actually chains reasoning, which can lead to to When model number tokens, she/it give and answers, that can lead to to more high accuracy. This especially effectively for tasks, requiring numerical computations or specific conclusions. explanation consists in that, that number tokens probability errors in intermediate which then on final answer. Cons@64 not requires model or her/its — this simply output, which can apply to LLM • Self-reported

Pass@1 In research models Pass@1 — this metric, used for evaluation performance language models in tasks, requiring exact output, especially in and mathematical tasks. Definition: Pass@1 measures probability that, that model will solve task with first attempts without necessity generate several options. In difference from metrics, on (such how Pass@k, where k > 1), Pass@1 evaluates ability model generate one solution immediately, that important for real scenarios use, where users correct answer with first times. Application: - Evaluation solutions tasks programming, where one solution can lead to to or work - tasks, where model should not only find answer, but and provide step-by-step solution - Evaluation reliability model in where attempts or : Pass@1 = (Number tasks, solved with first attempts) / (number tasks) Pass@1 especially how score reliability model and her/its abilities to exact reasoning at capabilities set • Self-reported

Pass@1 Metric performance, which measures probability that, that model correctly will solve task with first attempts. In context reasoning with help (sampling-based reasoning) this means probability obtaining correct answer from one output model. Pass@1 is calculated by means of model with different (seeds) and solutions. This metric especially important in scenarios, where at user is only one attempt obtain answer from model, that reflects majority real with LLM. High score Pass@1 indicates on then, that model and solves tasks without necessity in attempts • Self-reported