Nova Pro

0-shot Chain-of-Thought AI: 0-shot Chain-of-Thought • Self-reported

0-shot pass@1 AI: ** • Self-reported

0-shot Chain-of-Thought AI: 0-shot Chain-of-Thought • Self-reported

0-shot Chain-of-Thought AI: 0-shot Chain-of-Thought • Self-reported

## 0-shot 0-shot AI, few-shot () fine-tuning (). 0-shot evaluation : 1. 2. 3. 0-shot : - "", : - 0-shot (GPQA, MATH), (HumanEval) • Self-reported

6-shot Chain-of-Thought AI: 6 (chain-of-thought) • Self-reported

relaxed accuracy • Self-reported

ANLS (ANLS) - ANLS (LCS) ANLS : ANLS = (LCS()) / ((), ()) LCS - ANLS 0 1, 1 0 ANLS • Self-reported

AI: : Chain-of-thought (CoT) prompting is a technique that helps large language models (LLMs) tackle challenging problems by breaking down their reasoning into manageable steps. First introduced in the paper "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" (Wei et al., 2022), CoT prompting has become one of the most important techniques for improving the reasoning capabilities of LLMs. CoT prompting can be implemented in various ways: - Few-shot CoT: The prompt includes examples that demonstrate step-by-step reasoning for similar problems - Zero-shot CoT: The model is simply instructed to "think step by step" with no examples provided - Self-consistency with CoT: The model generates multiple reasoning paths and selects the most consistent answer • Self-reported

0-shot Chain-of-Thought AI: 0-shot Chain-of-Thought • Self-reported

3-shot Chain-of-Thought • Self-reported

accuracy • Self-reported

accuracy • Self-reported

accuracy • Self-reported

0-shot accuracy • Self-reported

accuracy • Self-reported

# 0-shot • Self-reported

accuracy • Self-reported

Accuracy AI: 1 step accuracy, in %, [n]: The percentage of n-step reasoning traces that are correct at each step. For example, the proportion of n-step reasoning traces that get step 1 right, the proportion that get step 2 right given that step 1 is right, etc. This metric is useful for identifying where models make errors in multi-step reasoning, and how error rates change along the course of a reasoning trace • Self-reported

ROUGE-L ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation ) (LCS). LCS — — "", — "", LCS "" (3 ). ROUGE-L : (Recall) = _LCS / __Accuracy (Precision) = _LCS / __F-= (1 + β²) × (× Accuracy) / (β² × + Accuracy) β 1. ROUGE-L • Self-reported

weighted accuracy • Self-reported

COMET22 Score • Self-reported

spBleu BLEU. BLEU, spBleu (AST) (AST), AST "int foo(int a)" "INT_TYPE IDENTIFIER(INT_TYPE IDENTIFIER)". spBleu N-BLEU-spBleu • Self-reported

COMET22 COMET22, COMET22 COMET22 • Self-reported

spBleu BLEU BLEU spBleu "A B" "C D". — — • Self-reported

CIDEr CIDEr (Consensus-based Image Description Evaluation) – : - n-accuracy, TF-IDF CIDEr n-(n-) • Self-reported

Standard evaluation • Self-reported