Llama 3.2 3B Instruct

0-shot, accuracy ChatBot: AI • Self-reported

5-shot, macro_avg/acc • Self-reported

8-attempts, em_maj1@1 AI: ChloeAI Prompt: We use 8 shots of previous QA pairs in a retrieval setting, where we retrieve relevant context by embedding similarity. We define test accuracy as the majority vote (maj1) of the model's answers over all 8 trials for a single exact match (em). This aggregates over potential randomness in responses. • Self-reported

0-shot, final_em For each example we used one query to model. We not answers model, final extraction answer. answer (final_em): We we determine answer how process answer from model after that, how she/it fully solution tasks. For this we following : (1) If answer already in format (for example, "The answer is 42"), we final answer (in given case "42"). (2) If task in format with multiple choice, and model indicates option (for example, "(A)"), we this option. (3) In case we final answer in or model. If several numbers, we number • Self-reported

Chain thinking, em AI: I don't understand the "em" in this text. Let me reason about this step by step. In the context of prompt engineering and AI methods, "CoT" clearly refers to "Chain of Thought", which is a prompting technique where the model is encouraged to break down its reasoning into sequential steps. The "em" could potentially refer to: 1. "Expectation maximization" - a statistical algorithm 2. "em" as in emphasis in HTML/markdown (like *this*) 3. Some kind of metric or modifier related to CoT 4. A typo or abbreviation for something else Since this is just a two-word fragment without context, the most likely interpretation is that it's referring to Chain of Thought reasoning with some kind of "em" qualifier or metric associated with it. But without more context, I can only provide this basic translation of the terms as they appear. • Self-reported

0-shot, accuracy AI: In this category, we compute the accuracy of the model's predictions on our pre-determined list of questions directly from the model's top 1 output, without any prompting or support. • Self-reported

0-shot, acc standard 0-shot evaluation without which-or additional information or demonstrations. Accuracy is calculated on set, in order to Since methodology 0-shot is used how in evaluation, so and at testing in this match between evaluation and • Self-reported

0-shot, accuracy AI: Prompt Steerability • Self-reported

Average value (accuracy instructions/prompts strict/) • Self-reported

0-shot, longbook_choice/acc • Self-reported

0-shot, longbook_qa/f1 • Self-reported

0-shot, macro_avg/acc • Self-reported

0-shot, reproduction AI: model used which in ? • Self-reported

0-shot, micro_avg/rougeL • Self-reported

1-shot, rougeL • Self-reported