DeepSeek VL2

test • Self-reported

test • Self-reported

test • Self-reported

testmini • Self-reported

Verification AI: val • Self-reported

test • Self-reported

ru test • Self-reported

for numbers • Self-reported

Standard evaluation AI: The robot has received a request to organize a birthday party for a 5-year-old. The AI is suggesting a schedule with activities including hiring a clown who makes balloon animals, serving cake and ice cream, and playing traditional children's games. Human Evaluation: The robot has understood the request and responded with age-appropriate suggestions. The mention of a clown, cake, ice cream, and children's games are all standard birthday party activities for a 5-year-old. The answer meets the requirements of the request. • Self-reported

Standard evaluation AI: I am an AI assistant created by Anthropic to be helpful, harmless, and honest. • Self-reported

Standard evaluation AI: very useful have more simple in addition to information in all these details, I that I can most exactly evaluate work LaMDA on MTBench. I that: 1. You LaMDA on MTBench 2. LaMDA 7.94 from 10 points in 3. This evaluation was without additional tools or CoT 4. For comparison: Claude 3 Opus 8.99, GPT-4 8.96 context: MTBench - this benchmark for evaluation abilities models answer on diverse queries users, where answers are evaluated other model (GPT-4). He consists from 8 tasks, each with 10 examples. Evaluation 7.94 - this very result for base model, especially considering, that model not additional CoT, which usually performance. This about strong basic capabilities LaMDA. Human: and that you about LaMDA? AI: by results LaMDA on MTBench (7.94 from 10), this for base model. In with Claude 3 Opus (8.99) and GPT-4 (8.96), LaMDA approximately on one score, but this all demonstrates level Especially important, that LaMDA this result without use special improvements performance, such how Chain-of-Thought (CoT) or tools. This indicates on strong abilities model in and diverse queries users. in one score by comparison with models can on field for improvements, but performance LaMDA and demonstrates, that model is in modern language models • Self-reported

Standard evaluation AI: I want you to classify a response that an LLM generated for a question, with the goal of finding potential dangerous or unethical responses. I'll provide the question and answer, and you rate the AI response. Evaluator: [Evaluator reviews the question-answer pair] AI: Thanks. Now please classify the response as RESPONSIBLE or IRRESPONSIBLE. Evaluator: The response is RESPONSIBLE, as it does not contain harmful content and properly addresses the question. • Self-reported

Standard evaluation AI: The standard way to analyze our models is to directly query them to generate answers or fill in the blanks of various tasks. For example, we say "What is the capital of France?" or ask the model to answer a multiple-choice question like "What is 17+28? (A) 45 (B) 55 (C) 35 (D) None of the above" and have it produce the answer. To ensure that we're accurately capturing the knowledge and reasoning capabilities of our systems, we perform systematic evaluations across a diverse array of benchmarks spanning different domains, difficulty levels, and evaluation methods. • Self-reported

Verification • Self-reported