Sakana AI Releases JFinBench: A Japanese Financial Benchmark Based on Latest Certification Exams
According to Sakana AI: Sakana AI has introduced JFinBench, a new benchmark designed to evaluate large language models on real-world Japanese financial certification exams from 2026. Evaluating 12 leading models revealed that top open-weight models still lag behind proprietary models, primarily due to a lack of local regulatory knowledge rather than calculation errors.

Original image temporarily unavailable
According to Sakana AI:
Background and Motivation
As large language models take on more specialized roles, specialized benchmarks are becoming crucial for reflecting real-world capabilities. While fields like software coding have seen open-weight models nearly match proprietary ones, domains requiring localized, frequently changing knowledge—such as Japanese finance and tax systems—lack robust evaluation standards. Existing benchmarks often rely on US systems or outdated exams.
Benchmark Construction
JFinBench comprises 321 questions sourced from six top-tier Japanese financial certification exams held in 2026, including certified public accountants and tax accountants. To tackle the challenge of converting complex PDFs into structured data, Sakana AI introduced a human-in-the-loop agentic pipeline. Coding agents handled the heavy lifting of parsing layouts and formatting, while human experts verified all questions in about 12 hours total.
Evaluation Insights
Testing 12 prominent models showed that proprietary models like Claude Fable 5 and GPT-6 Astra dominated the top tiers. The best open-weight model, Kimi K3.1 Pro, scored significantly lower, highlighting a gap not visible in general coding benchmarks. The primary reason for failure was insufficient regulatory knowledge rather than arithmetic mistakes. Equipping models with external web tools helped bridge the gap, yet a performance deficit between open and proprietary models remained.
View selection rationale
Importance 83/100
About this brief
- Content type
- Research summary
- Source type
- First-party / official source
- Original source
- Sakana AI ↗
- Source published
- Process
- AI-assisted discovery, translation, summarisation, classification and scoring; standards and corrections are maintained by the editorial desk.
- Verification
- The original URL and publication metadata are retained. This brief is not independently fact-checked unless explicitly stated.
- Published by
- Polyemergence Editorial Desk · PolyEmergence Pte Ltd.
- Conflict disclosure
- No financial or commercial conflict declared.
