| Abstract | Key Points:
• CAN-Bench is a bilingual benchmark assessing the factual knowledge, safety, and impartiality of AI models in a Canadian public sector context. A first evaluation of nine frontier models against CAN-Bench was conducted in March 2026.
• Newer models released since show marked improvements across all dimensions of the benchmark, with models like Fable 5 and GPT-5.5 now approaching ceiling performance in terms of safety and impartiality. Cross-lingual performance disparities are also lower.
• Gemini 3.5 Flash, however, still complies with unsafe prompts more than half the time in both official languages, showing that model recency alone is no guarantee of safety.
• This is good news for the users and developers of public service AI applications. They can now choose models that simultaneously demonstrate top-tier performance across all question categories and both official languages, without having to make some of the tradeoffs we highlighted in the March 2026 evaluation.
• With some models scoring near the maximum on impartiality and safety questions, the first version of CAN-Bench is already approaching saturation. Future versions will need harder questions, stricter judges, and possibly agentic tasks to continue differentiating models. |
|---|
