| Download | - View final version: Towards custom AI benchmarking for the Government of Canada (PDF, 1.1 MiB)
|
|---|
| Author | Search for: Bernier-Colborne, Gabriel1; Search for: Gauthier, Yvan1; Search for: Vajjala, Sowmya1ORCID identifier: https://orcid.org/0000-0002-4033-9936 |
|---|
| Affiliation | - National Research Council Canada. Digital Technologies
|
|---|
| Format | Text, Article |
|---|
| Conference | The 39th Canadian Conference on Artificial Intelligence, May 25-29, 2026, Vancouver, British Columbia, Canada |
|---|
| Subject | LLM evaluation; custom AI benchmarking |
|---|
| Abstract | The Government of Canada (GC) has several options when selecting artificial intelligence (AI) systems to support its operations. At the same time, AI safety issues motivate it to assess these models for various risks that could harm users seeking information about government operations. Thus, evaluation of AI systems has become an area of concern for the GC. Existing AI benchmarks do not suffice to inform the evaluation/selection process as they are generally not adequate for this. To address this problem, we are building CAN-Bench, a bilingual benchmark designed for the Canadian public service context. Based on a dataset compiled from public GC documents, we automatically generate a bilingual set of high-quality questions around government knowledge, safety, and public service values. This paper describes the methodology for benchmark construction and a comparison of various AI models on the benchmark. Our results indicate that while the AI models we tested are good at answering general knowledge questions about government policies, they are not always aligned with public sector values such as non-partisanship, and can potentially provide unsafe responses in some scenarios. |
|---|
| Date published | 2026-05 |
|---|
| Publisher | The 39th Canadian Conference on Artificial Intelligence
MLRResearchPress |
|---|
| Copyright statement | |
|---|
| Licence | |
|---|
| In | |
|---|
| Language | English |
|---|
| Peer reviewed | Yes |
|---|
| Export citation | Export as RIS |
|---|
| Report a correction | Report a correction (opens in a new tab) |
|---|
| Record identifier | 55be3028-79fe-48b4-b2ed-5468b1c771e0 |
|---|
| Record created | 2026-05-13 |
|---|
| Record modified | 2026-06-10 |
|---|
