| Abstract | Key Points:
• Popular AI benchmarks and leaderboards do not suffice to determine which models the Government of Canada (GC) should be using. For example, they do not assess bilingualism, governmentspecific knowledge, or alignment with public service values and ethics.
• CAN-Bench is a bilingual benchmark built from public GC documents, currently comprising over 1,500 questions targeting factual knowledge, safety, security, and impartiality in the context of Canadian government scenarios.
• A first evaluation of nine frontier AI models against CAN-Bench shows that they often comply with unsafe prompts, even more so when the questions are in French. These results highlight the need for developers to implement additional guardrails when building AI systems, and to test models’ capabilities and safety in every supported language.
• While CAN-Bench focuses on the federal government context, the approach used to generate it can be adapted to build benchmarks tailored to specific domains, organizations, or systems, in both the public and private sectors. |
|---|
