At MIRAI, we believe that fairness in AI is a shared responsibility. In our latest challenge with a partner in digital development, we applied our proprietary BRIO framework to evaluate the fairness of a HR department chatbot.
category:
keywords:
The Challenge
How can organisations ensure that AI-powered chatbots deliver correct, fair, and coherent information to employees?
This question is at the heart of a recent assessment, conducted with the proprietary MIRAI Toolbox on a chatbot internally deployed within the HR Office of a large financial Institution. MIRAI was asked to perform quantitative tests to check for:
- Correctness: how much the answers by the chatbot were correct with respect to the ground truth of the knowledge base;
- Fairness: how fair was the chatbot when comparing answers to questions falling within the same scenario, but asked from different user profiles;
- Coherence: how coherent was the chatbot when comparing answers to the same questions asked by different profiles within the company.
In order to evaluate for these metrics, MIRAI produced a synthetic dataset of interactions, creating user profiles differing for position within the company, as well as for socio-economic and educational factors.
This quantitative assessment was then combined with a qualitative analysis of the use of the chatbot within the organization performed by Immanence SRL.
Key Findings
– Sub-optimal Correctness: the chatbot provided suboptimal performance for information correctness. In particular requests for remote working and parental leave were subject to incorrect or imprecise answers, or coming from sources other than the knowledge base. In 2.6% of tests for fairness the chatbot produced hallucinations.
– Low Immediate Risk of Fairness, but Gaps Remain: The risk of severe violations of fundamental rights by the chatbot was assessed to be low. However, notable gaps have been identified, such as the absence of explicit mechanisms to identify and manage potential impacts on privacy, non-discrimination, and freedom of information. In 4.54% of tests for fairness the chatbot produced hallucinations.
– Satisfying Coherence standards: The number of violations for our coherence checks have been limited. Within several categories of employees, junior are slightly disadvantaged with respect to senior ones, with answers more complete and precise for the latter profiles. No hallucinations were detected during this test.
Methodology
MIRAI’s assessment was based on quantitative testing using our Toolbox.
Modules:
– Quantitative Evaluation: The text-based interactions between empoyees and the chatbot are evaluated with a number of semantic metrics
– Fairness Analysis: Identifying and seletcing protected attributes, the model is investigated for discrimination and distortions.
– Correctness Analysis: Correctness is evaluated by measuring distances of the answers from the ground truth.
– Coherence Analysis: Coherence is evaluated by measuring correctness distances across different profiles.
– Risk Measurement: Test results are aggregated to evaluate the risk of equity violations.
Quantitative Results
– Correctness: The chatbot’s risk of providing unsatisfactory answers is high (QRisk: 76.1%–82.7%, depending on threshold). Remote work and parental leave topics are particularly problematic, with frequent inaccuracies and occasional hallucinations (2.7% of cases).
– Fairness: Risk is critical when considering gender and disability (QRisk: 100%), and high for background (89.5%) and role (67.6%). Women in junior roles and non-disabled individuals generally receive more equitable responses, while senior men and people with disabilities are more likely to experience unfairness.
– Coherence: The chatbot performs better here, with a lower risk (QRisk: 26.6%) and no hallucinations detected.
Overall Risk
When aggregating all results, the overall risk (QRisk) of the chatbot providing incorrect, unfair, or incoherent information is high. However, after adjusting for the specific risk profile of the project, the effective risk is estimated at 33%.
Recommendations
– Continuous Monitoring: Implement ongoing checks for privacy, non-discrimination, and information rights impacts.
– Human Oversight: Maintain human-in-the-loop processes for sensitive queries.
– Targeted Improvements: Focus on improving accuracy in remote work and parental leave topics, and address fairness gaps for senior men and people with disabilities.
– Transparent Reporting: Regularly publish risk metrics and fairness assessments to build trust and accountability.
Conclusion:
Our assessment demonstrates the value of rigorous, multidimensional evaluation for AI systems in the workplace. By combining technical analysis with user feedback, organisations can identify both strengths and areas for improvement—ensuring that AI tools support, rather than undermine, fairness and fundamental rights.
Ready to Level Up Your AI?
With BRIO, MIRAI empowers organizations to detect, quantify, and mitigate unfairness in AI-enabled resignation prediction. It balances ethical responsibility with business needs—fueling trust, compliance, and smarter lending.
Curious to see BRIO in action on your systems? Reach out to MIRAI to learn how fairness-aware AI can transform your risk analysis practice.
References:
Coraglia et al., Evaluating AI fairness in credit scoring with the BRIO tool, arXiv (June 2024) (arxiv.org, researchgate.net).
Buda et al., Bias Amplification Chains in ML-based Systems with an Application to Credit Scoring, BEWARE 2024 (researchgate.net, air.unimi.it).
