In 2018, Reuters revealed that Amazon had developed an AI recruiting tool to screen CVs for software engineering roles. The system was trained on ten years of hiring data — which reflected a decade of predominantly male hiring in tech. The AI learned that being male was correlated with being hired, and began penalising CVs that contained the word “women’s” (as in “women’s chess club captain”) and downgrading graduates of all-women’s colleges.
Amazon scrapped the tool. But the underlying problem did not go away. Every organisation using AI for decisions that affect people — hiring, performance reviews, promotions, lending, pricing, customer service — faces the same risk. AI systems trained on historical data will learn and reproduce the biases embedded in that data. Unless you actively detect and prevent it, AI will automate discrimination at a scale and speed that human decision-making never could.
À retenir
- AI bias occurs when systems produce systematically unfair outcomes for specific groups — often without anyone intending it
- Major companies including Amazon, Apple, Google, and the UK Home Office have been affected by documented AI bias incidents
- The EU AI Act classifies AI used in employment and HR decisions as high-risk, with strict bias testing and transparency requirements
- Prevention requires a combination of diverse training data, regular bias audits, human oversight, and organisational commitment
Types of AI bias in the workplace
AI bias is not a single phenomenon. It manifests in multiple forms, each with different causes and different solutions.
Historical bias
When AI is trained on historical data that reflects past discrimination, it learns to replicate those patterns. Amazon’s hiring tool is the canonical example: the training data reflected decades of gender imbalance in tech hiring, so the AI learned that maleness predicted hiring success.
Historical bias is particularly insidious because it can make discriminatory outcomes appear objective. “The algorithm says so” becomes a defence for perpetuating the exact patterns that equality legislation was designed to eliminate.
Representation bias
When the training data does not accurately represent the population the AI will serve, the system performs poorly for underrepresented groups. Facial recognition systems trained predominantly on lighter-skinned faces perform significantly worse on darker-skinned faces. A landmark 2018 MIT study by Joy Buolamwini found that commercial facial recognition systems had error rates of 0.8% for lighter-skinned males but 34.7% for darker-skinned females (Gender Shades project).
In the workplace, representation bias affects AI systems used for video interview analysis, employee engagement scoring, and performance assessment tools — any system that was trained on data that over-represents certain demographic groups.
34.7%
error rate for darker-skinned females in commercial facial recognition, versus 0.8% for lighter-skinned males
Source : MIT Gender Shades Project, Buolamwini & Gebru, 2018
Proxy discrimination
Even when protected characteristics (gender, ethnicity, age, disability) are explicitly excluded from AI inputs, the system can discriminate through proxy variables — data points that correlate with protected characteristics. Postcode correlates with ethnicity. University attended correlates with socioeconomic background. Gaps in employment history correlate with gender (maternity leave). Typing speed in online assessments correlates with age and disability.
The Apple Card controversy illustrates this. In 2019, multiple users reported that Apple’s credit card, underwritten by Goldman Sachs, offered women significantly lower credit limits than their husbands — even when the women had higher credit scores and identical financial profiles. Goldman Sachs stated that gender was not an input variable. But the algorithm’s other inputs apparently contained proxies that produced gendered outcomes.
Measurement bias
This occurs when the metrics used to train or evaluate AI systems are themselves biased. If an AI performance review tool is trained to identify “high performers” based on metrics that favour certain working styles (e.g., hours logged, email volume, visible participation in meetings), it will systematically undervalue employees who deliver results through different approaches — often disproportionately affecting neurodiverse employees, introverts, part-time workers, and those with caring responsibilities.
Feedback loop bias
AI systems that influence the decisions they learn from create self-reinforcing bias cycles. A hiring AI that screens out candidates from certain backgrounds means those candidates never get hired, which means the training data continues to show those backgrounds as unsuccessful, which reinforces the original bias. The system becomes more biased over time, not less.
Bias is not just an ethical problem — it’s a legal one. The UK Equality Act 2010 prohibits direct and indirect discrimination in employment. The EU AI Act classifies AI used in employment as high-risk with mandatory bias testing. Under GDPR, individuals have the right not to be subject to decisions based solely on automated processing that significantly affects them, and to receive meaningful information about the logic involved. Organisations deploying biased AI systems face regulatory action, litigation, and substantial financial penalties.
Real-world examples that shaped the debate
Amazon’s recruiting AI (2018)
Amazon’s CV screening tool was trained on historical hiring data and learned to discriminate against women. The system downgraded CVs containing words associated with women and penalised candidates from women’s colleges. Amazon disbanded the team and scrapped the tool, but the case demonstrated that even the world’s most sophisticated technology companies could build discriminatory AI without intending to.
Apple Card and Goldman Sachs (2019)
Apple Card users reported that the algorithm gave women significantly lower credit limits than their male partners, even with equivalent or superior financial profiles. The case prompted an investigation by the New York Department of Financial Services. Goldman Sachs stated that gender was not an input — highlighting the proxy discrimination problem.
UK Home Office visa algorithm (2020)
The Home Office used an AI system to stream visa applications into fast-track and complex-track categories. An investigation by the Joint Council for the Welfare of Immigrants revealed that the algorithm used nationality as a factor, effectively creating a “hostile environment” for applicants from certain countries. The system was scrapped after legal challenge.
Google Photos labelling (2015)
Google Photos’ image recognition AI labelled photographs of Black people as “gorillas.” The incident revealed catastrophic representation bias in the training data. Google’s initial fix was simply to remove the “gorilla” label from the system entirely — a workaround, not a solution. As of 2024, Google had still not fully resolved the underlying classification issue.
HireVue video interview analysis
HireVue’s AI-powered video interview platform, which analysed candidates’ facial expressions, tone of voice, and word choice to predict job performance, faced sustained criticism for potential bias against candidates with disabilities, non-native English speakers, and people from different cultural backgrounds. In 2021, HireVue discontinued the facial analysis component following an FTC complaint and academic criticism.
44%
of HR leaders report concerns about bias in their organisation's AI hiring tools
Source : SHRM AI in the Workplace Survey, 2025
How to detect AI bias
Detection requires systematic effort. You cannot identify bias by looking at overall accuracy metrics alone — a system can be 95% accurate overall while being 60% accurate for a specific demographic group.
Disaggregated performance analysis
Evaluate your AI system’s performance separately for each protected characteristic group. Does the hiring AI recommend the same proportion of male and female candidates? Does the credit scoring system approve applications at similar rates across ethnic groups? Do performance prediction tools produce comparable accuracy across age groups?
This analysis requires collecting demographic data — which raises its own privacy considerations. The GDPR framework permits processing sensitive personal data for equality monitoring purposes, but appropriate safeguards must be in place.
Fairness metrics
Several mathematical definitions of fairness can be applied to AI systems:
- Demographic parity: the system produces positive outcomes at equal rates across groups
- Equal opportunity: the system has equal true positive rates across groups
- Predictive parity: positive predictions are equally accurate across groups
- Individual fairness: similar individuals receive similar outcomes
These definitions sometimes conflict — achieving perfect demographic parity may require sacrificing predictive accuracy, and vice versa. Organisations must decide which fairness criteria are most appropriate for their specific context and legal obligations.
Regular auditing
Bias testing is not a one-time exercise. AI systems can develop new biases as their training data evolves, as the population they serve changes, or as feedback loops amplify initial small biases over time. Establish a regular auditing schedule — quarterly at minimum for high-risk systems — and document results. Our AI risk assessment guide provides a framework for this process.
The EU AI Act requires providers of high-risk AI systems (including employment AI) to implement bias testing as part of their risk management system, maintain documentation of testing results, and ensure ongoing monitoring post-deployment. The AI governance framework your organisation establishes must include bias as a standing agenda item.
How to prevent AI bias
Diverse and representative training data
The most direct way to reduce bias is to ensure training data is diverse and representative. This means auditing training datasets for demographic representation, supplementing underrepresented groups, and removing or rebalancing data that encodes historical discrimination.
Bias-aware model development
Technical approaches include adversarial debiasing (training models to be unable to predict protected characteristics from other features), fairness constraints during training (penalising the model for disparate outcomes), and post-processing calibration (adjusting outputs to achieve fairness targets).
Human oversight
No AI system making consequential decisions about people should operate without meaningful human oversight. “Meaningful” means the human reviewer has the training, information, authority, and time to genuinely evaluate the AI’s recommendation — not just rubber-stamp it. This is not just good practice; it is a legal requirement under the EU AI Act for high-risk systems and under GDPR for automated decision-making.
Transparent AI policies
Develop and publish clear AI policies that address bias explicitly. Employees, candidates, and customers should understand when AI is being used in decisions that affect them, what safeguards are in place, and how they can challenge outcomes.
Organisational diversity
Diverse teams build less biased AI. Research consistently shows that homogeneous teams are less likely to identify bias in AI systems because they share blind spots. Ensure that AI development, procurement, and governance teams include diverse perspectives — not just demographic diversity but diversity of experience, discipline, and viewpoint.
Test your understanding of AI bias
Building a bias-aware workforce
AI bias prevention starts with awareness. Every employee who interacts with AI systems — from the developers building them to the managers relying on their outputs — needs to understand how bias manifests, why it matters, and what their role is in preventing it.
Brain delivers AI training that builds practical bias awareness across your organisation. Interactive scenarios based on real-world cases. Role-specific content for HR, procurement, compliance, and leadership teams. Training on EU AI Act requirements for high-risk AI systems. Compliance documentation that demonstrates your organisation’s commitment to fair and responsible AI use.
Related articles
AI Risk Assessment: Free Template + Scoring Matrix (2026)
Conduct an AI risk assessment with our free template. Scoring matrix, 4 risk categories, and step-by-step methodology aligned with EU AI Act.
What Is Shadow AI? 5 Risks + How to Manage It (2026)
Shadow AI is unauthorised AI use by employees. Discover why it's dangerous and get a practical framework to manage it effectively.
AI Hallucinations: 12 Real Examples + How to Detect Them
Why does AI make things up? 12 real-world hallucination examples, detection methods, enterprise risks and proven mitigation strategies.