What is bias and fairness in machine learning, and how do you measure it?

In this article

1. Before you start
2. Why interviewers ask this
3. Bias and fairness in machine learning from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one part of large language model (LLM) evaluation, because a model that scores well overall can still fail a group.

Why interviewers ask this

Fairness questions test whether a candidate can turn a vague worry, "the model is biased", into a measurement and a decision. Candidates report prompts such as discussing demographic parity and disparate impact in a hiring recommender, explaining why a search for a gendered term returns mostly one gender, and describing how to run responsible AI in a deployed system.

A weak answer lists metric names. A strong answer names the groups and the harm first, computes rates per group, and admits that the metrics conflict. A Staff-level answer adds who decides which metric wins, what the fix costs in accuracy, and how the system is monitored after launch, because the world and the data keep moving.

Bias and fairness in machine learning from first principles

Demographic parity

Demographic parity is a fairness criterion that requires a model to give the positive decision, such as an interview invitation, at the same rate to every group. In Hardt, Price and Srebro's notation, the chance of a positive prediction given group 0 equals the chance given group 1. It ignores the true outcome.

Equalized odds

Equalized odds is a fairness criterion that requires a model to have the same true positive rate and the same false positive rate in every group. It conditions on the true outcome. The weaker equal opportunity criterion asks only for equal true positive rates.

Disparate impact ratio

The disparate impact ratio is the selection rate of one group divided by the selection rate of the group with the highest rate. A ratio far below one means the model selects that group much less often.

In plain words

Imagine a school exam marked by a machine. If it marks one class's papers wrongly more often than another's, the average score hides the problem. Fairness work asks the same thing of a model: do the errors fall evenly on everyone? The analogy breaks in one place. An exam has one right answer, but "fair" has several mathematical meanings, and they can disagree.

In one precise sentence, algorithmic bias is a systematic difference in a model's decisions or errors across groups that harms one of them, and fairness is the property you check with group-level statistics. Hardt, Price and Srebro (2016) propose a criterion for discrimination against a specified sensitive attribute in supervised learning. The criterion depends only on the joint statistics of the predictor, the target and the protected attribute.

Where does bias in a machine learning model come from?

Bias in a machine learning model comes from the data, the labels, the features and the way the model is used, not only from the algorithm. Suresh and Guttag (2019) describe sources of harm across the whole machine learning life cycle, and five are worth knowing. Historical bias means past decisions, such as who was hired before, are used as labels and teach the model old patterns.

Representation bias, also called sampling bias, means a group is under-represented in the training data, such as fewer low-light or darker-skin images. Measurement bias means a proxy target or feature measures groups differently, such as arrests standing in for crime. Aggregation bias means one model is fitted to groups that behave differently. Feedback loops after deployment mean the model's decisions shape the data it sees next. Each source needs a different check, so name the likely one before you pick a fix.

Step 1: Name the groups and the harm

Name the groups and the harm is the first move, because a fairness metric is meaningless until you know who could be hurt and how. A group is defined by a protected attribute, such as gender or age. A harm is either an allocation harm, where a group gets fewer opportunities such as interviews or loans, or a quality-of-service harm, where the model simply works worse for them, such as a face or speech model with higher error for one group.

Input: the product's purpose and the people it touches. Output: a list of groups, the decision the model makes about them, and the cost of each kind of error. Rejecting a qualified applicant and approving an unqualified one have different costs, and that cost decides later which rate you protect. You now know what to measure; the next step measures it.

Step 2: Measure rates for every group

Measure rates for every group by splitting the evaluation set by group and computing the same numbers in each. The core numbers are the selection rate (the share who receive the positive decision), the true positive rate (TPR), the share of truly positive people the model catches, the false positive rate (FPR), the share of truly negative people it wrongly accepts, and precision (the share of accepted people who are truly positive).

Input: predictions, true labels and group labels. Output: one row of rates per group. Alongside these, record each group's base rate. If the base rates differ, the groups are not the same population, and that single fact is what makes several fairness goals impossible together; the next step shows why. Slicing by group is also what an overall accuracy score hides, so a model can look excellent in aggregate and fail a subgroup.

Step 3: Choose a fairness criterion, and accept the conflict

Choose a fairness criterion by deciding which rate you will equalize, because the common ones cannot all hold for any useful (non-trivial) classifier when base rates differ. Demographic parity equalizes selection rates. Equal opportunity equalizes TPR. Equalized odds equalizes TPR and FPR. Calibration asks that a given score mean the same probability in every group. Kleinberg, Mullainathan and Raghavan (2016) show that the calibration and balance conditions cannot all hold, except when base rates are equal or prediction is perfect. Balance means the truly positive, and separately the truly negative, get the same average score in every group, a score version of equalized odds.

Input: the costs from step 1 and the rates from step 2. Output: one chosen criterion and a written reason. In hiring, a team may care most about equal TPR, so equally qualified people are found at equal rates. A legal screen differs: the four-fifths rule in 29 Code of Federal Regulations (CFR) 1607.4 compares selection rates against a ratio of 0.8 and is a screening flag for review, not a causal test.

Step 4: Mitigate, then monitor

Mitigate and monitor means fixing the gap at one of three stages and then watching it after launch. Pre-processing changes the data, for example by rebalancing or relabeling. In-processing adds a fairness term or constraint to training. Post-processing adjusts the decision thresholds per group; Hardt, Price and Srebro show that a learned predictor can be adjusted after training to satisfy their criterion, and that this shifts the cost of poor classification from disadvantaged groups to the decision maker.

Input: the chosen criterion. Output: a changed model and a dashboard of per-group rates. Every fix has a cost, usually in overall accuracy or in another rate, so record it. Monitoring matters because a model's decisions change who appears in future data, and a gap that closed at launch can reopen.

1 · Groups and harm2 · Measure3 · Criterion4 · Fix and watchReplay

Step-by-step description

1. Name who could be hurt and how.
2. Compute the same rates for every group.
3. Choose which rate to equalize.
4. Mitigate, then monitor by group.

• Name the groups and the harm first.
• Measure rates per group before choosing a fairness criterion.
• Mitigate and monitor after the criterion is chosen.

Worked example

A hiring screen scores 200 applicants from group A and 200 from group B. The counts are illustrative: 100 of group A and 60 of group B are truly qualified, so the base rates are 0.5 and 0.3.¹ The model selects 80 qualified and 20 unqualified applicants from A, and 36 qualified and 24 unqualified from B.¹

Group A is selected at a rate of 0.5, group B at 0.3. The demographic parity difference is 0.2, and the disparate impact ratio is 0.3 / 0.5 = 0.6, well under the 0.8 screening ratio.¹ The TPR is 0.8 for A and 0.6 for B, and the FPR is 0.2 for A and 0.1714 for B. Precision is 0.8 for A and 0.6 for B.¹

Lowering group B's threshold until its TPR reaches 0.8 selects 48 qualified and 36 unqualified applicants, a selection rate of 0.42 and a ratio of 0.84.¹ The cost: B's FPR rises to 0.2571 against 0.2 for A, and B's precision falls to 0.5714.¹ Fixing one gap opened another, which is the conflict of step 3 in numbers.

Text description of the diagram

• Group A is selected at a rate of 0.5 and group B at 0.3.
• The true positive rate is 0.8 for A and 0.6 for B.
• Precision is 0.8 for A and 0.6 for B.

Trade-offs and edge cases

Fairness is a trade-off, not a switch. Each criterion compares a different rate, so choosing one is choosing which error to equalize. Demographic parity ignores the true label, so it can demand selecting unqualified people to hit equal rates when base rates differ. Equalized odds uses the label, but it needs trustworthy labels, and labels such as past hiring decisions can themselves carry the bias.

Text description of the diagram

• Demographic parity compares selection rates and ignores the true label.
• Equal opportunity compares true positive rates.
• Equalized odds compares true positive and false positive rates.

Edge cases follow. Removing the protected attribute does not make a model fair, because other features such as postal code can stand in for it. Small groups give noisy rates, so report counts beside every rate. People belong to several groups at once, so a model fair for each single attribute can still fail an intersection.

Large language models add more cases: a transformer-based model is trained on human-written text, which can carry social bias that the model may reproduce, and a judge model scoring outputs can add its own bias, as the page on LLM-as-a-judge explains. Retrieval systems can skew results by group too, so evaluation tools such as those in the Ragas guide should be run on slices. Finally, measuring by group needs the sensitive attribute, and collecting it carries its own risks, which the privacy in machine learning systems page covers.

How to answer it in an interview

The spoken structure: define bias, measure per group, name the conflict, pick a metric with a reason, then mitigate and monitor.

Common mistakes

• "I removed gender from the features, so the model is fair." (wrong) Other features can carry the same information. (right) "I removed it from the inputs but still measure outcomes by gender, because proxies remain."
• "We will satisfy every fairness metric." (wrong) With different base rates they conflict. (right) "I will pick the metric that matches the harm and state what it costs."
• "Overall accuracy is high, so the model is fine." (wrong) The average hides subgroup failure. (right) "I report every rate per group, with counts."
• "Demographic parity and equal opportunity are the same." (wrong) The first ignores the true label; the second equalizes TPR among the truly positive. (right) "They compare different rates and can disagree."

Follow-up questions

How would you discuss demographic parity and disparate impact in a hiring recommender?

Demographic parity asks for equal selection rates across groups, and the disparate impact ratio is the lowest group's rate divided by the highest group's. In the worked example the ratio is 0.6, which would be flagged against the 0.8 screening ratio; I would then check TPR to see whether qualified people are being missed or the base rates differ, since the ratio alone is not a causal test.

Does removing the protected attribute make a model fair?

No. Features correlated with the attribute, such as location, let the model recover group information, so you keep the attribute in the evaluation data to measure outcomes by group even if it is not a model input.

Why might an image search for a gendered term return mostly one gender?

If the training images are heavily skewed, a model that copies the data reproduces the skew. A skew of 50 male images to 1 female image, as reported by a candidate and used here as an illustration, means about 0.98 of a proportional result would be male.¹ Fixes include rebalancing, reweighting and re-ranking results for diversity.

Why might a vision model perform worse on some image qualities and demographics?

A vision model can work worse on low-light, blurry or low-resolution images, and those images are not spread evenly across demographic groups, so a quality gap becomes a group gap. This is a quality-of-service harm. Slice the evaluation set by image quality and by group, including the intersection, report counts beside each rate, and add training data for the weak slices.

What are the risks of assuming LLMs think or feel like humans?

A strong answer says that people then over-trust fluent output, read intent into errors, and skip testing across groups and contexts. Treat the model as a statistical system whose behavior you measure, because fluent text is not evidence that it understands or is fair.

How do you operationalize responsible AI in a deployed system?

Define harms and owners before launch, evaluate per-group metrics in the release checklist, log decisions so they can be audited, set alert thresholds on group gaps, and give users a way to appeal. The aim is to make the four steps above routine rather than a one-off review.

Sources

1. Equality of Opportunity in Supervised Learning (arXiv 1610.02413) — arXiv (Hardt, Price, Srebro), 7 Oct 2016
2. Inherent Trade-Offs in the Fair Determination of Risk Scores (arXiv 1609.05807) — arXiv (Kleinberg, Mullainathan, Raghavan), 17 Nov 2016
3. A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle (arXiv 1901.10002) — arXiv (Suresh and Guttag), 24 Jan 2019
4. 29 CFR 1607.4: Information on impact (Uniform Guidelines on Employee Selection Procedures) — eCFR (United States government), 26 Oct 2026