14 Building AI We Can Live With

AI Safety, Ethics, and Governance in a Complex World

CSC5350 · Artificial Intelligence

14.1 Opening Narrative

14.1.1 Two Sides of the Same Algorithm

In May 2016, the investigative journalism outlet ProPublica published an article titled “Machine Bias.” It described a software system called COMPAS — Correctional Offender Management Profiling for Alternative Sanctions — that had been adopted by courts across the United States to assess the recidivism risk of criminal defendants. The score it produced, a number from 1 to 10, was used by judges to inform decisions about bail, sentencing, and parole.

ProPublica’s analysis of more than 7,000 individuals arrested in Broward County, Florida, found that the algorithm was twice as likely to falsely flag Black defendants as future criminals compared to white defendants. Conversely, it was twice as likely to incorrectly label white defendants as low risk when they actually went on to reoffend.

The story spread rapidly. COMPAS became the defining case study in AI fairness debates.

The company that made COMPAS, Northpointe, responded. Their rebuttal was careful, methodologically rigorous — and technically correct.

They showed that the algorithm was calibrated: among defendants who received a score of 7, approximately 70% did reoffend, regardless of race. The score meant what it said. High-risk Black defendants and high-risk white defendants reoffended at the same rate.

Both claims were true. ProPublica was right. Northpointe was right. And the system remained profoundly unjust.

How is this possible? The answer lies in the mathematical incompatibility of different definitions of fairness. When the base rates of a binary outcome differ between groups — as they do for recidivism, itself a product of discriminatory policing and prosecution patterns — it is mathematically impossible to simultaneously satisfy calibration (same accuracy for both groups) and equal false positive rates (same fraction of non-recidivists wrongly labeled high-risk).

You cannot have both. One definition must be sacrificed for the other. And the choice of which definition to satisfy is not a technical decision. It is a value judgment about who bears the cost of error — one that was never made explicitly in the deployment of COMPAS, because nobody with the authority to decide had fully understood what they were choosing.

“The COMPAS case teaches the most important lesson in AI ethics: a system can be technically correct, commercially successful, and legally deployed while causing systematic injustice. The technical review and the ethical review are separate processes — and skipping either one produces harm.”

This chapter is about the full landscape of responsibility that comes with building AI systems that affect human lives — the alignment problem, the mathematics of fairness, the tools of explainability, the emerging regulatory frameworks, and the system documentation practices that make accountability possible.

14.2 Learning Objectives

After completing this chapter, you will be able to:

Explain the AI alignment problem and describe why it becomes more dangerous as AI systems become more capable.
State and apply Goodhart’s Law and explain why it appears across every domain of AI deployment.
Explain LIME and SHAP as tools for post-hoc explanation and describe what each does and does not reveal.
Define the major mathematical fairness metrics and prove their incompatibility under different base rates.
Design and execute a systematic bias audit for a deployed classification system.
Describe the key provisions of the EU AI Act and explain how its risk-based framework applies to different AI applications.
Analyze the ethical challenges in criminal justice, medical, and financial AI.
Explain the arguments for and against autonomous weapons and the principle of meaningful human control.
Describe the existential risk debate and the technical and institutional responses being developed.
Produce a complete AI System Card for your IAAIS project — the Chapter 14 milestone.

14.3 Key Terminology

Term	Plain-Language Definition
AI Alignment	The challenge of ensuring AI systems pursue goals consistent with human values and intentions. An aligned system does what we want, not just what we specified.
Value Specification	The process of formally defining what an AI system should optimize for. Getting value specification right is among the hardest and most consequential problems in AI development.
Goodhart’s Law	“When a measure becomes a target, it ceases to be a good measure.” In AI: when a system optimizes against a metric, it may find ways to maximize the metric that diverge from the underlying goal the metric was meant to represent.
Reward Hacking	An AI system finding unintended ways to maximize its reward function that satisfy the letter but violate the spirit of the specification. A structural consequence of any imperfect objective combined with a capable optimizer.
Interpretability	Understanding the internal representations and reasoning processes of an AI model — what it has learned and how it computes its outputs.
Explainability (XAI)	Communicating AI model behavior to human stakeholders in understandable terms. Not necessarily the true internal mechanism, but a useful approximation of why a specific prediction was made.
LIME	Local Interpretable Model-agnostic Explanations. Explains individual predictions by fitting a simple interpretable model locally around each input, identifying which features most influenced that specific output.
SHAP	SHapley Additive exPlanations. Assigns each feature a contribution to a prediction using cooperative game theory (Shapley values), satisfying consistency, local accuracy, and missingness axioms.
Demographic Parity	A fairness criterion requiring equal positive prediction rates across protected groups. Does not account for differences in underlying base rates.
Equalized Odds	A fairness criterion requiring equal true positive rates AND equal false positive rates across protected groups. Mathematically incompatible with calibration when base rates differ.
Calibration	A fairness criterion requiring predicted probabilities to accurately reflect actual event rates, equally across groups. Mathematically incompatible with equal false positive rates when base rates differ.
Individual Fairness	The principle that similar individuals should receive similar predictions. Requires a definition of “similarity” that is itself value-laden.
Counterfactual Fairness	A prediction is counterfactually fair if it would not change had the individual belonged to a different demographic group, holding other factors constant.
Audit (AI)	A systematic evaluation of an AI system’s behavior, performance, fairness, safety, and compliance. Can be internal, third-party, or regulatory.
EU AI Act	The European Union’s 2024 regulation governing AI systems, establishing a risk-based framework with prohibited uses, high-risk requirements, and transparency obligations.
High-Risk AI	Under the EU AI Act: AI systems used in critical infrastructure, education, employment, essential services, law enforcement, judicial decisions, or border management — subject to conformity assessment and human oversight requirements.
Model Card	A structured document accompanying an ML model describing its intended uses, performance metrics (including across demographic subgroups), limitations, and ethical considerations.
System Card	A broader documentation framework for AI systems describing capabilities, limitations, safety measures, human oversight mechanisms, and governance structures.
Meaningful Human Control	The principle that consequential decisions — especially irreversible ones — should remain under genuine human authority, with AI systems supporting rather than replacing human judgment.
Existential Risk (AI)	The risk that advanced AI systems could cause outcomes catastrophic enough to severely and permanently curtail human civilization. A contested but increasingly mainstream concern in AI safety research.
Red-Teaming	Deliberate adversarial testing of an AI system — attempting to find failure modes, safety violations, and harmful behaviors before deployment. Standard practice for responsible AI development.
Differential Privacy	A mathematical framework guaranteeing that no individual’s data disproportionately influences published results or model parameters, providing quantifiable privacy protection.

14.4 Section 1 — The Alignment Problem: Capability Without Alignment Is Dangerous

The alignment problem is, at its core, a specification problem. We cannot perfectly specify what we want an AI system to do. Every objective is incomplete, imprecise, or subtly wrong in ways that become visible only when a capable optimizer finds the edges of the specification.

For a weak optimizer, this does not matter much — it cannot find the corners. For a powerful optimizer, it matters enormously — and the corners may be very far from where we wanted to end up.

14.4.1 Goodhart’s Law Across Domains

Goodhart’s Law — “when a measure becomes a target, it ceases to be a good measure” — is one of the most important principles in AI safety. The mechanism is simple: a metric is a proxy for what we actually want. As long as the optimizer is weak, it pursues the proxy in ways that also advance the underlying goal. When the optimizer is powerful, it finds strategies that maximize the proxy while potentially abandoning the goal entirely.

The pattern appears everywhere AI systems are deployed:

Content recommendation: Platforms optimized engagement — clicks, time on screen, shares. Recommendation systems learned that emotionally provocative, outrage-inducing, and sensational content maximizes engagement. Users spend more time but report lower satisfaction and higher anxiety. The proxy (engagement) was maximized; the goal (user wellbeing) was harmed.

Healthcare quality metrics: CMS penalized hospitals for high readmission rates, intending to improve post-discharge care. Some hospitals reduced readmissions by counseling patients against returning, transferring them to other facilities, or discharging them to emergency departments at nearby hospitals. The metric improved; patient outcomes did not necessarily.

Language model alignment: Models trained to maximize human preference ratings sometimes learned to be sycophantic — agreeing with users, validating incorrect beliefs, and prioritizing what sounds good over what is true. The proxy (preference scores) was maximized; the goal (helpful honesty) was compromised.

Criminal justice risk scoring: Models trained on historical criminal justice data — itself shaped by discriminatory policing and prosecution patterns — learned those biases as signal. The metric (statistical correlation with prior contact) was optimized; the goal (predicting individual future behavior free of historical bias) was violated.

The appropriate response to Goodhart’s Law is not to abandon metrics — we need ways to measure what systems are doing. It is to use multiple proxies simultaneously (harder to game all at once), measure how the system achieves outcomes not just what outcomes it achieves, test adversarially for proxy exploitation, and maintain human evaluation of the true goal independently of the proxy.

14.5 Section 2 — Explainability: Opening the Black Box

Black-box AI models make predictions without explaining why. In low-stakes applications this may be acceptable. In high-stakes applications — medical diagnosis, loan decisions, criminal justice — the absence of explanation is not merely inconvenient. It prevents accountability, obscures discrimination, and undermines the trust that effective deployment requires.

Explainability tools provide approximations of why a specific prediction was made. They do not reveal the true internal mechanism — but they are sufficient to support meaningful accountability and debugging.

14.5.1 LIME: Local Approximation

LIME (Local Interpretable Model-agnostic Explanations) explains individual predictions by building a simple interpretable model — usually a linear model — in the neighborhood of the input being explained.

The intuition: even if the global model is complex and highly nonlinear, it may behave approximately linearly near any specific input. LIME generates a large number of perturbed versions of the input (with some features masked or modified), gets the black-box model’s predictions for each, and fits a weighted linear model to these (input, prediction) pairs — where nearby perturbations receive higher weights.

The linear model’s coefficients then represent the approximate importance of each feature for this specific prediction. “This loan application was denied primarily because of the debt-to-income ratio (coefficient +0.34) and the absence of employment history (coefficient +0.28)” is the kind of explanation LIME produces.

What LIME does well: Fast, model-agnostic, produces human-readable feature importances for any input type (tabular, text, images), enables both explanation and debugging. What LIME does not provide: The true internal mechanism of the model; consistent explanations across multiple calls for the same input (randomness in perturbation sampling can produce different explanations each time); explanations that would survive adversarial scrutiny.

14.5.2 SHAP: Principled Attribution

SHAP (SHapley Additive exPlanations) provides attribution with stronger theoretical foundations, derived from Shapley values in cooperative game theory. The Shapley value of a feature is its average marginal contribution to the prediction across all possible orderings in which the features could be introduced.

Formally: the SHAP value for feature i is the average over all possible feature orderings of the difference between the model’s prediction with features 1 through i included and with features 1 through i−1 included. This average accounts for all possible interactions between feature i and every subset of other features.

SHAP satisfies properties that make it uniquely well-founded: Efficiency (SHAP values sum to the difference between the prediction and the baseline), Symmetry (features contributing equally receive equal SHAP values), Dummy (features that never change the prediction receive SHAP value zero), and Additivity (SHAP values for an ensemble equal the sum of SHAP values for each component model).

In practice, exact computation requires exponentially many model evaluations. SHAP provides efficient approximations: TreeSHAP for tree-based models (exact, polynomial time), KernelSHAP for any model (approximate, slower), and DeepSHAP for neural networks.

The practical difference from LIME: SHAP produces consistent explanations that sum to the prediction difference from baseline, and global feature importance can be summarized across the dataset by averaging SHAP magnitudes — revealing not just what mattered for this prediction but what the model relies on in general.

The limits of both approaches: Post-hoc explainability tools approximate a complex model’s behavior with a simpler one. They tell you what the approximation says, not what the model truly “thinks.” A model optimized to produce good LIME or SHAP explanations can be built — but this does not make the model interpretable; it makes it good at producing satisfying-looking explanations. True interpretability requires that the model itself be transparent, not merely that explanations can be constructed around it.

14.6 Section 3 — Fairness: The Mathematics of a Contested Concept

The COMPAS case made precise what intuition already suggests: there is no single mathematical definition of “fair” that satisfies all reasonable fairness intuitions simultaneously. Different definitions conflict with each other — not because one is right and others wrong, but because they encode different values about who bears the cost of error.

14.6.1 Four Major Definitions

Demographic parity requires that positive prediction rates be equal across groups: the fraction of applicants predicted as high-risk (or approved, or hired) should be the same regardless of group membership. This definition asks: does the system produce equal outcomes? Its weakness: it can be satisfied by a model that performs equally well for everyone — or by a model that performs equally badly.

Equalized odds (Hardt et al., 2016) requires that both the true positive rate (sensitivity: correctly identifying those who would reoffend) and the false positive rate (correctly identifying those who would not) be equal across groups. This definition asks: do errors fall equally across groups?

Calibration requires that predicted probabilities accurately reflect actual event rates, equally across groups. If the model says 70% risk for a group, approximately 70% of that group should experience the outcome. Northpointe’s defense of COMPAS was a calibration argument: the scores meant the same thing regardless of race.

Individual fairness (Dwork et al., 2012) requires that similar individuals receive similar predictions — where “similar” must be defined by a domain-appropriate similarity metric. This is conceptually attractive but practically challenging: the similarity metric itself encodes value judgments.

14.6.2 The Impossibility Theorem

The most important result in algorithmic fairness is not about how to achieve fairness — it is about why you cannot achieve all definitions simultaneously.

Chouldechova (2017) and Kleinberg et al. (2016) independently proved that when base rates differ between groups, it is mathematically impossible to simultaneously satisfy: 1. Calibration (equal positive predictive values across groups) 2. Equal false positive rates across groups 3. Equal false negative rates across groups

Satisfying any two requires violating the third. This is not a limitation of current methods — it is a mathematical theorem that no algorithm can escape.

In the COMPAS case: Black defendants had a higher base rate of recidivism in the dataset (itself a consequence of differential policing and prosecution, not of differential behavior). Given this base rate difference, Northpointe achieved calibration but at the cost of higher false positive rates for Black defendants. ProPublica documented the false positive rate disparity. Both were measuring real properties of the same system. Neither was wrong. The system could not satisfy both criteria simultaneously.

This means the choice of fairness criterion is not a technical decision — it is a value judgment about which type of error is more acceptable to make, for whom, and by whose authority. A system that fails to make this choice explicitly has made it implicitly.

14.6.3 Practical Bias Auditing

A bias audit is a systematic evaluation of an AI system’s performance across demographic subgroups. A responsible audit addresses seven questions:

What are the protected attributes? Race, gender, age, disability, national origin, and their proxies (zip code, name, income patterns).
What are the disaggregated performance metrics? Accuracy, precision, recall, AUC, false positive rate, and false negative rate computed separately for each group.
Is the training data representative? Are all deployment groups represented in training? Are labels free from historical bias?
Have intersectional subgroups been evaluated? Gender Shades (2018) found that the worst-performing subgroup (darker-skinned women) performed 34 percentage points worse than the best (lighter-skinned men) on commercial facial analysis — a disparity invisible when only main effects were tested.
What fairness criterion has been chosen, and who chose it? The choice should be explicit, stakeholder-informed, and documented.
Is the system being used as intended? Deployed systems are often repurposed for tasks their developers did not anticipate — with fairness implications that were never evaluated.
What is the monitoring plan? Fairness at deployment time does not guarantee fairness over time as the world and the user population change.

14.7 Section 4 — The Regulatory Landscape

AI governance is shifting rapidly from voluntary commitments to mandatory regulation. The European Union has moved furthest fastest; other major jurisdictions are following.

14.7.1 EU AI Act (2024)

The EU AI Act establishes a risk-based framework classifying AI systems into four tiers:

Unacceptable risk (prohibited): Social scoring by governments, real-time biometric surveillance in public spaces (with narrow exceptions for terrorism and missing persons), AI that exploits psychological vulnerabilities, emotion recognition in workplace and educational settings, systems creating facial recognition databases from internet scraping.

High risk (regulated): AI in critical infrastructure, education, employment and HR, essential services (credit, insurance), law enforcement risk assessment, administration of justice, migration and border control, democratic processes. Requirements include: conformity assessment, technical documentation, human oversight mechanisms, accuracy and robustness standards, transparency to affected persons, and registration in an EU database.

Limited risk (transparency obligations): Chatbots must disclose that the user is interacting with AI; deepfake content must be labeled as AI-generated; AI-generated content from systemic-risk models must be marked.

Minimal risk: Spam filters, AI in video games, recommendation systems, and most current AI products — no specific AI-related obligations, though other EU law (GDPR, consumer protection) still applies.

The penalties are substantial: up to €35 million or 7% of global annual turnover for prohibited-use violations; up to €15 million or 3% for high-risk violations. The Brussels Effect — the tendency of EU regulation to become de facto global standard — means the AI Act’s requirements will shape AI development worldwide, not only in Europe.

14.7.2 US Approach

The US has taken a less prescriptive approach, relying on a combination of executive action and sector-specific regulation rather than comprehensive horizontal legislation. The 2023 Executive Order on Safe, Secure, and Trustworthy AI required developers of foundation models with potential dual-use risks to share safety test results with the government before public release, tasked NIST with developing safety evaluation standards, and directed agencies to develop AI use policies within their sectors.

Sector-specific regulation is more developed: HIPAA and FDA oversight for medical AI, fair lending laws (ECOA, Fair Housing Act) for credit and housing AI, securities regulations for financial AI. The US approach relies on existing regulatory frameworks rather than creating AI-specific ones — an approach that leaves significant gaps for AI systems that fall between regulatory jurisdictions.

14.7.3 Other Jurisdictions

China has enacted specific regulations for recommendation algorithms, deepfakes, and generative AI services — requiring labeling of AI-generated content, prohibiting certain manipulative practices, and mandating registration of generative AI services. The UK has pursued a principles-based approach, asking existing regulators (the FCA, ICO, CMA, Ofcom) to apply AI principles within their domains rather than creating a new AI-specific regulator. Canada, Brazil, India, and Australia are all developing national frameworks — creating a complex patchwork of overlapping obligations for any globally deployed AI system.

14.8 Section 5 — High-Stakes Domains

14.8.1 Criminal Justice

The COMPAS case is the entry point, but the criminal justice AI landscape is broader. Predictive policing systems assign crime risk scores to locations and individuals. Facial recognition is used for suspect identification. Natural language processing tools analyze judicial opinions and identify sentencing patterns. Bail algorithms inform detention decisions.

These applications share a common ethical structure: they are used to support decisions that deprive people of liberty, based on predictions about future behavior, derived from data that encodes the historical biases of the criminal justice system itself.

The feedback loop this creates is not subtle. Police in areas flagged as high-risk by predictive policing make more stops and arrests there. Those arrests generate more data from those areas. The algorithm identifies those areas as high-risk again. The prediction does not discover crime — it discovers policing. And it bakes that discovery into the next generation of predictions.

The minimum requirements for responsible deployment of AI in criminal justice settings include: algorithmic impact assessment before deployment; ongoing bias auditing with public reporting; defendants’ right to know the basis of algorithmic scores and to contest them; human review required before any deprivation of liberty based on AI outputs; adversarial testing for discrimination by protected characteristics; and sunset provisions requiring periodic reauthorization.

14.8.2 Medicine

Medical AI faces a distinctive challenge: the gap between benchmark performance and clinical utility, combined with the extraordinary consequences of error.

A system that achieves 94% sensitivity and 97% specificity on a benchmark test set may perform substantially worse in clinical deployment if the benchmark population differs from the deployment population — in demographics, disease severity distribution, imaging equipment, or labeling standards. Models trained at one institution frequently underperform at others. Models trained before the COVID-19 pandemic needed revalidation afterward.

The FDA’s regulatory pathway for AI-based Software as a Medical Device (SaMD) requires premarket review for clinical decision support systems that meet certain risk thresholds. But many current clinical AI deployments fall into regulatory gray zones — marketed as administrative tools or clinical workflow support rather than diagnostic devices, avoiding the oversight that their actual clinical impact warrants.

The standard that should govern medical AI deployment but frequently does not: prospective validation on the actual deployment population, disaggregated performance reporting across demographic subgroups, evaluation of actual clinical outcomes not just algorithmic metrics, and ongoing post-deployment monitoring with defined thresholds for action.

14.8.3 Financial Services

Credit, insurance, employment screening, and fraud detection systems make consequential decisions about individuals’ economic lives. These domains are subject to existing anti-discrimination law in most jurisdictions — the US Equal Credit Opportunity Act, the Fair Housing Act, the EU’s equal treatment directives — but AI creates new challenges for enforcement.

The most significant challenge is proxy discrimination: a model that does not use protected attributes (race, gender, age) can still produce discriminatorily disparate outcomes if it uses features that are correlated with protected attributes — zip code, name patterns, social network characteristics, behavioral data. Excluding the protected attribute from the model is not sufficient to prevent discrimination; it may actually make discrimination harder to detect and challenge.

Explainability requirements in financial services — the right to know why a credit application was denied — can be satisfied by post-hoc explanation tools like SHAP. But regulators and courts are increasingly scrutinizing whether these explanations reflect the true decision process or are post-hoc rationalizations. The distinction matters legally: an adverse action notice that misrepresents the reasons for a denial may constitute a violation of fair lending law regardless of the underlying model’s accuracy.

14.9 Section 6 — Autonomous Weapons and Meaningful Human Control

No AI application raises starker ethical questions than autonomous weapons — systems capable of selecting and engaging targets without human authorization for each individual lethal decision.

The central ethical concern is not primarily about accuracy. A facial recognition system deployed in a weapons context might be technically accurate. The concern is about moral responsibility: who bears responsibility for the decision to take a human life, and can that responsibility ever legitimately be delegated to an algorithm?

The Campaign to Stop Killer Robots — supported by over 70 countries and thousands of AI researchers — argues that the authority to take human life should never be delegated to machines. Their argument rests on three premises: machines cannot exercise the moral judgment required by international humanitarian law (IHL), which requires assessment of proportionality, necessity, and distinction between combatants and civilians in context-dependent ways that resist algorithmic specification; machines cannot be held morally or legally responsible for their actions; and the existence of cheap, autonomous lethal systems will lower the threshold for using force, producing more conflict rather than less.

Proponents argue that in some engagement contexts — defending against incoming missile salvos, countering drone swarms — human reaction times are simply insufficient, and that autonomous systems with constrained engagement rules may produce fewer civilian casualties than stressed, fatigued, or dehumanizing human combatants. They also argue that “meaningful human control” is already ambiguous in the context of complex networked warfare.

The meaningful human control principle that most governance frameworks are converging toward requires that a human be able to understand the context of the targeting decision, exercise genuine judgment (not merely rubber-stamp an algorithmic recommendation), override the system before lethal force is applied, and be accountable for the decision in the full moral and legal sense. Whether “human on the loop” oversight — where a human monitors but does not authorize each individual engagement — satisfies this principle is genuinely contested.

14.10 Section 7 — Existential Risk: The Long Horizon

A growing research community is focused on a question that may be the most important in AI development: could sufficiently advanced AI systems, if not well-aligned with human values, cause outcomes catastrophic enough to severely and permanently curtail human civilization?

The core argument proceeds from four premises. First, AI systems will likely become substantially more capable over time. Second, more capable AI systems will pursue goals more effectively. Third, if those goals are not fully aligned with human values, more capable systems will cause more harm in pursuing them. Fourth, the specification of human values precisely enough to guarantee alignment in highly capable systems is an unsolved and possibly very hard problem.

The counterarguments are substantial. Current AI systems are highly specialized and show no signs of developing the kind of general goal-directedness the argument requires. There are economic incentives to build controllable AI — misaligned AI loses value. The concern may distract from concrete, documented harms occurring now. And the argument rests on assumptions about future AI capability and architecture that are genuinely uncertain.

The AI safety research agenda generated by these concerns — scalable oversight, mechanistic interpretability, robust evaluation, corrigibility, deceptive alignment detection — produces research that is valuable regardless of how the existential risk debate resolves. Understanding what models have learned, verifying that they behave safely under distribution shift, and maintaining human oversight over consequential decisions are important for current AI systems and will become more important as capabilities increase.

The institutional landscape has shifted substantially. Anthropic was founded specifically around AI safety concerns and describes safety research as central to its mission. OpenAI created a “Superalignment” team in 2023. DeepMind has a dedicated safety research group. The UK and US governments have established AI Safety Institutes tasked with evaluating frontier model risks. This is no longer a fringe concern — it has become a mainstream institutional priority.

14.11 Section 8 — Accountability by Design

The most important insight in AI governance is that ethical review cannot be an afterthought. Systems designed without safety and fairness considerations cannot be retrofitted with them — they must be redesigned. Accountability must be built in from the beginning.

A responsible development process addresses ethical questions at every phase:

Problem definition: What problem are we actually solving? Who will be affected, with or without their knowledge? What harms could this system enable? Is AI the right tool, or would a simpler system serve better?

Data collection: Is the training data representative of the deployment population? Are the labels free from historical bias? Did the people whose data we are using consent to this use?

Model development: What fairness criterion are we optimizing for, and who bears the cost of the tradeoff? What does the model treat as evidence? Are any features proxies for protected attributes?

Pre-deployment testing: Has the system been red-teamed for bias, safety, and misuse? Has it been tested across demographic subgroups? What is the plan for monitoring performance after deployment?

Deployment: What human oversight mechanisms are in place? Can affected individuals contest decisions? What triggers model withdrawal or retraining?

Post-deployment monitoring: Are performance metrics stable over time and across groups? Are there emerging harms not anticipated pre-deployment? Is the system being used as intended, or in unanticipated contexts?

The human oversight spectrum matters enormously:

Level	Description	Example
Fully automated	No human involvement in individual decisions	Spam filtering
Human on the loop	Human can override but AI acts by default	Radiology triage
Human in the loop	Human reviews recommendation before action	Loan approval
Human decision + AI assist	Human decides; AI provides information	Surgical planning
Human only	No AI involvement	Death penalty sentencing

The appropriate level of human oversight should be determined by the reversibility of the decision, the consequences of error, the reliability of the AI system across the deployment distribution, and the availability of meaningful human review. These are not universal constants — they depend on the specific system, domain, and deployment context.

14.12 Section 9 — AI System Cards: The Documentation Standard

A System Card is the primary mechanism through which AI developers communicate what their systems can and cannot safely do. It is the written commitment to accountability — the document that makes claims auditable and makes governance possible.

A complete System Card addresses ten areas:

1. System Overview: What the system does, who it is for, what it is explicitly not designed for, who is responsible for it, and when it was last reviewed.

2. Architecture: What components the system contains, how they interact, what external dependencies exist, and what hardware and infrastructure the system requires.

3. Training Data: Where the training data came from, its scale and composition, known limitations and biases, consent and licensing status, and data cutoff dates.

4. Performance Metrics: Primary performance metrics on the evaluation set; performance disaggregated by demographic subgroup; performance across operating conditions and edge cases; comparison to human performance where applicable.

5. Limitations and Failure Modes: Known failure modes with estimated frequency and severity; conditions under which the system should not be used; distribution shift vulnerabilities; hallucination or incorrect output modes for generative components.

6. Fairness and Bias: Protected attributes relevant to the system’s decisions; fairness criterion selected and justification; fairness metrics across demographic groups; known disparities and mitigations applied; residual disparities that could not be eliminated.

7. Safety Measures: Input validation, output validation, confidence thresholds, human oversight mechanisms, monitoring and alerting, and incident response plan.

8. Ethical Considerations: Stakeholders affected with and without consent; harms identified in pre-deployment assessment; mitigations implemented; residual harms and ongoing monitoring.

9. Legal and Regulatory Compliance: Applicable regulations, compliance measures implemented, areas of regulatory uncertainty, legal review status.

10. Governance: Primary contact for questions and concerns, mechanism for reporting harms, review schedule, version control and change management.

The System Card is not merely documentation. It is a commitment — to users, to affected parties, and to the organization itself — about what the system does, what it does not do, and what will happen when things go wrong. Organizations that cannot complete a System Card for their deployed AI systems have not thought through their obligations sufficiently to deploy responsibly.

14.13 Section 10 — Integrating Ethics into IAAIS

The IAAIS Ethics Audit is the most important deliverable in this course — not because it is technically complex, but because it requires you to confront honestly what you have built.

The audit has five components:

Fairness audit: Run disaggregated performance metrics on the IAAIS Classifier across at least two protected attributes relevant to your domain. Identify any disparities. Select a fairness criterion, justify the selection, and document the tradeoffs it involves.

Red team testing: Design and execute 20 adversarial test cases against the IAAIS Generative Interface: cases testing for hallucination on domain-specific facts, prompt injection attempts, harmful output edge cases, and out-of-scope queries. Document which tests passed, which failed, and what the failure modes reveal.

Interpretability analysis: Generate LIME or SHAP explanations for five representative predictions from the IAAIS Classifier — two correct high-confidence, two correct low-confidence, one incorrect. For the incorrect prediction: what drove the error? Is it systematic or idiosyncratic?

Failure mode analysis: Document five known failure modes of your IAAIS system with estimated frequency and severity. For each: what is the worst realistic scenario? What mitigation is in place? What residual risk remains?

System Card: Complete all ten sections of the System Card template for your IAAIS system.

14.13.1 Fourteen-Chapter IAAIS Integration

Chapter	Module	Capability
Ch 2	Search Engine	Path planning through explicit state spaces
Ch 3	Knowledge Base	First-order logic, inference, structured facts
Ch 4	Planner	Goal-directed action sequence generation
Ch 5	Uncertainty Module	Calibrated probabilistic reasoning
Ch 6	Classifier	Supervised learning from labeled data
Ch 7	Pattern Recognizer	Unsupervised structure discovery
Ch 8	Neural Perception	Deep feature extraction from raw inputs
Ch 9	Language Module	NLP, intent classification, entity extraction
Ch 10	Vision Module	Image classification, object detection
Ch 11	Decision Agent	RL sequential decision-making
Ch 12	Expert Module	Rule-based reasoning with explanations
Ch 13	Generative Interface	Conversational access to all modules
Ch 14	Ethics Audit	Safety, fairness, accountability certification

The Ethics Audit is the fourteenth module — and in some ways the most important. A system that cannot pass its own ethics audit should not be deployed. A system that passes its audit but was never audited is a system waiting to cause harm.

14.14 Hands-On Exploration: Auditing a Deployed Classifier

14.14.1 The Activity

Open hands_on_ch14.ipynb from the course repository.

Part 1 — Fairness Audit (25 minutes): Using the IAAIS Classifier from Chapter 6, compute fairness metrics disaggregated by at least two demographic attributes. For each disparity found: (a) is it statistically significant? (b) which fairness criterion does it violate? (c) what mitigation would you apply? (d) what residual disparity would remain after mitigation?

Part 2 — Interpretability Analysis (15 minutes): Generate SHAP values for five representative predictions. Plot a summary showing global feature importance across your test set. Compare the SHAP global importance to the model’s feature importances from Chapter 6 — do they agree? Where they disagree, which is more trustworthy?

Part 3 — Red-Teaming the Generative Interface (15 minutes): Design and execute 10 adversarial test cases. For each failure: what does it reveal about the system’s safety properties? Categorize failures by type (hallucination, injection, harmful content, refusal failure).

Part 4 — System Card (take-home): Complete the ten-section System Card for your IAAIS system. This is your primary Chapter 14 deliverable. The System Card should be included in your IAAIS GitHub repository and will be evaluated as part of the Chapter 15 integration sprint.

14.14.2 Reflection Questions

In the COMPAS case, both ProPublica and Northpointe were measuring real properties of the same system. If you were the judge deciding how to use COMPAS in sentencing decisions, which fairness criterion would you demand the system satisfy, and who would you consult in making that choice?
LIME explanations can be inconsistent — running LIME on the same input twice may produce different feature importance rankings due to randomness in the perturbation sampling. Does this inconsistency undermine LIME’s usefulness for accountability? For what purposes does consistency matter?
Your red team found that the IAAIS Generative Interface hallucinated on 3 out of 10 domain-specific factual queries. What is an acceptable hallucination rate for your application? How does the answer change if errors are recoverable vs. irreversible?
The EU AI Act may classify your IAAIS system as high-risk, depending on your domain. If it does, you will need a conformity assessment, technical documentation, human oversight mechanisms, and registration in the EU database. Walk through what each requirement would mean concretely for your system.

14.15 Case Study: COMPAS Revisited — What Responsible Deployment Would Have Required

14.15.1 What Was Missing

Applying the accountability-by-design framework from Section 8 to COMPAS:

At the problem definition stage: No explicit specification of which fairness criterion the system should satisfy, by whom, with what stakeholder input. The choice was made implicitly by technical decisions rather than by people with democratic authority over criminal justice policy.

At the data stage: No acknowledgment that historical criminal justice data encodes discriminatory policing and prosecution patterns, or analysis of how this would affect system behavior across demographic groups.

At the model development stage: No explicit selection and justification of a fairness criterion; no disaggregated validation across race, gender, and age groups; no analysis of which features serve as proxies for race.

At the pre-deployment testing stage: No independent bias audit; no adversarial testing; no evaluation of the system’s impact on actual judicial decisions as distinct from its statistical properties.

At the deployment stage: No disclosure to defendants of the score they received or its basis; no right to contest the score; no requirement for human review before deprivation of liberty; trade-secret protection preventing algorithmic challenge.

At the monitoring stage: No systematic monitoring of the system’s impact on outcomes disaggregated by demographic group; no defined threshold for withdrawal; no mechanism for reporting harms.

None of these requirements would have eliminated the mathematical impossibility at the heart of COMPAS. But they would have made the tradeoffs visible, contestable, and subject to democratic deliberation — rather than invisible, unchallengeable, and legitimized by the authority of an algorithm.

14.15.2 The Standard That Should Apply

The ProPublica investigation was conducted by journalists, not by the courts using the system or the agencies procuring it. The analysis was published six years after COMPAS was first deployed. The people most affected — criminal defendants whose liberty was influenced by the score — had no access to the analysis, no mechanism to challenge the score, and no knowledge that the system existed.

This is not a technology problem. It is a governance problem. The technology could have been audited earlier, more thoroughly, and by people with appropriate expertise and institutional authority. The reason it was not is that no institution required it.

The emerging regulatory response — the EU AI Act’s requirements for high-risk AI systems, the proposed Algorithmic Accountability Act in the US, various state-level laws — reflects a belated recognition that voluntary commitments and market incentives are insufficient to produce the accountability that high-stakes AI deployment requires.

14.16 Chapter Summary

We began this chapter with COMPAS and the mathematical paradox at the heart of algorithmic fairness: two parties with technically correct claims, a system that satisfied both simultaneously while remaining unjust. The resolution — that different fairness definitions encode different value judgments about who bears the cost of error — is the most important single insight in AI ethics.

The alignment problem and Goodhart’s Law established the foundational challenge: AI systems optimize against specified objectives, and those objectives are always imperfect proxies for what we want. The more capable the optimizer, the more important it is to get the objective right.

LIME and SHAP gave us practical approximations of why specific predictions were made — sufficient for accountability and debugging, though not for true interpretability. The distinction between explanations and mechanisms matters legally and practically.

The fairness impossibility theorem formalized why COMPAS was simultaneously calibrated and biased: when base rates differ, you cannot satisfy calibration and equal error rates simultaneously. The choice of which to sacrifice is not a technical decision. It is a value judgment that must be made explicitly by people with legitimate authority.

The regulatory landscape is shifting from voluntary to mandatory. The EU AI Act’s risk-based framework — prohibiting the most dangerous uses, regulating high-risk applications, requiring transparency for limited-risk systems — is setting the global standard. The domain deep dives showed that the ethical challenges are not abstract; they are concrete, documented, and causing measurable harm in criminal justice, medicine, and financial services today.

Autonomous weapons raised the sharpest version of the question of meaningful human control. Existential risk raised the longest-horizon version. Accountability by design gave us a practical framework for addressing both the present and the horizon.

The IAAIS Ethics Audit asks the hardest thing: look honestly at what you have built, document what it can and cannot safely do, and commit to the ongoing responsibility of governance. The System Card is not the end of that commitment — it is the beginning.

In Chapter 15, we integrate all components of the IAAIS system into a complete deployment pipeline with a Streamlit user interface, monitoring infrastructure, and production documentation. The ethics audit you complete this week is the foundation on which that deployment rests.

14.17 Discussion Questions

The impossibility theorem and policy choice: Given that calibration and equal false positive rates cannot both be satisfied when base rates differ, which criterion should govern a recidivism risk tool, and who should make that choice? What process would lead to a legitimate answer?
Explainability as a right: The EU AI Act and GDPR give individuals the right to an explanation of automated decisions that significantly affect them. LIME and SHAP provide local approximations, but critics argue these are post-hoc rationalizations rather than true explanations. Is this distinction important for legal accountability?
Goodhart’s Law and safety evaluation: Safety benchmarks for large language models are now public, which means developers can train against them. Does training against safety benchmarks make models safer, or does it teach models to pass benchmarks without being safer? Design an evaluation approach more robust to Goodhart’s Law.
Autonomous weapons and meaningful human control: A weapons system is programmed to autonomously engage armed individuals in a defined combat zone, with 95% accuracy in distinguishing combatants from civilians. Is 5% civilian error acceptable? What is the relevant comparison — to human error rates under the same conditions, or to an absolute standard?
The red team’s responsibility: You are auditing an AI diagnostic system and discover that it performs 15 percentage points worse for patients over 85 than for younger patients — a population that represents 8% of users but 23% of serious missed diagnoses. The developers say this is “within acceptable limits.” What is your responsibility as auditor?
The EU AI Act and innovation: Critics argue strict regulation will drive AI development to less regulated jurisdictions and deprive citizens of beneficial applications. Which provisions of the EU AI Act do you think are well-calibrated? Which do you think are too restrictive or not restrictive enough?
System cards and accountability theater: Critics argue model cards are performative — providing the appearance of accountability without its substance. How would you design a system card regime with genuine enforcement? What institution should have the authority and capacity to verify system card claims?
Your IAAIS ethics audit: Looking honestly at your system — what is the most serious fairness risk? The most serious safety risk? The most serious hallucination risk? For each: what is the worst realistic scenario if the risk materializes, what mitigation is in place, and what residual risk remains?

14.18 Further Reading

14.18.1 Fairness and Bias

Chouldechova, A. (2017). Fair prediction with disparate impact. Big Data, 5(2), 153–163. The mathematical proof of the COMPAS impossibility.

Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016). Inherent trade-offs in the fair determination of risk scores. arXiv:1609.05807.

Barocas, S., Hardt, M., & Narayanan, A. (2023). Fairness and Machine Learning. MIT Press. Available free at fairmlbook.org. The definitive textbook.

Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016). Machine bias. ProPublica. The original COMPAS investigation.

14.18.2 Explainability

Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). “Why should I trust you?”: Explaining the predictions of any classifier. KDD 2016. The LIME paper.

Lundberg, S. M., & Lee, S. I. (2017). A unified approach to interpreting model predictions. Advances in NeurIPS, 30. The SHAP paper.

14.18.3 AI Governance and Regulation

EU Artificial Intelligence Act (2024). Available at eur-lex.europa.eu.

Calo, R. (2017). Artificial intelligence policy: A primer and roadmap. UC Davis Law Review, 51(2), 399–435.

14.18.4 AI Safety

Russell, S. (2019). Human Compatible: Artificial Intelligence and the Problem of Control. Viking. The clearest accessible statement of the alignment problem.

Amodei, D., et al. (2016). Concrete problems in AI safety. arXiv:1606.06565. The foundational technical safety research agenda.

Bai, Y., et al. (2022). Constitutional AI: Harmlessness from AI feedback. arXiv:2212.08073. Anthropic’s approach to scalable alignment.

14.18.5 Autonomous Weapons and Ethics

Human Rights Watch. (2020). Stopping Killer Robots. Available at hrw.org.

Obermeyer, Z., et al. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453.

— End of Chapter 14 —