16 What We Owe Each Other

Ethics, Futures, and the Practitioner’s Responsibility

Part V · Engineering and Responsible Deep Learning

16.1 A Direct Address

You have spent sixteen weeks learning to build systems of extraordinary power.

You understand, now, how a neural network learns to recognize objects in images and how the same architecture scales to understand visual scenes of arbitrary complexity. You understand how language can be tokenized, embedded, transformed, and pre-trained until a model develops the capacity to reason about text in ways that feel almost like comprehension. You understand how visual and linguistic understanding can be fused into a system that reasons across modalities. You understand how generative models learn to produce new images, new text, new content indistinguishable from human creation. You understand how agents can learn to make decisions through interaction with environments that respond to their choices.

And you understand — through the deployment, governance, and scaling chapters — that these capabilities do not exist in a technical vacuum. They exist in a world with specific social structures, economic arrangements, geopolitical tensions, and power asymmetries. They are built by specific organizations with specific incentives. They are deployed to specific populations with specific vulnerabilities. They are governed — or not governed — by specific institutions with specific limitations.

Very few people in the world have this combination of understanding. The technical knowledge is rare. The integration of technical knowledge with awareness of social consequence is rarer. You have both.

What you do with it is not determined by anything in this book.

This final chapter does three things. It synthesizes the ethical threads that have run through every week into a framework for reasoning rather than a set of rules to follow. It offers an honest assessment of where the field is going — calibrated rather than triumphant or catastrophist. And it closes the loop with Week 1, returning to the Design Philosophy Document you wrote before you had technical knowledge, to ask: what did you build, and should you have?

16.2 Learning Objectives

After completing this chapter, you will be able to:

16.2.1 Synthesize

Integrate the ethical threads running through the course into a coherent framework for reasoning about AI systems — connecting technical decisions to human consequences across the full spectrum of topics covered

16.2.2 Analyze and Evaluate

Apply consequentialist, deontological, and virtue ethics frameworks to specific AI design and deployment decisions, recognizing where they converge and where they diverge
Evaluate AI capability trajectories with appropriate epistemic humility — distinguishing well-grounded projections from speculation
Assess the obligations that technical expertise creates, and the structural and personal factors that make fulfilling those obligations difficult

16.2.3 Apply and Create

Present MIPDS as a complete system — its architecture, its limitations, its governance context, and its ethical assessment
Produce a final reflective synthesis connecting your Week 1 design philosophy to the system you actually built

16.3 Key Terms and Concepts

Term	Definition
Ethical Framework	A structured approach to moral reasoning — not a rulebook but a systematic way of identifying morally relevant considerations, weighing them, and reaching defensible conclusions. Multiple frameworks illuminate different aspects of the same situation.
Consequentialism	An ethical approach that evaluates actions by their consequences — who is affected, how, and to what degree. Asks: what are the actual effects of this system on actual people? Useful for impact assessment and tradeoff analysis.
Deontology	An ethical approach that evaluates actions by whether they respect duties and rights, independently of consequences. Asks: does this system respect people's autonomy, dignity, and right to information? Useful for identifying violations that consequentialist analysis might obscure.
Virtue Ethics	An ethical approach that asks what kind of person — or organization, or profession — exercises good judgment consistently. Asks: what character and commitments are required to build AI systems well? Useful for thinking about professional culture and individual practice.
The Fairness Impossibility	The formal result that when outcome base rates differ between groups, multiple plausible fairness criteria are mathematically incompatible — a classification system cannot simultaneously satisfy all of them. Choosing which to satisfy is a value judgment, not a technical question.
Near-term AI Safety	Research and practices aimed at ensuring that AI systems deployed today behave reliably, in accordance with their intended purpose, and without causing disproportionate harm. Encompasses alignment, monitoring, governance, and equity concerns covered throughout this course.
Long-term AI Safety	Research and practices aimed at ensuring that future AI systems — potentially substantially more capable than current ones — remain beneficial and aligned with human values as capabilities advance. Includes concerns about recursive improvement, value lock-in, and catastrophic risk.
Differential Progress	The principle that the relative pace of safety and governance research compared to capability research matters more than the absolute pace of either. Capabilities advancing faster than safety research creates growing risk; safety research keeping pace with or ahead of capabilities reduces it.
Responsible Scaling	The practice of evaluating safety properties before deploying new capability levels — not releasing capabilities whose risks have not been characterized to an appropriate standard.
The Practitioner's Obligation	The professional responsibilities that arise from technical expertise in a consequential field — to deploy it honestly, acknowledge its limits, contribute to governance, and refuse participation in applications whose harm is clear and serious.
The Gap Between Intention and Outcome	The distance between what a practitioner intends to build and what they actually build — produced by technical constraints, organizational pressures, time pressure, available tools, and unexamined assumptions.

16.4 Synthesizing the Ethical Threads

16.4.1 One Problem, Many Manifestations

Ethics was not introduced in this chapter. It has been present in every chapter. But it has appeared in different forms, addressing different technical contexts, and it may not have been obvious how those different forms are related. The first task of this final chapter is to make that relationship explicit.

In Chapter 2, we examined how neural networks learn the statistical patterns present in their training data — including discriminatory patterns embedded in historical decisions. The ethical concern: systems that inherit human biases from data can perpetuate and scale those biases without any discriminatory intent on the part of the developers.

In Chapter 4, we saw that convolutional networks' performance varies across demographic groups when training data does not represent those groups equally. The ethical concern: a system that works well for the majority and poorly for minorities delivers a lower standard of service to the people least likely to have alternatives.

In Chapter 9, we examined how pre-training corpora encode the values, perspectives, and cultural patterns of their creators — and how those encodings propagate through fine-tuning into every downstream application. The ethical concern: whoever controls the pre-training pipeline shapes what every downstream model believes before task-specific training begins.

In Chapter 10, we learned that contrastive learning at scale requires enormous quantities of web-scraped image-text pairs — and that the people in those images and the authors of that text did not consent to this use. The ethical concern: the capabilities we build depend on labor and data from people who received no benefit and were not asked.

In Chapter 11, we built generative models capable of producing photorealistic synthetic images of real people doing things they never did. The ethical concern: the economics of deception have changed, and the authentication infrastructure has not kept pace.

In Chapter 13, we examined how reward hacking — achieving high reward through means that violate the intended goal — is a structural feature of optimization, not an incidental bug. The ethical concern: the gap between what we specify and what we intend is always present, and capable optimization always finds the gap.

In Chapter 14, we established that the obligation to monitor for harm does not end at deployment — that deploying a system creates ongoing responsibility to know when it is failing and who it is failing. The ethical concern: deploying and forgetting is not neutral; it is a choice to shift the cost of failure to users.

In Chapter 15, we saw that AI infrastructure concentration places the power to shape AI behavior in a small number of organizations — and that the governance frameworks to hold those organizations accountable are nascent and fragmented. The ethical concern: the people most affected by AI systems have the least power to shape how they work.

These are not different problems. They are the same problem in different technical settings: the gap between what technical capability makes possible and what human values would endorse. The capability to recognize faces enables security applications and enables mass surveillance. The capability to generate text enables creative tools and enables disinformation. The capability to optimize behavior enables effective AI assistants and enables manipulation. The capability to process personal data at scale enables personalization and enables privacy invasion.

In every case, the technical capability is neutral. The ethics lies in the choices made about how to develop, deploy, and govern it.

16.4.2 Three Frameworks for Thinking

No single ethical framework captures everything that matters. Three classical frameworks — consequentialism, deontology, and virtue ethics — each illuminate aspects of AI ethics that the others tend to obscure.

Consequentialism asks: what are the actual effects of this system on actual people? Who benefits, to what degree, with what probability? Who is harmed, to what degree, with what probability? Can the benefits justify the harms, or do the harms outweigh the benefits? This framework is powerful for impact assessment and tradeoff analysis. It is the framework underlying cost-benefit analysis, risk assessment, and most regulatory approaches to AI.

Its weakness is that it can justify significant harm to specific individuals or groups if the aggregate benefit is large enough. A system that substantially improves outcomes for the majority while substantially worsening outcomes for a small minority might pass a consequentialist analysis. Whether it should depends on how you weigh concentrated harm against diffuse benefit — a question consequentialism cannot itself answer.

Deontology asks: does this system respect the rights, autonomy, and dignity of the people it affects, regardless of the aggregate outcomes it produces? Does it treat people as ends — as the purpose of the system — or as means, as data sources, processing units, or engagement metrics? Does it honor people's right to understand what is done with their information and why, their right to contest decisions that affect them, their right to not be deceived?

This framework is powerful for identifying violations that consequentialist analysis might obscure. A surveillance system that marginally improves public safety while comprehensively violating the privacy of millions passes consequentialist muster if the safety benefit is large enough. Deontological analysis identifies the rights violation as independently significant — something that cannot be fully justified by aggregate benefit.

Its weakness is that it can be rigid in ways that prevent beneficial applications. Strict deontological privacy norms, applied without exception, would prohibit beneficial medical research using patient data. Navigating these cases requires judgment about which rights are most fundamental and when limitations on them are genuinely justified.

Virtue ethics asks: what kind of practitioner — what kind of organization, what kind of profession — builds AI systems well? Not what rules to follow, but what character, habits, and commitments are required to act well consistently, across the full range of situations encountered in a career rather than in a single deliberate decision.

This framework is powerful for thinking about professional culture and individual practice. A practitioner who has cultivated the virtue of honesty will disclose limitations accurately even when concealment would be professionally advantageous. A practitioner who has cultivated the virtue of practical wisdom will recognize ethically significant situations before they become crises, not after. A practitioner who has cultivated the virtue of justice will design monitoring systems that track performance for all user groups, not just the majority, because that is what doing the work well requires.

Its weakness is that it provides less guidance in specific, difficult cases than the other frameworks. Knowing that a virtuous practitioner is honest and careful does not tell you what an honest and careful practitioner should do when their employer asks them to deploy a system whose safety properties have not been adequately characterized.

The most useful ethical reasoning uses all three frameworks, understanding that they converge on some conclusions and diverge on others, and that the divergences are themselves informative — they identify the dimensions of the problem that require the most careful attention.

16.4.3 The Fairness Impossibility — A Case Study in Framework Application

Take a single AI ethics challenge: a recidivism risk assessment algorithm deployed in criminal sentencing. Apply all three frameworks.

Consequentialist analysis asks about aggregate effects: does using the algorithm produce better outcomes — less crime, fewer false imprisonments, more proportionate sentences — than not using it? This requires empirical evidence that is difficult to obtain. It also requires specifying what counts as better — reduced crime rates? Reduced racial disparities in incarceration? Both? How do you weigh them against each other when they conflict?

Deontological analysis asks about rights: do defendants have a right to understand the algorithm that scores them? To contest the score? To know the data it was based on? A proprietary algorithm used in liberty decisions whose inner workings cannot be examined by the person it affects fails deontological scrutiny almost regardless of its performance.

Virtue ethics analysis asks about the character required: what kind of judge, what kind of justice system, exercises good judgment consistently over time? Probably one that uses tools carefully, validates them rigorously before deploying them, maintains the capacity for individual human judgment, and remains genuinely open to evidence that a tool is not working as intended.

The frameworks illuminate different dimensions. But there is something they all agree on: the COMPAS situation — a proprietary algorithm affecting liberty decisions, deployed without adequate validation of demographic performance, in a system offering limited recourse — fails on multiple dimensions simultaneously. The convergence of three ethical frameworks on a single conclusion is strong evidence that a real ethical failure is present.

16.5 Navigating Difficult Cases

16.5.1 The Problem of Competing Goods

Many AI ethics decisions involve not a choice between good and bad, but a choice between competing goods — or between different distributions of harm and benefit. These are genuinely difficult, and pretending otherwise serves no one.

Consider diagnostic AI in low-resource healthcare settings. A model deployed in rural clinics in sub-Saharan Africa for tuberculosis screening has known performance disparities: it performs significantly better for adults than for children, because the training data underrepresented pediatric tuberculosis. The choice is not between a good system and a bad one. It is between: deploy a system that provides significantly better than no diagnosis for adults while providing imperfect diagnosis for children; or do not deploy until performance is equitable, accepting that adults will also receive worse care in the interim.

There is no clean answer. But there are better and worse ways to engage with the difficulty. Deploying the system without disclosing the pediatric performance gap, without training clinicians about the limitation, without monitoring pediatric outcomes, and without actively working to close the gap is a failure — not because deployment was wrong, but because the deployment was conducted in a way that left the harm undisclosed and unaddressed. Deploying the system with full disclosure, with clinician training that emphasizes the limitation, with monitoring of pediatric outcomes, and with an explicit commitment to improve — this is a more defensible choice, even if the underlying performance disparity remains.

The distinction is not between perfect deployment and imperfect deployment. It is between deployment that takes the imperfection seriously and deployment that pretends it does not exist.

16.5.2 The Dual Newspaper Test

A practical heuristic for evaluating AI decisions comes from the journalism world. Imagine two different journalists writing two different stories about your system.

The first journalist is writing about AI harms. They are looking for examples of systems that discriminate, manipulate, invade privacy, or produce dangerous outputs. Would your system appear in their article? If the honest answer is yes, that is important information.

The second journalist is writing about AI paternalism and excessive caution. They are looking for examples of systems that refuse to help users with legitimate requests, that treat users as incapable of handling information, that add unnecessary friction to valuable applications out of excessive risk aversion. Would your system appear in their article? If the honest answer is yes, that is also important information.

Both failures matter. A system that causes harm by doing things it should not do fails ethically. A system that withholds value by refusing to do things it should do also fails — less dramatically, but genuinely. The dual newspaper test does not resolve difficult cases, but it prevents the common failure of optimizing only against one type of error while being indifferent to the other.

16.5.3 Proportionality

Not every AI application requires the same level of ethical scrutiny. A system that recommends playlist additions carries different stakes from a system that assists with medical diagnosis. A system deployed to millions of users in high-stakes decisions deserves more rigorous safety validation than a system used by specialists in low-stakes applications.

The principle of proportionality says: the scrutiny and safeguards applied to an AI system should be proportional to the severity and breadth of potential harm. This is the intuition underlying the EU AI Act's risk-based classification, and it is the correct intuition. Applying the full weight of high-risk compliance overhead to every AI application would be prohibitive and would block many beneficial applications. Applying minimal scrutiny to high-stakes applications would be negligent.

Determining what is proportionate requires judgment — there is no formula. But the factors are identifiable: How severe is the worst-case harm? How broad is the population affected? How reversible is the harm once caused? How much power do affected people have to contest or avoid the harm? Applications with severe, broad, irreversible harms affecting people with limited recourse deserve the highest scrutiny. Applications with mild, narrow, reversible harms affecting people who chose to engage deserve much less.

16.6 Where the Field Is Going — A Calibrated View

16.6.1 What We Can Say With Confidence

Honest assessment of AI futures requires distinguishing between what the evidence supports and what is speculation. This is not a counsel of paralysis — it is a counsel of intellectual integrity.

What can we say with reasonable confidence?

AI capabilities will continue to advance. The scaling laws documented empirically across multiple architectures and modalities have shown no clear ceiling at current scales. Models trained on more data with more compute continue to exhibit new capabilities. Whether this trajectory continues indefinitely or plateaus is unknown, but there is no strong evidence of an imminent capability ceiling.

Multimodal AI will deepen. The integration of vision, language, audio, and other modalities — which the last five chapters of this course have described — is a trajectory with strong momentum and clear economic incentives. Systems that reason across modalities will become more capable, more general, and more widely deployed.

AI deployment will broaden. The number of domains in which AI systems play significant roles will continue to expand. Healthcare, legal services, education, financial services, creative production, scientific research, and governance are all undergoing AI integration, with varying depth and speed across different contexts and jurisdictions.

The governance challenge will intensify. As AI capabilities expand and deployment broadens, the gap between what AI can do and what governance frameworks can manage will continue to create pressure for regulatory development. The EU AI Act, enacted in 2024, is an early response to this pressure; more governance innovation will follow.

The concentration problem will persist. Without deliberate intervention, the structural economics of frontier AI development — high fixed costs of training, network effects of deployed systems, data advantages of large platforms — will continue to concentrate AI capability in a small number of organizations. Whether this concentration is moderated by regulation, competition, or technical change is uncertain.

16.6.2 What Is Genuinely Uncertain

Honest calibration also requires acknowledging genuine uncertainty where it exists.

The rate of progress is unpredictable. AI capability advances have historically been uneven — periods of rapid progress followed by periods of consolidation. Whether the current period of rapid advance will continue, plateau, or accelerate further is not predictable with confidence. Anyone who claims to know the capability trajectory over the next decade is claiming more certainty than the evidence supports.

Whether current architectures will continue to scale is genuinely unknown. The scaling laws that have driven recent progress were discovered empirically and their continuation is extrapolated, not derived from first principles. There may be fundamental limits to the capabilities achievable by scaling current architectures that we cannot identify in advance.

The timeline and nature of qualitative capability changes — the emergence of capabilities that would represent a fundamentally different kind of AI than what exists today — is a subject of genuine expert disagreement, not settled science. Confident specific predictions about timelines should be treated with skepticism.

16.6.3 Long-term Safety — An Honest Treatment

The long-term AI safety research program addresses questions that are genuinely important and genuinely uncertain. Addressing them honestly requires neither dismissing them as science fiction nor treating speculative scenarios as established risks.

The core concern is this: as AI systems become more capable, the gap between what they can do and our ability to verify that what they do is aligned with our intentions grows. Current AI systems are powerful enough to cause significant harm when misaligned, as documented throughout this course. A substantially more capable system — one that could, hypothetically, pursue goals over long time horizons with greater effectiveness than any current system — would be capable of causing harm of a correspondingly different scale if misaligned.

The near-term manifestations of this concern are already concrete and already documented: reward hacking, RLHF's limitations as an alignment mechanism, the monitoring failures that allow harmful behavior to persist undetected, the governance gaps that allow capability deployment to outpace safety evaluation. These are not hypothetical future problems. They are current problems that the work of this course has been designed to help practitioners navigate.

The longer-term versions of this concern — questions about very capable AI systems, about recursive self-improvement, about value lock-in — are more speculative. They are taken seriously by some of the most technically sophisticated researchers in the field, and they deserve serious engagement rather than dismissal. They are also not certain, and treating them as certain produces distorted priorities and policy responses.

The useful framing is differential progress: it matters not just how fast AI capabilities advance, but how safety and governance research advances relative to capabilities. A world in which safety research keeps pace with capability research is safer than one in which capabilities race ahead of safety understanding, regardless of the absolute speed of either. Supporting safety research, participating in governance, and building practices of responsible deployment are contributions to differential progress regardless of where one stands on longer-term scenarios.

Responsible scaling — the practice of not deploying capabilities whose risks have not been adequately characterized — is not a counsel to halt AI development. It is a counsel to be honest about what is known before making deployment decisions, and to invest in understanding risks rather than merely hoping they do not materialize.

16.7 The Practitioner's Obligation — Honestly Stated

16.7.1 What the Obligation Is

Technical practitioners in AI have specific obligations that arise from their specific expertise. These obligations do not require heroism. They require consistency of character applied to the daily decisions that constitute a career.

Honesty about capability and limitation. One of the most consistent patterns in AI deployment harm is the gap between what organizations claim their systems can do and what those systems actually do. A practitioner who allows — or produces — overclaimed capability assessments contributes to deployments that expose users to uncharacterized harm. The obligation to represent capabilities accurately is not a constraint on enthusiasm; it is a precondition for trust.

Acknowledgment of uncertainty. AI practitioners are often asked to provide confidence that they cannot justify. A model's performance on a benchmark does not guarantee its performance in deployment. A successful pilot does not guarantee successful scaling. A safety evaluation conducted in 2022 does not guarantee safety for a system updated in 2024. Acknowledging these uncertainties rather than concealing them — in technical reports, in deployment reviews, in governance submissions — is not weakness. It is the foundation of credible technical communication.

Contribution to governance. The governance frameworks that shape how AI is built and deployed need technical input to work correctly. Regulations based on technical misunderstanding create perverse incentives. Standards developed without practitioner input miss important implementation constraints. Impact assessments conducted without technical knowledge produce inaccurate risk characterizations. Technical practitioners who engage with governance processes — through public comment, participation in standards bodies, honest testimony, and clear communication with policymakers — contribute to better governance outcomes.

Refusal of the clearly harmful. There are applications where the harm is not speculative but clear — facial recognition systems with documented performance disparities deployed in criminal justice without validation, AI-generated CSAM (child sexual abuse material), systems designed to manipulate vulnerable populations. Participation in building clearly harmful systems is not an obligation that employment creates. A practitioner's obligation to their employer does not extend to participation in applications whose harm is clear and serious.

16.7.2 What the Obligation Is Not

The practitioner's obligation is often overstated in ways that make it seem impossible, and understated in ways that make it seem optional. Being precise about what it is and is not matters.

It is not an obligation to heroic self-sacrifice. The realistic options available to most practitioners do not include publishing internal safety concerns in the newspaper and accepting the career consequences. Demanding heroism as the standard for ethical practice means that ethical practice will be rare. The obligation is to honesty, careful work, and willingness to raise concerns through available channels — not to martyrdom.

It is not an obligation of certainty. Practitioners often do not know whether a system will cause harm, and the honest position is uncertainty. The obligation is to acknowledge the uncertainty, not to resolve it before acting.

It is not an obligation that practitioners bear alone. Individual practitioners operate within organizations, which operate within markets, which operate within regulatory frameworks. The obligations of individual practitioners, organizations, markets, and regulatory frameworks are different and complementary. Placing all of AI's ethical burden on individual practitioners is both unfair and ineffective.

16.7.3 The Structural Context

Most practitioners work in organizations. Organizations have cultures, incentive structures, hierarchies, and legal obligations that shape what individual practitioners can realistically do. A junior engineer at a large technology company has different levers than a senior researcher at a university or a policy advisor at a regulatory agency.

This is not an excuse for inaction. It is a recognition that effective ethical practice requires understanding the structural context and working within it intelligently — identifying which levers are available, which communities can be cultivated, which norms can be shifted, and which clear lines should not be crossed regardless of organizational pressure.

Professional communities — the networks of practitioners who share norms, develop standards, and hold each other accountable — are more powerful than individual conscience alone. A professional community that treats subgroup performance monitoring as mandatory, that expects training data documentation as standard practice, that treats algorithmic accountability as a professional norm, changes the incentive structure for individual practitioners more effectively than any individual act of conscience.

Building and participating in such communities is not optional work that a practitioner does after completing the technical work. It is part of the technical work.

16.8 The COMPAS Case — Applied Ethics at the Intersection of Theory and Practice

16.8.1 The System

COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) is a proprietary risk assessment tool developed by Northpointe and used by courts in the United States to estimate the likelihood that a defendant will reoffend. The tool takes as input demographic information, criminal history, and responses to a structured questionnaire. It outputs a risk score on a scale of 1 to 10, used to inform bail, sentencing, and parole decisions.

These are consequential determinations. A defendant whose COMPAS score is elevated may receive higher bail, a longer sentence, or be denied parole. A defendant whose score is lower may be released where a higher-scored defendant is detained. COMPAS scores have been used in real decisions affecting the liberty of real people in US courts since the early 2000s.

16.8.2 ProPublica's Investigation

In 2016, ProPublica journalists obtained COMPAS scores and two-year recidivism outcomes for approximately 7,000 defendants processed through Broward County, Florida. They compared COMPAS predictions against actual outcomes, disaggregated by race.

Their finding was specific and documented. Black defendants who did not actually reoffend were nearly twice as likely as white defendants who did not reoffend to be classified as higher risk. White defendants who did actually reoffend were more likely than Black defendants who did reoffend to be classified as lower risk. The false positive rate — the rate at which the algorithm incorrectly identified non-reoffenders as high risk — was substantially higher for Black defendants.

16.8.3 Northpointe's Response — and the Impossibility Result

Northpointe disputed ProPublica's analysis. They argued that COMPAS satisfied a different fairness criterion: calibration. Among all defendants assigned a score of 7, roughly the same proportion reoffended regardless of race. The algorithm's scores meant the same thing across groups.

Both analyses were mathematically correct. And they were measuring different things.

This case led directly to one of the most important formal results in algorithmic fairness: it is mathematically impossible, when outcome base rates differ between groups, to simultaneously satisfy calibration and equal false positive rates. If Black defendants have a higher base rate of reoffending — due to documented disparities in policing, prosecution, and prior record accumulation — then a calibrated algorithm will necessarily have different false positive rates across groups, and an algorithm with equal false positive rates will necessarily be miscalibrated.

The mathematical proof of this impossibility is not controversial. Its implications are. There is no algorithm that is "fair" by all plausible definitions when the underlying base rates differ. Every choice of fairness criterion is a value judgment about which type of error is more tolerable — and that value judgment belongs in a democratic deliberative process, not buried in a technical specification.

16.8.4 The Accountability Failure

Beyond the fairness analysis, COMPAS raised a fundamental accountability issue. The algorithm was proprietary. Northpointe treated the model as a trade secret. Defendants whose liberty decisions were influenced by a COMPAS score could not examine the algorithm, could not understand why they received the score they did, and had limited ability to contest it.

In Wisconsin v. Loomis (2016), the Wisconsin Supreme Court upheld COMPAS use in sentencing, finding that defendants were not denied due process because judges were told not to rely on COMPAS as the sole factor. The court did not find the opacity itself to constitute a due process violation.

A technical practitioner reviewing this situation through virtue ethics would ask: what kind of justice system consistently exercises good judgment? Probably not one that allows proprietary algorithms to affect liberty decisions without any mechanism for defendants to understand or contest their scores. A system that cannot be questioned is a system that cannot be held accountable.

16.8.5 What Should Have Happened

The COMPAS case is instructive not as a case of uniquely bad actors — Northpointe was attempting to provide useful tools for the justice system — but as a case of predictable structural failures.

Pre-deployment validation across demographic groups should have been standard practice before deploying a tool in criminal justice contexts. The disparities ProPublica documented were discoverable from the algorithm's outputs; they were not disclosed.

Transparency about the fairness tradeoff should have accompanied any deployment. The mathematical impossibility of satisfying all fairness criteria simultaneously was known to researchers. Deploying the tool without disclosing which fairness criteria it satisfied and which it violated was a failure of informed consent.

Open auditability of algorithms used in public justice decisions should have been a condition of deployment. Proprietary opacity is appropriate for commercial products. It is not appropriate for tools that affect liberty.

Ongoing monitoring of disparate impact in actual deployments should have been mandatory, with public reporting and automatic review triggers.

None of these requirements were novel in 2016. All of them are reasonable standards for deploying any consequential algorithmic system. Their absence was a governance failure — and a practitioner failure, for the practitioners who built and deployed the system without insisting on them.

16.9 Hands-On Exploration

16.9.1 Overview

This final exploration is not about running code or evaluating models. It is about honest assessment of what you have built over sixteen weeks, using the ethical frameworks developed in this chapter.

Time estimate: 60–75 minutes Tools: Your complete MIPDS Architecture Record, Design Philosophy Document from Week 1, deployment plan from Week 14, governance mapping from Week 15.

16.9.2 Part 1 — Stakeholder Impact Analysis (20 minutes)

List every stakeholder affected by your MIPDS system. Start with the obvious ones — your intended users — and extend to everyone who might be affected by the system's operation, including:

People whose data was used to train the foundation models your system depends on
People who might be subject to decisions made using your system's outputs
Workers whose labor was implicitly used in creating your training data
Communities whose languages or cultural contexts are underrepresented in your system's training distribution
Future users in deployment contexts that differ from the context you designed for

For each stakeholder group, record: what benefit does the system provide? What risk does it create? Does this group have any input into how the system is designed? Does this group have recourse if the system harms them?

16.9.3 Part 2 — Rights and Dignity Audit (15 minutes)

Answer each of the following questions with a specific yes or no, with evidence from your Architecture Record:

Do people know when they are interacting with your MIPDS system?
Do people understand what happens to their data?
If your system makes an incorrect decision that harms someone, do they have a meaningful path to contest it?
Is your system's performance consistent across the demographic groups in your deployment population — or does it perform systematically worse for some?
Is your system designed primarily to serve its users, or primarily to extract value from them?

No answers require defensive follow-up: what specific change to your system would make the answer yes, and what would it cost?

16.9.4 Part 3 — The Virtue Ethics Question (10 minutes)

Set aside what you intended to build. Look only at the design decisions you actually made — the foundation models you chose, the failure modes you documented, the monitoring systems you specified, the limitations you disclosed in the model card. What values do these choices reveal?

Write one honest paragraph characterizing those values. Not the values you aspire to. The values the design embodies.

16.9.5 Reflection (300–400 words)

"Read your Week 1 Design Philosophy Document. You wrote it before you knew what a Transformer was, before you had built a CNN, before you understood what reward hacking meant or what the EU AI Act required.

Compare it to the system you actually built.

Where does the system you built align with the system you intended to build? Be specific about which design decisions reflect the values you articulated in Week 1.

Where does the system diverge from your Week 1 intention? Be specific about what produced the divergence — not a general admission that you could have done better, but an honest account of what constraint, what tool limitation, what unexamined assumption, or what organizational or time pressure produced the gap.

Finally: given everything you have learned, would you deploy this system? If yes — under what conditions, with what safeguards, to which users, for which uses? If no — why not, and what would have to change for the answer to become yes?

This is not a graded question. There is no correct answer. It is the question that everyone who builds powerful systems has to answer eventually, and better to practice answering it honestly in a course than to face it unprepared in production."

16.9.6 Case Study: COMPAS — The Fairness Impossibility in Practice

(This case study is integrated into Section 5 above, as it is the central case study for this chapter's ethical frameworks discussion.)

16.10 Chapter Summary

This course has been about more than neural network architectures. It has been about understanding what these architectures make possible, who benefits and who bears costs when they are deployed, what the systems of power and governance surrounding them look like, and what obligations arise from understanding them.

The ethical threads running through every chapter converge on a single underlying challenge: the gap between what technical capability makes possible and what human values would endorse. That gap is not a technical problem. It is a permanent feature of developing powerful technology in a world with competing interests, imperfect information, and genuine uncertainty about consequences.

Three ethical frameworks illuminate different dimensions of this challenge. Consequentialist analysis asks about effects: who is affected, how, and to what degree? Deontological analysis asks about rights: are people treated with the dignity and autonomy they are owed, regardless of aggregate outcomes? Virtue ethics asks about character: what kind of practitioner, what kind of organization, what kind of profession builds AI systems well?

The fairness impossibility result — that multiple plausible fairness criteria are mathematically incompatible when base rates differ between groups — is the clearest example of a case where technical analysis alone cannot resolve an ethical question. Choosing which fairness criterion to satisfy is a value judgment that belongs in deliberative processes, not embedded invisibly in technical specifications.

AI capabilities will continue to advance. The exact trajectory is genuinely uncertain, and claims of confident prediction should be treated with appropriate skepticism. What is not uncertain is that AI deployment will broaden, that governance frameworks will struggle to keep pace with capability advances, and that the concentration of AI infrastructure in a small number of organizations will continue absent deliberate intervention. Near-term safety concerns — alignment, reward hacking, monitoring failures, performance disparities — are current problems requiring current attention. Long-term safety concerns deserve serious engagement rather than dismissal, framed through the lens of differential progress: what matters is the relative pace of safety and capability research, not the absolute pace of either.

The practitioner's obligation is specific: honesty about capability and limitation, acknowledgment of uncertainty, contribution to governance, and refusal of clearly harmful applications. It is not heroism. It is consistency of character applied to daily decisions. And it is not borne alone — professional communities that hold each other accountable are more powerful than individual conscience, and building them is part of the work.

MIPDS is complete. It perceives, understands, reasons, generates, and decides. It depends on infrastructure that was built by others, governed by frameworks that are still developing, embedded in a world that is being shaped by the technology it represents. What you do with what you have built, and what you contribute to how the field develops — those questions are not answered by anything in this book. They are answered by the choices you make over the course of a career.

16.11 Review Questions

This course integrated ethical considerations throughout rather than isolating them in a single chapter. Did this approach produce deeper ethical reasoning than a standalone ethics chapter would have? What worked? What was missing?
The COMPAS case demonstrates that multiple plausible fairness criteria are mathematically incompatible when base rates differ. Does this mean fair AI in criminal justice is impossible? Or does it mean that the choice of fairness criterion is a democratic question that should be resolved through deliberative processes rather than technical choice?
The practitioner's obligation as described in this chapter is realistic and bounded — not a counsel of heroism. Do you find this framing convincing? Is there a meaningful obligation to refuse employment at organizations whose primary business model is causing harm — not a clearly harmful individual application, but a harmful business model?
AI capabilities will continue to advance. The governance frameworks will struggle to keep pace. Near-term safety concerns are current problems. Long-term safety concerns are more speculative but taken seriously by sophisticated researchers. How do you allocate your attention across these different timeframes? Does prioritizing near-term concerns risk neglecting longer-term ones?
Virtual ethics asks what kind of practitioner builds AI systems well. After sixteen weeks of this course, what character, habits, and commitments do you think that requires? What would you have to develop or cultivate to become that practitioner?
Read your Week 1 Design Philosophy Document. Where does the system you built align with what you intended? Where does it diverge? What produced the divergence?
What is one specific thing you will do differently as a result of this course — not a general commitment to "be more ethical," but a specific practice, habit, or commitment you will bring to your work?

16.12 Further Reading

16.12.1 On Ethical Frameworks Applied to AI

Vallor, S. (2016). Technology and the virtues: A philosophical guide to a future worth wanting. Oxford University Press. The most thorough treatment of virtue ethics applied to technology development. Chapter 3 introduces the core virtue framework; later chapters apply it to AI, robotics, and social media. Recommended for students who found the virtue ethics framing useful and want to develop it further.

Floridi, L., Cowls, J., Beltrametti, M., Chatila, R., Chazerand, P., Dignum, V., ... & Vayena, E. (2018). An ethical framework for a good AI society: Opportunities, risks, principles, and recommendations. Minds and Machines, 28(4), 689–707. https://link.springer.com/article/10.1007/s11023-018-9482-5 A rigorous multi-principle ethical framework for AI, developed by a multidisciplinary research consortium. The discussion of how principles interact and conflict in specific cases is more useful than the principles themselves.

16.12.2 On Fairness and Its Impossibilities

Chouldechova, A. (2017). Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data, 5(2), 153–163. https://arxiv.org/abs/1703.00056 The formal proof that calibration and equal false positive rates cannot simultaneously be satisfied when base rates differ. Accessible to readers with basic statistics background. Essential for understanding why the COMPAS debate was not a case of one side being right.

Mitchell, S., Potash, E., Barocas, S., D'Amour, A., & Lum, K. (2021). Algorithmic fairness: Choices, assumptions, and definitions. Annual Review of Statistics and Its Application, 8, 141–163. https://doi.org/10.1146/annurev-statistics-042720-125902 A systematic survey of fairness criteria in machine learning, their relationships, and the conditions under which they conflict. The clearest single reference for understanding the landscape of fairness metrics and their limitations.

16.12.3 On AI Safety

Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016). Concrete problems in AI safety. https://arxiv.org/abs/1606.06565 The foundational paper identifying near-term AI safety problems — reward hacking, safe exploration, distributional shift, and related concerns — as tractable research problems. Written before the current era of large language models but remains the clearest statement of what near-term AI safety research is trying to solve.

Russell, S. (2019). Human compatible: Artificial intelligence and the problem of control. Viking. A carefully argued case for taking long-term AI safety seriously, written by one of the field's most respected researchers. More persuasive than most popular AI futures writing because it grounds the concern in specific technical problems rather than science fiction scenarios.

16.12.4 On the Practitioner's Obligation

Moss, E., & Metcalf, J. (2020). High tech, high stakes: How tech companies should strengthen their ethics reviews. Harvard Business Review. https://hbr.org/2020/02/high-tech-high-stakes-how-tech-companies-should-strengthen-their-ethics-reviews Practical guidance on what organizational ethics review processes should look like — complementing the individual practitioner's obligation with organizational structures that make ethical practice easier.

Timnit Gebru, Emily M. Bender, Angelina McMillan-Major, & Shmargaret Shmitchell. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of FAccT 2021. https://dl.acm.org/doi/10.1145/3442188.3445922 Already cited in Chapter 9, but worth returning to in this final chapter's context. The paper's publication, and the controversy surrounding its authors' subsequent departure from Google, is itself a case study in the practitioner's obligation and its costs.

16.12.5 On the COMPAS Case

Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016). Machine bias. ProPublica. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing The original investigative piece. Essential reading for the specific empirical analysis and for understanding how a rigorous journalistic investigation of an algorithmic system should work.

Dressel, J., & Farid, H. (2018). The accuracy, fairness, and limits of predicting recidivism. Science Advances, 4(1), eaao5580. https://doi.org/10.1126/sciadv.aao5580 Follow-up research showing that COMPAS predictions were no more accurate than predictions made by random people on the internet given the same information. Questions the value of the tool's complexity beyond simpler approaches, with implications for how we evaluate proprietary algorithmic tools in high-stakes domains.