16 What Comes Next

Frontier AI, Open Problems, and the Future We Are Building

CSC5350 · Artificial Intelligence

16.1 Opening Narrative

16.1.1 The Inflection Point

In the spring of 2023, a cognitive scientist named Gary Marcus and an AI researcher named Yann LeCun were engaged in a public disagreement that had been running for years. Marcus argued that large language models were fundamentally limited — sophisticated pattern matchers that could not reason, plan, or understand causality, and whose apparent capabilities masked a brittleness that would become apparent under any serious examination. LeCun argued that the current architectures were a stepping stone, that their limitations were engineering problems not fundamental barriers, and that the path to genuine machine intelligence was closer than critics imagined.

Both were partly right, which is the most uncomfortable position in any debate.

The models of 2023 could write poetry, debug code, pass bar exams, and explain quantum mechanics. They could also confidently state that Napoleon was defeated at Waterloo in 1815 and that the capital of Australia was Sydney. They hallucinated citations, failed at multi-step arithmetic, struggled with spatial reasoning, and produced authoritative-sounding nonsense on topics outside their training distribution. The same system that passed a medical licensing exam could be reliably fooled by a patient describing symptoms in an unusual order.

What made this moment genuinely unprecedented was not the capabilities of any individual system. It was the pace of change. The GPT-3 paper was published in May 2020. GPT-4 followed in March 2023 — three years and roughly three orders of magnitude of compute later. The capabilities were not the same system with some improvements. They were qualitatively different. Whatever had emerged at scale was not just “more autocomplete.”

And yet nobody — not the researchers who built these systems, not the philosophers who studied intelligence, not the engineers who deployed them — fully understood what had emerged. The models were too large to interpret mechanistically, trained on data too vast to audit, and capable of behaviors that were not explicitly trained for and could not be entirely predicted. They were, in the most precise sense, alien minds: products of human culture and language, shaped by human feedback and human values, and yet operating through processes that no human fully understood.

This is the world you are entering as an AI practitioner. Not a world where AI is a solved problem, or a world where the path forward is clear, or a world where the consequences of what is being built are well understood. A world at an inflection point — where the decisions made in the next decade will shape the trajectory of a technology whose ultimate consequences are genuinely uncertain.

“We are building systems we do not fully understand, deploying them at a speed that outpaces our ability to evaluate them, in a world whose institutions were not designed to govern them. The question is not whether this is risky. The question is whether we can build the understanding, the institutions, and the practices that make the risk acceptable.”

16.2 Learning Objectives

After completing this chapter, you will be able to:

Describe the major open problems in AI research — reasoning, causality, grounding, robustness, and sample efficiency — and explain why they resist solution by scaling alone.
Explain the AGI debate: what AGI means, why the timeline question is so contested, and what the different camps claim and why.
Describe AI’s dual relationship with climate: its growing energy footprint and its potential as a tool for climate science, energy optimization, and materials discovery.
Analyze the global AI governance landscape — national strategies, multilateral frameworks, and the coordination problems that make AI governance difficult.
Describe the future of work implications of AI — the tasks most and least susceptible to automation, the labor market dynamics likely to follow, and the policy responses being proposed.
Explain the concept of human-AI collaboration and describe the conditions under which AI augmentation outperforms either humans or AI alone.
Identify the most important open questions in the domains covered by this course — what the next generation of researchers will be working on.
Present your complete IAAIS system — architecture, capabilities, limitations, ethics audit, and domain impact — as the course capstone.

16.3 Key Terminology

Term	Plain-Language Definition
AGI (Artificial General Intelligence)	AI systems with the ability to perform any intellectual task that a human can perform, at human or superhuman level. Definitions vary; the term encompasses both narrow capability benchmarks and deeper notions of flexible, transferable reasoning.
Emergent Capability	An ability that appears in AI systems at sufficient scale without being explicitly trained for. The mechanism is not fully understood; emergence challenges theories that predict capability gains should be smooth and predictable.
Scaling Laws	Empirical relationships describing how AI model performance improves with model size, dataset size, and compute budget. Established by Kaplan et al. (2020); suggest performance follows power laws across many orders of magnitude.
Scaling Hypothesis	The conjecture that scaling current architectures — more parameters, more data, more compute — is sufficient to produce human-level or superhuman general intelligence. Contested; proponents point to emergent capabilities; skeptics point to persistent limitations.
Reasoning Gap	The observed failure of current large language models to perform reliably on tasks requiring multi-step formal reasoning, particularly when the reasoning chain is long, the intermediate steps are novel, or the task requires tracking state.
Causal Reasoning	Reasoning about cause and effect — not just correlation. Humans can reason about counterfactuals (“what would have happened if…”), interventions (“what will happen if I do…”), and mechanisms (“why did this cause that”). Current AI systems largely lack robust causal reasoning.
Grounding	The connection between symbolic representations and the real-world entities or experiences they refer to. A language model knows the word “hot” from statistical context; it has not experienced heat. Whether this matters for reasoning is a deep open question.
Hallucination (structural)	The tendency of autoregressive language models to generate fluent, confident text that is factually incorrect or fabricated. A structural feature of the training objective, not fully eliminable by alignment techniques alone.
Sample Efficiency	How much data a learning system requires to reach a given level of performance. Humans learn many tasks from very few examples; current AI systems typically require vastly more data than humans for equivalent performance.
Foundation Model	A large model trained on broad data at scale that can be adapted to many downstream tasks. GPT-4, Claude, Gemini, and Stable Diffusion are foundation models. The term emphasizes the role these models play as the foundation on which more specialized systems are built.
Multimodal Reasoning	The ability to reason coherently across multiple data types — text, images, audio, structured data — in a unified, integrated way rather than switching between separate models.
AI Governance	The policies, regulations, standards, norms, and institutions that shape how AI is developed, deployed, and used. Includes national law, international agreements, industry self-regulation, and technical standards.
Brussels Effect	The tendency for EU regulation to become a de facto global standard, because companies operating globally prefer to comply with a single stringent standard rather than maintain different systems for different jurisdictions. The EU AI Act is expected to have this effect.
AI Safety Institute	Government organizations established to evaluate the safety of frontier AI systems. The UK and US both established AI Safety Institutes in 2023–2024; other countries are following.
Responsible Scaling Policy	A commitment by AI developers to conduct safety evaluations at defined capability thresholds and to pause or constrain deployment if safety concerns are found. Anthropic and other developers have published RSPs.
Human-AI Collaboration	Work arrangements in which humans and AI systems contribute complementary capabilities to a shared task, with the human providing judgment, context, and accountability and the AI providing speed, scale, and pattern recognition.
Task Automation	The substitution of AI or robotic systems for human labor in performing a defined task. Distinct from job automation: most jobs contain multiple tasks, and automation of some tasks may augment rather than eliminate the job.
Complementarity	The property of human and AI capabilities that makes them more productive together than separately. Tasks requiring creativity, judgment, emotional intelligence, ethical reasoning, and physical dexterity are most complementary with current AI; tasks requiring speed, consistency, and pattern recognition are most substitutable.

16.4 Section 1 — What AI Cannot Yet Do: The Open Problems

Fifteen chapters of this textbook have described what AI can do — and it is genuinely remarkable. But the most instructive frontier is the boundary between what current systems can and cannot do. Understanding that boundary is the first requirement for anyone who wants to advance it.

16.4.1 The Reasoning Gap

Large language models can solve many reasoning problems by pattern-matching to similar problems in their training data. They struggle reliably with problems that require extending chains of reasoning beyond the patterns they have seen — particularly when the chain is long, when intermediate steps must be tracked explicitly, and when the task requires working with truly novel combinations of concepts.

The most carefully studied version of this is compositional generalization: the ability to solve problems that combine familiar components in unfamiliar ways. A system that can answer questions about red triangles and blue circles separately may fail on questions about red circles, if that specific combination was not well-represented in training data. Humans generalize compositionally almost effortlessly — it is one of the most distinctive features of human cognition. Current AI systems do so unreliably.

Chain-of-thought prompting (Chapter 13) substantially improves performance on multi-step reasoning by asking the model to externalize its reasoning process. But this improvement is fragile: models can produce plausible-sounding chains of reasoning that contain errors, and the final answer can be wrong even when the stated reasoning appears correct. The reasoning chain, in these cases, is a performance rather than a reliable process.

16.4.2 Causality

Judea Pearl’s distinction between association and causation is one of the most important in the philosophy of science. Association tells you that two things co-occur. Causation tells you that one thing would change if the other were changed. Most of what machine learning learns from observational data is association. Causation requires something more — either experimental data (randomized controlled trials, A/B tests) or causal models that encode assumptions about the data-generating process.

Current AI systems are primarily associative. They can identify patterns with extraordinary sophistication and scale, but they cannot reliably distinguish correlational from causal relationships, and they cannot robustly reason about interventions and counterfactuals. This limits their applicability in medicine (where treatment decisions require causal reasoning), social policy (where interventions must be distinguished from spurious correlations), and anywhere that the distribution at deployment differs from the distribution at training.

Pearl’s “ladder of causation” — association, intervention, counterfactual — describes three levels of causal reasoning. Current AI systems are primarily on the first rung.

16.4.3 Grounding and Embodiment

When a child learns the word “heavy,” they learn it through physical experience — the effort of lifting objects, the resistance that increases with mass, the danger of dropping things. The word is grounded in bodily experience. When a language model learns the word “heavy,” it learns it through statistical context — the words that appear near it in text. Whether this difference matters for reasoning about weight and effort and danger is an open empirical question, but there is suggestive evidence that grounding in physical experience provides a kind of understanding that statistical co-occurrence does not.

Embodied AI — systems that interact with the physical world through sensors and actuators — is one active research direction. Robots that must manipulate objects learn something about the physics of the world that pure language training does not provide. Multimodal systems trained on images alongside text have access to visual information that provides additional grounding. Whether these additions close the grounding gap or merely extend the association surface is genuinely unclear.

16.4.4 Sample Efficiency

A child learns the concept of a dog from perhaps a few hundred examples encountered over several years. GPT-4 was trained on text describing dogs in hundreds of millions of contexts. And yet the child’s dog concept, tested in novel ways, is arguably more robust and generalizable than the model’s.

Sample efficiency — how much data is required to learn a concept — is one of the deepest unsolved problems in machine learning. Human learning exploits structure that current AI systems do not: prior knowledge, causal models of the world, the ability to learn actively by asking questions, and the ability to compose existing concepts into new ones without seeing examples of the composed concept.

Meta-learning (learning to learn), few-shot methods, and systems that incorporate causal or probabilistic structure are active research directions. None has yet achieved the sample efficiency of human learning in the general case.

16.5 Section 2 — The AGI Debate

No question in AI generates more heat and less light than the question of when — or whether — artificial general intelligence will arrive. The debate is worth understanding carefully, both because the question matters and because the ways people argue about it reveal important assumptions about the nature of intelligence.

16.5.1 What Is Being Debated

“AGI” is not a technically precise term. Different people mean different things by it:

The benchmark definition: a system that exceeds human performance on a comprehensive suite of cognitive tasks. By this definition, we already have AGI in specific domains (chess, Go, protein structure prediction) and are making rapid progress toward broader benchmark performance.

The economic definition: a system capable of performing any economically valuable cognitive task currently performed by humans, at human quality and cost. By this definition, AGI has not arrived but may be decades rather than centuries away.

The robust generalization definition: a system that can learn any task from minimal examples, transfer knowledge reliably across domains, reason causally about novel situations, and exhibit the flexible adaptability of human intelligence. By this definition, AGI may be further away — or may require fundamental architectural innovations not yet identified.

The consciousness definition: a system that genuinely understands, experiences, and reasons rather than merely producing outputs that resemble understanding, experience, and reasoning. By this definition, it is unclear whether AGI is achievable in principle, and the question becomes entangled with deep philosophical problems about the nature of consciousness that remain unsolved.

16.5.2 The Scaling Hypothesis and Its Critics

The dominant view among AI researchers who work on large language models and foundation models is that scaling — more parameters, more data, more compute, applied to current architectures — will continue to produce capability improvements, possibly including capabilities sufficient for AGI by some definitions.

The evidence for this view is the empirical scaling laws established by Kaplan et al. (2020): model performance on language tasks follows smooth power law relationships with model size, dataset size, and compute across many orders of magnitude, with no apparent plateau. The emergent capabilities observed at scale — few-shot learning, chain-of-thought reasoning, code generation — appeared unexpectedly and without explicit training, suggesting that scale alone can produce qualitatively new capabilities.

The skeptics make several counterarguments. First, the scaling laws apply to next-token prediction, not to the downstream capabilities we actually care about — and these may not scale in the same way. Second, the history of AI includes multiple periods of rapid capability growth followed by plateaus and “AI winters,” and current progress may be a feature of the particular data and compute regime we are in rather than a reliable extrapolation. Third, the persistent limitations in reasoning, causality, and grounding suggest that scale alone does not address the architectural limitations — that current systems may be getting better at the wrong thing.

The honest answer is that nobody knows. The confidence with which specific timelines are asserted by prominent AI figures is not matched by the epistemic state of the field. This is a domain where careful humility about prediction is warranted and where the consequences of being wrong — in either direction — are substantial.

16.6 Section 3 — AI and Climate: Two Sides of the Same Ledger

Artificial intelligence has a profound relationship with climate change — as a contributor to the problem and as a potential tool for addressing it. Both sides of this ledger deserve serious attention.

16.6.1 The Energy Cost of AI

Training frontier AI models requires enormous amounts of compute, and compute requires electricity. The training run for GPT-3 was estimated to require approximately 1,287 MWh of electricity — comparable to the annual energy consumption of over 100 average US homes. Subsequent models have been substantially larger. The total energy footprint of AI — across training, fine-tuning, inference, and the data center infrastructure that supports all of this — is growing rapidly.

The carbon footprint of AI depends on the energy source. Data centers powered by renewable energy have dramatically lower emissions than those powered by fossil fuels. The geography of AI computation matters: Microsoft, Google, and Amazon all have commitments to operate on renewable energy, but these commitments are measured in aggregate and do not guarantee that the electrons powering any specific computation are clean. The grid integration challenges — how data centers interact with local energy grids, particularly during peak demand — are real and complex.

The inference cost of deployed AI systems is, in aggregate, potentially larger than training costs. GPT-4 is used billions of times per day. Each inference requires energy. The cumulative energy consumption of inference at scale is difficult to estimate but potentially significant. Efficient model design — smaller models that perform adequately for specific tasks, quantization, pruning, distillation — is not merely an engineering consideration; it is an environmental one.

16.6.2 AI as a Climate Tool

Against the energy costs, AI offers genuine capabilities for climate science, clean energy, and decarbonization:

Climate modeling: AI has accelerated weather and climate simulation by orders of magnitude. Google DeepMind’s GraphCast and Huawei’s Pangu-Weather produce 10-day global weather forecasts in under a minute, comparable in accuracy to numerical weather prediction models that take hours on supercomputers. Beyond weather, AI is being used to accelerate the development of higher-resolution, longer-horizon climate models.

Energy grid optimization: AI systems optimize the dispatch of electricity generation assets across complex grids with renewable energy sources whose output varies with weather and time of day. DeepMind’s work with Google’s data centers achieved 40% reduction in cooling energy through reinforcement learning. Similar approaches applied to electricity grids could substantially reduce the marginal emissions of electricity consumption.

Materials discovery: AI has transformed computational chemistry and materials science. AlphaFold’s protein structure predictions have opened new avenues for enzyme design — including enzymes that break down plastics and catalyze industrial reactions with lower energy requirements. GNoME (Graph Networks for Materials Exploration, DeepMind 2023) predicted the crystal structures of 2.2 million new stable materials, including hundreds of thousands of potential battery materials. Faster materials discovery could accelerate the development of improved solar cells, batteries, and catalysts that the energy transition requires.

Carbon capture and monitoring: Satellite imagery analyzed by computer vision systems is being used to monitor deforestation, track methane emissions from oil and gas infrastructure, and verify carbon offset projects. These monitoring capabilities are essential for the transparency that effective carbon markets and international climate agreements require.

The net effect of AI on climate is uncertain and depends critically on how the technology is deployed, what computation it substitutes for, and how quickly the electricity grid decarbonizes. What is clear is that AI is not climate-neutral and that the research community has a responsibility to account for the energy costs of its work alongside its scientific and commercial contributions.

16.7 Section 4 — Global AI Governance: The Coordination Challenge

AI development is global, fast-moving, and concentrated in a small number of organizations in a small number of countries. The governance challenge is to establish norms, standards, and institutions adequate to manage the risks of this technology across different political systems, competitive pressures, and levels of technical sophistication.

16.7.1 The State of Play

Three major geopolitical actors dominate AI development: the United States, the European Union, and China. Each has adopted a distinct regulatory approach reflecting different political values, institutional structures, and strategic interests.

The United States, home to OpenAI, Google DeepMind, Anthropic, Meta AI, and most other frontier AI developers, has pursued a light-touch regulatory approach combined with significant public investment in AI research and national security applications. The 2023 Executive Order on AI created voluntary commitments from major developers, tasked NIST with developing safety standards, and initiated agency-by-agency guidance — but stopped well short of binding regulation. The primary US concern has been maintaining technological leadership; safety regulation is viewed with concern that it might disadvantage US companies relative to foreign competitors.

The European Union has enacted the most comprehensive regulatory framework, the AI Act, establishing legally binding requirements for high-risk AI systems with significant penalties for non-compliance. The EU’s approach reflects its historical preference for strong consumer protection and fundamental rights frameworks, and its relative distance from the commercial AI frontier creates less competitive pressure against regulation.

China has enacted specific regulations for recommendation algorithms, generative AI, and deepfakes, reflecting a preference for sector-specific regulation with significant political content requirements. China’s domestic AI governance emphasizes alignment with state interests — a quite different conception of “alignment” than the technical safety usage — and its frontier AI development is primarily concentrated in Baidu, Alibaba, Tencent, and Huawei.

16.7.2 The Coordination Problem

International AI governance faces a classic coordination problem. Safety standards that impose costs on AI developers may disadvantage countries that adopt them relative to countries that do not. If Country A requires extensive safety evaluation before deploying frontier AI systems, and Country B imposes no such requirements, AI developers in Country A may face a competitive disadvantage. This creates pressure to weaken standards or to defer them until international agreement can be reached — which is itself slow and difficult.

The Bletchley Declaration, signed at the UK’s AI Safety Summit in November 2023 by 28 countries including the US, EU, China, and others, was the first significant multilateral agreement on AI safety. It was principally a statement of shared concern about frontier AI risks rather than a framework of binding commitments — but it established that international dialogue on AI safety was possible even across significant geopolitical divisions.

The AI Safety Institutes established by the UK and US in 2023, with counterparts being established in other countries, provide technical infrastructure for evaluating frontier model risks. Whether these institutions develop into a meaningful international coordination mechanism depends on political will that is not yet clearly present.

The governance gap is most visible in specific applications. Facial recognition in public spaces is banned in the EU but deployed at scale by both US technology companies and Chinese systems. Autonomous weapons systems are subject to ongoing UN discussion without binding agreement. AI-generated content in political advertising is subject to patchwork regulation across jurisdictions. The cross-border nature of AI systems means that any individual jurisdiction’s rules can be circumvented by locating development or deployment elsewhere.

16.8 Section 5 — The Future of Work

No question about AI’s societal impact is more contested or more consequential than its effect on human employment. The range of credible expert predictions is extraordinary: from “AI will primarily augment human work, creating new jobs as it automates old ones” to “AI is qualitatively different from previous automation and will eliminate more jobs than it creates, requiring fundamental restructuring of the relationship between work and income.”

16.8.1 What the Evidence Shows

The task-based framework for analyzing automation, developed by economists David Autor, Frank Levy, and Richard Murnane, provides the most useful analytical lens. Jobs consist of tasks, and tasks differ in their susceptibility to automation. Tasks that are routine, rule-based, and well-defined are most susceptible. Tasks that require judgment, creativity, physical dexterity in unstructured environments, or complex interpersonal interaction are least susceptible.

Previous waves of automation — from mechanization through computerization — primarily substituted for routine manual and cognitive tasks while complementing non-routine tasks. The result was “hollowing out” of the labor market: growth at the top (high-skill, high-wage cognitive workers) and the bottom (low-skill, low-wage service workers) relative to the middle (routine cognitive and manual workers).

Current AI is doing something different. Large language models are showing the ability to perform non-routine cognitive tasks at sophisticated levels — legal research, medical diagnosis support, financial analysis, code generation, content creation. These are not the routine cognitive tasks that were automated in previous waves; they are the tasks that augmented skilled workers and enabled their productivity.

The historical analogy that optimists reach for is agricultural automation: mechanization destroyed agricultural employment but created more jobs in industry and services as productivity gains created new economic activity. The historical analogy that pessimists reach for is the Industrial Revolution: genuine transformation, enormously beneficial in aggregate, but accompanied by decades of dislocation, immiseration, and social disruption for the people who bore the transition costs.

Both analogies capture something real. The question is whether the pace of current AI development allows time for economic adaptation and skills transition, or whether it outstrips the economy’s capacity to absorb the disruption.

16.8.2 Tasks, Jobs, and Augmentation

The distinction between task automation and job automation is crucial. Most jobs consist of many different tasks; automating some tasks does not necessarily eliminate the job. A radiologist’s job includes reviewing images, communicating with patients and referring physicians, integrating imaging findings with clinical context, managing a department, teaching residents, and participating in multidisciplinary tumor boards. AI may automate the image-review task; it does not, in the near term, automate the rest of the job.

The more productive framing — empirically supported across a number of studies — is complementarity: the productivity gains from human-AI collaboration often exceed those from either humans or AI alone. AI is fastest and most consistent; humans are most adaptable, empathetic, and able to exercise judgment in genuinely novel situations. Tasks structured to exploit this complementarity — AI as a research assistant, a first-pass analyst, a code drafter, a writing partner — show the most consistent productivity gains.

The 2023 study by MIT economists Noy and Zhang found that ChatGPT usage increased worker productivity by 37% on a set of professional writing tasks, with the largest gains for lower-skilled workers — who benefited most from the AI’s ability to raise their baseline quality. A Goldman Sachs analysis estimated that AI could automate tasks equivalent to 300 million full-time jobs globally, while also creating new roles and increasing productivity. Neither number tells you what will actually happen; both are useful inputs to thinking about what policy responses are warranted.

16.8.3 Policy Responses

The policy debate around AI and work is nascent and contested. Several categories of response are being discussed:

Education and skills: Rapid AI capabilities mean that the skills valued in the labor market are changing faster than education systems can adapt. Investment in adult retraining, AI literacy at all educational levels, and flexible credentialing systems are broadly supported, though the specific design questions are difficult.

Labor market regulation: Some jurisdictions are requiring transparency about AI use in employment decisions (what EU law already requires for consequential automated decisions), notice requirements when AI systems are used in the workplace, and limits on AI-driven surveillance of workers. These protections address current harms without foreclosing the productivity benefits of AI.

Social insurance: If AI produces significant economic dislocation — even if it creates as many jobs as it destroys, the transition costs are real and unequally distributed — social insurance systems must be adequate to support displaced workers through transition periods. Universal basic income proposals, wage insurance, and expanded unemployment systems are all being discussed, though implementation challenges are substantial.

Taxation and redistribution: The productivity gains from AI accrue primarily to capital owners (the companies that develop and deploy AI) and highly skilled workers (who are complemented by AI). The distributional consequences — increased returns to capital, potentially stagnant or declining wages for many workers — raise questions about whether existing tax systems are adequate to fund the public goods and social insurance that the transition requires.

16.9 Section 6 — Human-AI Collaboration: The Productive Frontier

The most consistently productive use of AI in the near term is not AI replacing humans — it is AI augmenting humans. Understanding the conditions under which augmentation works, and the conditions under which it fails or backfires, is practical knowledge for anyone deploying AI systems.

16.9.1 When Augmentation Works

Augmentation tends to work well when:

The AI provides complementary capabilities — speed, scale, consistency, pattern recognition across large datasets — that the human cannot easily provide alone, while the human provides judgment, context, and accountability that the AI cannot reliably provide.

The human can verify the AI’s outputs — detect errors, recognize hallucinations, catch outputs that are superficially plausible but wrong. AI augmentation consistently fails when humans cannot or do not verify AI outputs and treat them as reliable without critical review. This is the automation bias problem: people are more likely to accept AI outputs without critical scrutiny than they are to accept human outputs, even when the AI is making systematic errors.

The workflow allows meaningful human agency — the human can actually use the AI’s outputs as a starting point, modify them, override them, or discard them. Systems designed to present AI recommendations as authoritative — without easy mechanisms for humans to exercise judgment — tend to induce automation bias and to miss the failure cases that human judgment would catch.

The AI’s uncertainty is communicated honestly. Systems that express appropriate confidence are more useful than systems that are uniformly confident. A radiology AI that says “I am highly confident of pneumonia in the right lower lobe — please review” is more useful than one that simply lists a diagnosis with no confidence indication, and more useful than one that expresses high confidence when it is actually uncertain.

16.9.2 When Augmentation Fails

Augmentation tends to fail when:

The task requires causal reasoning or counterfactual thinking that the AI performs poorly. Using an AI to predict which customers will churn may work well (pattern recognition from historical data); using it to predict how customers will respond to a specific intervention requires causal reasoning that current systems may not provide.

The feedback loop is broken: the human accepts AI outputs without seeing the consequences of errors, so automation bias accumulates without the experience that would correct it. Clinicians who follow AI recommendations without tracking patient outcomes are in this situation; those who track outcomes can learn which AI recommendations to trust and which to scrutinize.

Deskilling occurs: humans stop developing or maintaining the skills that AI augments, and lose the ability to function when AI is unavailable or incorrect. Airline pilots who rely on autopilot lose proficiency in manual flying; workers who rely on AI for routine analysis may lose the ability to catch AI errors. The appropriate response is deliberate maintenance of human capability alongside AI augmentation, not the assumption that AI will always be available and correct.

16.10 Section 7 — Open Questions: What the Next Generation Will Work On

The most important ideas in AI’s next decade are probably not yet visible. But the open problems are clear enough to sketch:

Mechanistic interpretability at scale: Understanding what representations and computations large models have actually learned — not post-hoc approximations, but the actual mechanisms. The field of mechanistic interpretability (Anthropic, MIT, others) has made early progress on small models and specific circuits; extending this to frontier-scale models remains an open challenge.

Reliable reasoning: Building systems that reason correctly and verifiably, not systems that produce plausible-sounding reasoning chains that may contain errors. Formal verification of AI reasoning traces, neurosymbolic integration, and self-consistency checking are active research directions.

Efficient grounding: Connecting language to perception and action in ways that provide the kind of understanding that pure language training does not, without requiring the enormous embodied experience that biological agents use. Robotics-language integration is one frontier; multimodal training on interleaved text and sensory data is another.

Robustness to distribution shift: Systems that maintain reliable performance when the deployment distribution differs from the training distribution — which it always does, to some degree, in the real world. This requires either better generalization from training, better detection of distribution shift, or graceful degradation when it occurs.

Long-horizon reasoning and planning: Current systems struggle with tasks that require reasoning over long time horizons, maintaining consistent goals across many steps, and planning sequences of actions in complex environments. This is the open problem most directly relevant to autonomous AI agents.

Value learning and preference specification: Finding ways to specify what humans want that are robust to optimization pressure — the alignment problem restated as a research agenda. Debate continues between approaches based on learned reward models, constitutional principles, debate, and market mechanisms.

Energy efficiency: Building AI systems that achieve current capabilities with dramatically less energy — through more efficient architectures, more efficient hardware, and better understanding of what computation is actually necessary for intelligence. Both economically and environmentally important.

16.11 Section 8 — The IAAIS Final Capstone Presentation

Your semester has been a guided construction project. From the search algorithms of Chapter 2 to the deployment infrastructure of Chapter 15, you have built, module by module, a system capable of intelligent adaptive behavior in your chosen domain. The final capstone presentation is the opportunity to step back from the construction and present the complete building.

16.11.1 What the Presentation Covers

A complete IAAIS capstone presentation has five parts, each approximately 5–7 minutes, for a total of 25–35 minutes with time for questions:

Part 1 — Domain and Problem Statement: What problem does your IAAIS system address? Who are the stakeholders? What decisions does it support? Why is AI an appropriate tool for this problem, and what capabilities does each module contribute? This part should be accessible to a non-technical audience and should clearly articulate the value the system provides and the risks it manages.

Part 2 — System Architecture: A technical walkthrough of your complete IAAIS architecture. Show the integration diagram with all thirteen modules and their connections. Describe the data flows between modules. Identify the two or three module interactions that you found most architecturally interesting — places where the outputs of one module become the inputs of another in non-trivial ways.

Part 3 — Live Demonstration: A live demonstration of your deployed IAAIS system through the Streamlit interface. Demonstrate at minimum: a complete end-to-end query through the Generative Interface that engages at least three underlying modules; a Classifier prediction with SHAP explanation; and the System Monitor dashboard. Be prepared to explain any failure modes that occur during the demonstration — live systems are imperfect, and handling failure gracefully is itself a demonstration of good engineering.

Part 4 — Ethics Audit Results: Present the key findings from your Chapter 14 ethics audit. What fairness disparities did you find? What fairness criterion did you select, and what tradeoff does it involve? What were the most serious red-team failures? What residual risks remain in your deployed system, and what is your monitoring plan for them? This part should be honest about limitations — a presentation that claims no significant fairness concerns or failure modes is not credible.

Part 5 — Reflections and Future Work: What did you learn? What would you do differently? What is the most important capability your system lacks, and what would you need to build it? How does your system connect to the open problems described in this chapter — and what research direction would you pursue if you were continuing this work as a doctoral student?

16.11.2 Evaluation Criteria

A strong capstone presentation is evaluated across six dimensions:

Dimension	What evaluators look for
Technical depth	Accurate, detailed understanding of how each module works and why design choices were made
Integration quality	Evidence that modules work together, not just individually — real inter-module data flows
Demonstration reliability	System runs live; failures are handled gracefully and explained honestly
Ethics audit honesty	Real audit findings, real tradeoffs acknowledged, real residual risks identified
Communication clarity	Accessible to non-technical stakeholders; technical content explained without jargon
Reflective quality	Genuine engagement with what was learned, what failed, what would be done differently

16.11.3 A Note on Honesty

The most common weakness in capstone presentations is the absence of honest limitation acknowledgment. Students who claim their system has no significant fairness concerns, that all modules work perfectly, and that the ethics audit found nothing of concern are not presenting their system honestly — they are presenting a version of their system that does not exist.

The systems built in this course are impressive achievements. They are also first implementations with real limitations. The faculty evaluating your presentation are not looking for a perfect system. They are looking for evidence that you understand what you built, what it can do, what it cannot do, and what responsibilities come with deploying it.

The AI practitioners who are most valuable to employers and most trustworthy to users are not those who overstate their systems’ capabilities. They are those who understand the boundary between capability and limitation, communicate that boundary clearly, and build the monitoring and governance structures that make deployment safe even in the presence of limitations.

16.12 Hands-On Exploration: Your Capstone Presentation

16.12.1 The Activity

There is no lab notebook for Chapter 16. The hands-on component of this chapter is the preparation and delivery of your IAAIS Final Capstone Presentation.

Preparation checklist:

☐ Complete the IAAIS Full Integration Sprint (Chapter 15): all modules connected, Streamlit UI running, monitoring dashboard functional.

☐ Complete the IAAIS Ethics Audit (Chapter 14): fairness metrics computed across demographic subgroups, red-team testing documented, System Card complete.

☐ Prepare a 5-slide architecture diagram showing all thirteen modules and their data flows.

☐ Script the live demonstration sequence: which queries will you run, which modules will they engage, what outputs will you show.

☐ Prepare a one-page summary of your ethics audit findings: the three most significant findings and the mitigations you have applied.

☐ Practice the full presentation at least twice: once with slides only, once with live system demonstration.

Submission requirements:

Slide deck (maximum 20 slides)
GitHub repository with complete IAAIS codebase, documentation, and System Card
Ethics Audit report (from Chapter 14)
Brief reflection document (500–750 words): what you learned, what you would do differently, and what open question in AI you would most like to investigate further

16.13 Case Study: AlphaFold — A Preview of What AI Can Do for Science

16.13.1 The Problem

Proteins are the molecular machines of life. They fold into precise three-dimensional shapes, and their shape determines their function. Knowing a protein’s shape tells biologists how it interacts with other molecules, what diseases might result from mutations that alter its structure, and what drugs might bind to it.

Determining protein structures experimentally — through X-ray crystallography, cryo-electron microscopy, or NMR — is slow and expensive. The gap between the number of known protein sequences and the number of proteins with experimentally determined structures was, as of 2020, enormous: hundreds of millions of sequences, roughly 180,000 structures. For most proteins, the structure was unknown.

16.13.2 The Solution

AlphaFold 2, published by DeepMind in 2021, achieved accuracy in protein structure prediction equivalent to experimental methods — essentially solving the “protein folding problem” that had been a grand challenge in biology for fifty years. The system used a novel deep learning architecture — the Evoformer — that processed evolutionary information (how protein sequences vary across species) alongside structural information to produce highly accurate three-dimensional coordinate predictions.

The impact was immediate and enormous. DeepMind released the predicted structures of 200 million proteins — virtually every protein in the known universe — freely to the scientific community. Within months, researchers were using AlphaFold predictions to design new enzymes, understand disease mechanisms, and identify drug targets for diseases that had resisted treatment for decades.

16.13.3 What AlphaFold Demonstrates

AlphaFold is significant not just as a scientific achievement but as a demonstration of what AI can do when applied to the right problem with the right data. The protein folding problem had the properties that make machine learning tractable: vast amounts of training data (protein sequence-structure pairs), a clear evaluation metric (structural accuracy), and a problem domain where the underlying rules — the laws of physics governing molecular interactions — are fixed and universal.

Not every important scientific problem has these properties. Climate prediction, drug efficacy in humans, and social policy effectiveness are all domains where the training data is limited, the evaluation metrics are contested, and the underlying rules include human behavior and social dynamics that are neither fixed nor universal. Understanding which scientific problems are well-suited to AI and which require different approaches is one of the important research questions of the next decade.

AlphaFold also demonstrates the value of scientific data sharing. Its training depended on the Protein Data Bank, a freely available repository of experimental protein structures that the scientific community has contributed to for fifty years. The research norms of biology — sharing experimental data freely — created the resource that made AlphaFold possible. Research norms in other domains — where data is proprietary or closely held — may constrain the equivalent AlphaFold-scale breakthroughs from happening.

16.14 Chapter Summary

We began this chapter at an inflection point — the spring of 2023, when AI had become simultaneously more capable than most people expected and more limited than most coverage acknowledged. We end it with an honest accounting of both sides: what AI can do, what it cannot yet do, and the forces that will shape the distance between those two realities in the years ahead.

The open problems reminded us that current AI systems, impressive as they are, have structural limitations in reasoning, causality, grounding, and sample efficiency that scaling alone may not address. The AGI debate reminded us that the most consequential questions in AI development are genuinely uncertain, and that confident timelines in either direction are not warranted by the current state of the science.

The climate section held the double ledger: AI’s growing energy footprint alongside its genuine contributions to climate science, materials discovery, and energy optimization. The governance section described the coordination problem — global technology, national regulation, insufficient multilateral frameworks — and the early institutional responses that are beginning to address it.

The future of work explored the empirical evidence for what AI is doing to labor markets: automating some tasks, augmenting others, creating new ones, and producing distributional consequences that require policy responses. Human-AI collaboration described the conditions under which AI augmentation actually improves outcomes — and the conditions under which it fails or backfires.

The capstone presentation asked you to do the hardest thing: stand up, show what you built, and be honest about what it can and cannot do. This is, ultimately, what responsible AI practice looks like. Not a demonstration of flawless capability, but a clear-eyed presentation of real capability and real limitation, grounded in evidence, acknowledging uncertainty, and committed to the ongoing work of making the system better.

The course is complete. The work is not. The systems you have built this semester are beginnings, not endings — prototypes for systems that will improve as you develop your skills, as the field advances, and as the domains you work in accumulate the experience and data that better AI requires. The questions this course has raised — about what intelligence is, how it can be built, what it can do, and what responsibilities come with building it — are questions you will spend the rest of your career working on.

Welcome to the field.

16.15 Discussion Questions

The AGI timeline: You have heard arguments from both AI optimists (who believe AGI is decades away) and skeptics (who believe current architectures have fundamental limitations). What would you need to observe — in terms of AI capabilities or scientific understanding — to update significantly toward the optimist position? Toward the skeptic position?
AI and climate: A major AI company announces it will train its next frontier model using 100% renewable electricity. A critic responds that the company’s announcement merely shifts fossil fuel consumption to other users on the same grid rather than adding new renewable capacity. Evaluate this critique. What would constitute a genuinely carbon-neutral frontier model training run?
Governance and the coordination problem: The US, EU, and China have adopted different AI regulatory frameworks that reflect different values and strategic interests. Identify one domain — say, facial recognition in law enforcement — where these frameworks produce meaningfully different outcomes. Who bears the costs of regulatory divergence? Who benefits?
Automation and distribution: Suppose AI produces 20% economy-wide productivity growth over the next decade, while also automating tasks equivalent to 15% of current employment. If the productivity gains primarily accrue to capital owners and high-skilled workers, and the employment losses primarily affect middle-skill workers, describe the distributional consequences. What policy responses would you advocate?
Complementarity and automation bias: A study finds that radiologists who use an AI diagnostic assistant make fewer errors on cases where the AI is correct, but also accept incorrect AI diagnoses more often than they would make the same incorrect diagnosis independently. Net effect: comparable error rate to unassisted radiologists. Does this mean the AI is useless? Harmful? What would you change about the human-AI workflow to improve the outcome?
Open problems and your IAAIS system: Which of the open problems described in Section 1 — reasoning gap, causality, grounding, sample efficiency — is most relevant to your IAAIS domain? Give a concrete example of a case where your IAAIS system’s inability to address that problem produces a meaningful limitation. What research development would most improve your system’s capabilities?
AlphaFold and data sharing: AlphaFold was made possible by the Protein Data Bank, a fifty-year accumulation of freely shared experimental data. Identify a domain in your field where equivalent freely available data exists. Then identify a domain where proprietary data concentration is preventing an AlphaFold-equivalent breakthrough. What policy or institutional changes would shift the second domain toward the first?
Your reflection: This course has traced the history of AI from search algorithms through expert systems, machine learning, deep learning, and generative AI — arriving at frontier models and the open questions of the present. What idea from this course most surprised you? What assumption did you arrive with that was most challenged? And what question, raised somewhere in these sixteen chapters, do you most want to spend your career answering?

16.16 Further Reading

16.16.1 Open Problems and Frontiers

Marcus, G. (2022). Deep learning is hitting a wall. Nautilus. A critical perspective on current AI limitations — useful counterpoint to the dominant narrative of continuous progress.

LeCun, Y. (2022). A path towards autonomous machine intelligence. openreview.net. LeCun’s proposal for world models and how to address the limitations of current AI.

Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2017). Building machines that learn and think like people. Behavioral and Brain Sciences, 40. The cognitive science perspective on what AI is missing.

16.16.2 AGI

Bostrom, N. (2014). Superintelligence: Paths, Dangers, Strategies. Oxford University Press. The foundational long-termist argument — read critically.

Mitchell, M. (2019). Artificial Intelligence: A Guide for Thinking Humans. Farrar, Straus and Giroux. An accessible and honest assessment of what AI can and cannot do.

Chollet, F. (2019). On the measure of intelligence. arXiv:1911.01547. A careful attempt to define general intelligence in a measurable way.

16.16.3 AI and Climate

Rolnick, D., et al. (2022). Tackling climate change with machine learning. ACM Computing Surveys, 55(2). The comprehensive survey of AI applications in climate science and clean energy.

Patterson, D., et al. (2021). Carbon and the big picture perspective. arXiv:2104.10350. Analysis of the energy and carbon footprint of training large AI models.

Jumper, J., et al. (2021). Highly accurate protein structure prediction with AlphaFold. Nature, 596, 583–589. The AlphaFold paper.

16.16.4 Governance

Dafoe, A. (2018). AI governance: A research agenda. Future of Humanity Institute, University of Oxford. A framework for thinking about the governance challenges.

Calo, R. (2017). Artificial intelligence policy: A primer and roadmap. UC Davis Law Review, 51(2). Accessible policy overview.

Roberts, H., et al. (2021). The Chinese approach to AI governance. Oxford Internet Institute. A comparative perspective on China’s AI regulatory approach.

16.16.5 Future of Work

Autor, D., Levy, F., & Murnane, R. J. (2003). The skill content of recent technological change. Quarterly Journal of Economics, 118(4). The task-based framework for analyzing automation.

Noy, S., & Zhang, W. (2023). Experimental evidence on the productivity effects of generative artificial intelligence. Science, 381(6654), 187–192. The MIT study on ChatGPT’s productivity effects.

Acemoglu, D., & Restrepo, P. (2022). Tasks, automation, and the rise in US wage inequality. Econometrica, 90(5), 1973–2016. Rigorous economic analysis of automation’s distributional effects.

— End of Chapter 16 —

This completes the CSC5350 Artificial Intelligence Open Educational Resource Textbook, First Edition.

Chapters 1–16 cover the full arc from the foundational concepts of intelligent agents and search through classical reasoning, probabilistic inference, machine learning, deep learning, natural language processing, computer vision, reinforcement learning, expert systems, generative AI, AI safety and ethics, production deployment, and the open frontiers of the field.

The Intelligent Adaptive AI System (IAAIS) capstone project, developed across all sixteen chapters, serves as both a pedagogical spine and a portfolio artifact — a demonstration that the concepts of this course are not merely theoretical but can be assembled into a coherent, deployed intelligent system.

— Professor Moody Amakobe, Global Data Science Institute