The Question Hiding in Plain Sight

Every AI system encodes values. There is no neutral option. A system that recommends content has made choices about what counts as relevant. A system that flags harmful speech has made choices about what counts as harmful. A system that summarizes a debate has made choices about which positions deserve how much space. A hiring system has made choices about which attributes predict job performance. These choices are made by people — researchers, engineers, product managers, executives — mostly without public accountability, often without awareness that they are making value choices rather than technical ones. The question of who gets to make those choices, and through what process, is arguably the most consequential governance question of the current moment. It is also the question that receives the least attention relative to its importance.

How Values Currently Get Into AI Systems

Values enter AI systems through several channels, almost none of which involve explicit deliberation. They enter through the choice of training data — which texts, images, and interactions are used to shape what the system considers normal, appropriate, and true. They enter through the definition of the objective function — what the system is rewarded for achieving. They enter through design choices about what the system will and will not do. They enter through the evaluation criteria used to decide whether a system is ready to deploy. Each of these channels represents a site of value choice. Each is currently governed primarily by the preferences and assumptions of small teams at technology companies, shaped by competitive and financial pressures that are not well-aligned with public interest.

The Governance Models on the Table

Several different governance models have been proposed for how AI values should be determined. Each has genuine merits and genuine problems. Pure market governance argues that users should choose among competing AI systems with different value profiles, and competition will drive systems toward what people actually want. The problem is that users rarely have meaningful information about the values embedded in AI systems, and markets are notoriously bad at representing interests that are diffuse, future-oriented, or shared with non-users. Expert governance argues that technical and ethical experts should determine AI values, drawing on the best available knowledge. The problem is that experts are not representative, have their own blind spots and interests, and have a mixed historical record on value questions that extend beyond their expertise. Democratic governance argues that AI values should be determined through some form of public deliberation. The problem is that meaningful public deliberation about complex technical systems is very hard to organize, slow relative to technology development timelines, and subject to manipulation. Research from the Carnegie Endowment for International Peace examining AI governance proposals across countries has found that no single governance model performs well on all relevant criteria — accountability, expertise, speed, representativeness, and legitimacy trade off against each other in ways that make any pure model inadequate.

The Tangent: Indigenous Data Governance as a Model

There are communities that have spent decades developing frameworks for collective governance of information and knowledge. Indigenous data governance movements — particularly the CARE Principles for Indigenous Data Governance (Collective Benefit, Authority to Control, Responsibility, Ethics) — developed in response to a long history of researchers extracting indigenous knowledge without consent or benefit-sharing. The frameworks they have developed are sophisticated responses to exactly the problems AI governance faces: how to make consequential decisions about shared resources through legitimate collective processes, how to represent interests that are not well-captured by market mechanisms, how to protect against extraction by powerful outside actors. These frameworks are rarely cited in mainstream AI governance discussions. They should be.

Who Is Asking This Question Now

The current answer to "who decides AI values" is: the companies building the systems, constrained loosely by voluntary commitments, national regulations in a few jurisdictions, and market pressure. This is a governance arrangement that has emerged by default rather than by design. Research from the Alan Turing Institute examining power concentration in AI development has documented how a small number of actors — primarily large technology companies in the United States and China — are making value choices that affect billions of people across every country. The geographic and demographic concentration of these actors is extreme. The accountability mechanisms are minimal.

Why the Question Matters More Than the Answer

There is no perfect answer to who should decide AI values. Every answer involves tradeoffs and will produce imperfect outcomes. But the process through which the answer is reached matters enormously for legitimacy. When powerful systems that affect everyone's life are built by unaccountable actors who have made no effort to consult the people affected, the resulting systems may be technically excellent and still be experienced as impositions. Legitimacy requires some connection between those who make choices and those who live with their consequences. Building that connection is hard, slow, and necessary. The alternative is AI systems that are capable without being trusted — a situation that serves no one well.