The AI Fairness Gap: How We Judge Machines in Economic Games

Author: Denis Avetisyan

New research reveals people hold large language models to different standards of fairness than humans when it comes to resource allocation and economic interactions.

Studies using the Ultimatum and Dictator Games demonstrate that social norms applied to AI agents differ significantly from those applied to human players.

While increasingly integrated into daily life, the delegation of decisions to large language models (LLMs) raises questions about evolving social expectations. This research, ‘Do people expect different behavior from large language models acting on their behalf? Evidence from norm elicitations in two canonical economic games’, investigates whether individuals apply distinct norms when assessing fairness in economic interactions performed by LLMs versus humans. Results from experiments using the Ultimatum and Dictator Games demonstrate that people expect machines to adhere to different standards of resource allocation, though they do not necessarily view enforcement of those norms as inappropriate. As LLMs become more prevalent in decision-making, will these diverging expectations shape the future of human-machine collaboration and trust?

Beyond Rationality: The Roots of Fairness

Conventional economic theory posits that individuals make decisions based on rational self-interest, striving to maximize personal gain. However, behavioral studies consistently demonstrate deviations from this model, particularly when considering fairness. Research indicates that people frequently reject offers perceived as unfair, even if accepting those offers would result in a net benefit. This suggests a willingness to incur personal costs-foregoing potential gains-to punish perceived inequity. The phenomenon challenges the assumption of purely self-interested behavior, implying that social preferences and a concern for fairness play a significant role in human decision-making, often overriding the pursuit of maximum economic advantage.

The Ultimatum Game provides compelling evidence that humans aren’t simply motivated by maximizing personal profit. In this classic experiment, one player proposes how to divide a sum of money, while the other player can either accept the offer – receiving the proposed amount – or reject it, leaving both with nothing. Repeatedly, studies show that recipients will often reject offers they perceive as unfair – even if rejecting means receiving no money at all. This seemingly irrational behavior highlights the powerful role of fairness and social norms in human decision-making, suggesting a deeply ingrained aversion to inequity that overrides purely economic incentives. The consistent results across diverse cultures demonstrate this isn’t a quirk of a specific population, but a fundamental aspect of how people evaluate and respond to offers, implying that considerations beyond financial gain are central to social interactions.

Predicting human behavior extends far beyond simple economic calculations, necessitating an understanding of the social preferences that routinely override purely rational self-interest. This is increasingly vital not only within traditional social and economic contexts – such as negotiations, market dynamics, or charitable giving – but also as artificial intelligence systems become more integrated into daily life. Accurate modeling of human responses requires accounting for these ingrained sensitivities to fairness, cooperation, and reciprocity; an AI that consistently fails to recognize or respect these norms will likely encounter resistance, distrust, and ultimately, failure in any scenario involving human interaction. Consequently, incorporating these behavioral insights into AI design is crucial for building systems that are not only intelligent, but also effective and acceptable to those they serve.

Human choices frequently diverge from predictions based purely on maximizing personal benefit, indicating that deeply rooted social preferences significantly shape decision-making processes. Research reveals individuals often prioritize fairness and equity, even when accepting a suboptimal outcome for themselves rather than tolerating perceived injustice. This suggests an inherent concern for social norms and a desire to maintain equitable relationships, operating alongside, and sometimes overriding, the pursuit of individual gain. These preferences aren’t simply learned behaviors; evidence points to possible evolutionary origins, implying a biological predisposition towards cooperation and aversion to exploitation. Consequently, a complete understanding of human behavior requires acknowledging these ingrained social considerations, moving beyond models focused solely on rational self-interest.

Mapping Social Expectations: The Norm Elicitation Task

The Krupka-Weber Norm Elicitation Task is a structured methodology used to quantitatively assess perceptions of acceptable behavior within a given social context. Typically, participants are presented with hypothetical scenarios involving a principal and a potential recipient, and are asked to determine a fair allocation of a resource. Researchers then calculate the median acceptable offer as a proxy for the prevailing social norm. Variations of the task involve different resource types and social contexts, allowing for the identification of nuanced normative beliefs. The resulting data provides a measurable value representing the shared understanding of what constitutes equitable or appropriate behavior, moving beyond simple observation of actions to directly gauge underlying beliefs.

Traditional behavioral observation focuses on documenting what people do, but norm elicitation tasks are designed to ascertain perceptions of what ought to be done. These tasks present scenarios – often involving resource allocation or behavioral infractions – and directly ask participants to evaluate the fairness or appropriateness of different actions. This differs from simply recording observed behavior, as it aims to capture subjective beliefs about ideal conduct, even if those beliefs aren’t consistently reflected in actual actions. The resulting data provides a measure of injunctive norms – standards of behavior explicitly or implicitly communicated by a group – supplementing observational data with insights into underlying value judgments and expectations.

Social norms exert a substantial influence on both prosocial behavior and resource allocation. Research indicates individuals are significantly more likely to cooperate and contribute to collective benefits when they believe others share a commitment to fairness and reciprocity – perceptions directly shaped by prevailing norms. Specifically, the perceived social norm regarding contribution levels strongly predicts individual giving in public goods games and similar scenarios. Furthermore, norms dictate not only whether resources are shared, but how they are distributed; individuals often favor allocations that align with perceived normative standards of equity or need, even if those allocations are not Pareto optimal. Consequently, accurate assessment of these norms is crucial for predicting and potentially modifying behavior in contexts involving collective action and distributive justice.

Accurate identification of prevailing social norms enables improved prediction of behavioral responses within social dilemmas, as individuals frequently base decisions on perceptions of group expectations rather than purely rational self-interest. This predictive capability stems from the demonstrated correlation between perceived norms and actual behavior; interventions designed to leverage these norms – by highlighting existing prosocial expectations or subtly shifting perceptions – can therefore influence outcomes in scenarios involving cooperation, resource allocation, and collective action. Furthermore, understanding the specific normative landscape allows for targeted interventions, addressing discrepancies between desired and actual behavior by focusing on the underlying beliefs driving individual choices.

Delegated Reasoning: LLMs as Agents of Social Understanding

The application of Large Language Models (LLMs) as agents within established economic games provides a new methodology for the study of social norms. Traditionally, research in this area relies on human subjects responding to incentivized scenarios; however, LLMs can now function as ‘delegates’ performing the role of a player in these same games. This allows researchers to observe LLM decision-making – specifically, how an LLM allocates resources or responds to fairness considerations – and analyze these responses as indicators of the model’s implicit understanding of societal expectations regarding equitable behavior. By treating LLMs as participants, researchers can isolate and examine the normative reasoning processes embedded within these models, offering a complementary approach to traditional behavioral studies.

Analysis of Large Language Model (LLM) responses to economic scenarios involving resource allocation provides a method for evaluating their internalized understanding of social expectations. By presenting LLMs with tasks requiring decisions about fair distribution – such as dividing a sum of money between themselves and another party – researchers can observe patterns in their choices. These observed patterns are then analyzed to determine the extent to which the LLM’s behavior aligns with established norms of fairness, equity, and reciprocity as observed in human populations. The resulting data offers a quantifiable measure of the LLM’s implicit grasp of these complex social constructs, independent of explicitly programmed rules or training data focused on fairness definitions.

Utilizing Large Language Models (LLMs) as delegates in experimental economics research offers a means of reducing participant burden. Traditional behavioral studies often require significant cognitive effort from human subjects to process complex scenarios and make decisions, potentially introducing fatigue or bias. By shifting some decision-making responsibility to LLMs, researchers can simplify tasks for human participants, focusing their evaluation on the outputs of the LLM rather than requiring them to independently calculate optimal strategies. This approach minimizes the cognitive load on participants, allowing for larger sample sizes and more efficient data collection, while also potentially reducing emotional responses that could skew results in scenarios involving resource allocation or perceived fairness.

Research indicates a divergence in the application of fairness norms when evaluating resource allocation decisions made by Large Language Models (LLMs) versus those made by humans. Participants in our study exhibited a tendency to judge LLM-generated allocations based on an expectation of neutrality, penalizing both excessive generosity and stinginess. This contrasts with evaluations of human actors, where deviations from equitable splits are often attributed to individual character or intent. The observed pattern suggests that individuals implicitly hold LLMs to a different standard of impartiality, anticipating a more calculated and less emotionally-driven approach to resource distribution than they would expect from another person.

Navigating Cultural Landscapes: The Future of AI and Social Norms

Perceptions of fairness and acceptable conduct are not universal; rather, they are deeply rooted in cultural context, particularly that of Western societies which often dominates the datasets used to train large language models. This presents a significant challenge when interpreting the results of studies utilizing these models, as observed behaviors might reflect Western-centric biases rather than objective truths about human interaction. For example, concepts of reciprocity, distributive justice, and even appropriate levels of assertiveness can vary dramatically across cultures, meaning an LLM trained primarily on Western data may incorrectly assess actions from other cultural perspectives as unfair or unreasonable. Consequently, researchers must critically evaluate whether observed differences in LLM behavior represent genuine insights into human social norms, or simply a reflection of the cultural lens through which the AI has learned to perceive the world.

The development of truly equitable artificial intelligence demands a rigorous consideration of cultural influences on social norms and perceptions of fairness. AI systems are trained on data reflecting specific cultural viewpoints, potentially embedding and amplifying existing biases if these origins remain unexamined. A failure to account for diverse cultural expectations can lead to AI that misinterprets behaviors, delivers unjust outcomes, or is perceived as inappropriate across different societies. Consequently, proactive efforts to incorporate cross-cultural perspectives into AI design, data curation, and evaluation are essential – not merely to avoid perpetuating harm, but to foster AI that genuinely respects and adapts to the multifaceted nature of human social interaction. This necessitates moving beyond a singular definition of ‘fairness’ and embracing a framework that acknowledges and navigates the spectrum of culturally-defined norms.

Large language models offer a unique opportunity to investigate the complexities of human social norms, acting as computational mirrors reflecting – and potentially clarifying – deeply ingrained behavioral expectations. However, the insights gleaned from these models are inextricably linked to the cultural frameworks embedded within their training data and the interpretations applied by researchers. Without careful consideration of this cultural lens, analyses risk mistaking culturally specific behaviors as universal truths, or conversely, dismissing legitimate variations in social conduct. The true power of LLMs lies not simply in replicating observed norms, but in their capacity to expose the subtle nuances and contextual dependencies that shape them, thereby fostering a more comprehensive and culturally sensitive understanding of human interaction – a process demanding rigorous attention to the biases inherent in both the data and the analytical approach.

Statistical analyses revealed a marked divergence in how humans and large language models are perceived when evaluating fairness in interactive scenarios. Both Experiment 1 and Experiment 2 demonstrated statistically significant differences in appropriateness ratings – with a p-value less than .001 – indicating that expectations for AI behavior do not align with those for human actors. This disparity was not merely statistical; the Cliff’s Delta of 0.109 observed in Experiment 2 represents a substantial effect size, suggesting a meaningful difference in the acceptance of generous offers depending on whether they originated from a human or an AI proposer. These findings underscore a nuanced expectation for AI fairness, implying that individuals hold AI to a different standard – or at least, perceive deviations from expected behavior differently – than they do for other humans in comparable situations.

The study highlights a fascinating asymmetry in human expectation. People readily apply nuanced social norms-standards of fairness and equitable distribution-when evaluating human behavior, yet seem to demand a different, more rigid adherence to principle from large language models. This expectation of algorithmic impartiality, while seemingly logical, reveals a subtle desire for predictable systems. As Vinton Cerf observed, “Any sufficiently advanced technology is indistinguishable from magic.” The research suggests that this ‘magic’ isn’t free from scrutiny; rather, it’s subject to a new set of expectations, a demand for consistent, rule-based behavior that often surpasses what is expected of fallible humans. If the system looks clever, it’s probably fragile, and this fragility stems from the impossibility of perfectly encoding human ethical complexities.

Where Do We Go From Here?

The finding that individuals readily apply distinct normative expectations to large language models-expecting a particular flavor of ‘rational’ allocation-raises a crucial question. It is not simply that differing expectations exist, but rather, what are these expectations optimizing for? Is the observed preference for LLM ‘fairness’ a genuine moral judgement, or a reflection of instrumental beliefs about predictability and control? The research highlights a tendency to view these systems not as agents with internal states, but as tools to be calibrated for desired outcomes, a subtly dangerous simplification.

Future work must move beyond merely documenting the existence of these divergent norms. A deeper understanding requires disentangling the cognitive mechanisms at play: are these expectations consciously held, or emerge from more implicit processes? Furthermore, the choice of economic games-canonical as they are-limits the scope of inquiry. Do these patterns generalize to more complex social interactions, where the ‘rules’ are less clearly defined and the stakes are higher?

Simplicity, in this context, is not minimalism. It is the discipline of distinguishing the essential from the accidental. The study provides a valuable starting point, but a complete picture necessitates considering the broader ecosystem of AI interaction-the interplay between human beliefs, algorithmic design, and the evolving social contract governing these increasingly pervasive systems.

Original article: https://arxiv.org/pdf/2601.15312.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/

Beyond Rationality: The Roots of Fairness

Mapping Social Expectations: The Norm Elicitation Task

Delegated Reasoning: LLMs as Agents of Social Understanding

Navigating Cultural Landscapes: The Future of AI and Social Norms

Where Do We Go From Here?

See also: