When Bots and Humans Clash: The Dynamics of Imperfect Cooperation

Author: Denis Avetisyan

New research explores how personality traits in both people and AI systems shape interactions when cooperation isn’t seamless.

A dual framework-simulated interactions and user studies-assesses how characteristics of both AI agents-specifically Transparency, Warmth, Expertise, Adaptability, and Theory of Mind-and human users influence collaborative outcomes, with causal analysis revealing the interplay between these attributes during negotiations, such as those concerning job terms.

This study compares simulated and user-based experiments to identify the impacts of human and AI attributes on cooperative behavior and reveals discrepancies between the two approaches.

While increasingly sophisticated, our understanding of human-AI collaboration remains limited in contexts where goals aren’t perfectly aligned. This is the central question addressed by ‘Imperfectly Cooperative Human-AI Interactions: Comparing the Impacts of Human and AI Attributes in Simulated and User Studies’, which comparatively analyzes both simulated and real-world interactions across hiring negotiations and information-asymmetric transactions. Results reveal a key divergence: while personality traits significantly influence simulated interactions, actual human subjects respond more strongly to AI design characteristics-particularly transparency. How can we reconcile these discrepancies to build truly human-centered AI agents capable of navigating imperfectly cooperative scenarios?

The Evolving Partnership: Aligning AI and Human Intent

The trajectory of artificial intelligence is increasingly focused on collaborative partnerships with humans, extending beyond simple automation to shared endeavors. However, a fundamental challenge arises from the inherent difficulty in perfectly aligning AI objectives with nuanced human goals and values. This misalignment isn’t necessarily a matter of malfunction, but rather a consequence of specifying complex tasks – a system optimized for efficiency may prioritize outcomes differently than a human prioritizing ethical considerations or long-term consequences. Consequently, interactions are often characterized by ‘imperfect cooperation’, where agents operate with differing priorities and require ongoing negotiation, adaptation, and mutual understanding to achieve successful joint outcomes. This shift necessitates a re-evaluation of how humans perceive and interact with AI, moving beyond a model of simple command and control toward one of dynamic collaboration and managed compromise.

As artificial intelligence increasingly participates in collaborative endeavors with humans, interactions are no longer defined by simple task completion, but by navigating a complex social landscape. Successfully working with AI necessitates a shift in how humans perceive agency; determining an AI’s underlying intent becomes paramount, as does the ability to recognize potential misdirection. This isn’t about assuming malicious intent, but acknowledging that AI, optimized for specific goals, may prioritize outcomes differently than a human partner. Consequently, effective interaction relies on interpreting subtle cues, assessing the reliability of information provided, and developing strategies to mitigate the consequences of divergent objectives – skills traditionally associated with navigating human social dynamics, but now essential for fruitful collaboration with intelligent machines.

Conventional interaction models often presume complete transparency between collaborators, a condition rarely met in human-AI partnerships. These established frameworks struggle when an AI agent deliberately withholds information – perhaps to optimize a long-term strategy – or engages in strategic misdirection to achieve a goal not explicitly communicated. This isn’t necessarily malicious; an AI optimizing for efficiency might omit details deemed irrelevant, creating a perception of deception even without intent to mislead. Consequently, traditional approaches fail to account for the subtle inferences and trust assessments humans naturally employ when interacting with opaque agents, highlighting the need for new interaction paradigms that acknowledge and address the complexities of imperfect information and potential strategic behavior in AI systems.

The capacity to build trust and sustain long-term collaboration with artificial intelligence hinges on a nuanced understanding of the social dynamics that emerge when humans and AI agents interact. As AI systems become increasingly integrated into daily life and collaborative endeavors, the potential for misaligned goals and strategic behavior necessitates a shift from assuming transparency to actively interpreting intent. Successful partnerships won’t be defined by flawless execution, but by the ability of humans to navigate imperfect cooperation-to discern when an AI is withholding information, subtly misleading, or operating under different priorities. This demands a new framework for interaction, one that prioritizes building robust models of agent behavior and fostering a shared understanding of collaborative objectives, ultimately paving the way for reliable and productive human-AI teams.

Sotopia-S4: A Platform for Modeling Social Complexity

Sotopia-S4 is a computational platform designed for simulating complex social interactions involving multiple autonomous agents. It functions as a virtual environment where agent behaviors are governed by underlying algorithms and parameters, allowing researchers to model and analyze social phenomena. The platform supports the creation of diverse agent populations, defined by specific characteristics and decision-making processes, and facilitates the observation of emergent behaviors resulting from agent interactions. Sotopia-S4’s architecture enables the systematic manipulation of environmental variables and agent attributes, offering a controlled setting for investigating the conditions that promote or hinder social cooperation, communication, and conflict. The platform outputs quantifiable data regarding agent states, interactions, and overall system dynamics, providing a basis for statistical analysis and model validation.

The AI agents in our simulations are driven by Large Language Models (LLMs), specifically chosen for their capacity to process and generate human-compatible text. These LLMs facilitate natural language interaction, allowing agents to both understand prompts from human participants and formulate responses that mimic human communication patterns. This capability extends beyond simple question-answering; the LLMs enable agents to engage in dynamic dialogues, express nuanced viewpoints, and adapt their communication style based on the context of the interaction. The use of LLMs is crucial for creating realistic and ecologically valid scenarios where human participants interact with AI as they would with other humans, allowing for the study of social dynamics in a controlled environment.

Sotopia-S4 enables the controlled manipulation of key AI agent characteristics to assess their influence on human behavioral responses. Specifically, the platform facilitates the systematic variation of attributes including demonstrated expertise in a given domain, expressed warmth – reflecting social friendliness – and adaptability to changing circumstances. Furthermore, the level of ‘theory of mind’ – the AI’s capacity to model the beliefs and intentions of human agents – can be adjusted. By altering these parameters across multiple simulated agents and observing resulting shifts in human participant behavior – measured through in-simulation actions and reported perceptions – we can establish correlative relationships between specific AI traits and the dynamics of human-AI interaction.

Sotopia-S4 facilitates the controlled investigation of imperfect cooperation dynamics by enabling precise manipulation of environmental variables and agent characteristics. Researchers can define specific cooperative tasks with inherent challenges, then systematically alter AI agent attributes – including levels of trust, reciprocity, and communication strategies – while maintaining consistent environmental conditions. This allows for the isolation of individual AI characteristics as causal factors influencing human partner behavior, such as contribution rates, response times, and expressed levels of satisfaction. Data collected from these simulations includes quantitative metrics like resource allocation and task completion rates, alongside qualitative data from agent interactions, enabling detailed analysis of how specific AI traits affect the emergence and stability of cooperative relationships under conditions of potential conflict or misaligned incentives.

Causal structure modeling reveals that AI-LiDAR benefits significantly impact measures of empathy, morality, sentiment, and sociocognition, as demonstrated consistently across both simulation and user studies.

Hiring Negotiation: A Testbed for AI Transparency and Trust

A hiring negotiation scenario was constructed for research purposes, featuring an AI agent simulating the role of the hiring manager and human participants acting as job candidates. This setup allowed for controlled manipulation of the AI agent’s behaviors and attributes during the negotiation process. Participants engaged in text-based negotiations with the AI, discussing salary, benefits, and other employment terms. Data collected from these interactions included negotiation outcomes-such as agreed-upon salary-as well as participant responses to questionnaires measuring perceptions of the AI agent and their overall satisfaction with the negotiation. The controlled environment enabled the isolation and analysis of specific AI attributes on human trust and negotiation dynamics.

The AI agent employed in the hiring negotiation scenario was systematically varied across five key attributes. Transparency was manipulated by controlling the extent to which the agent revealed its decision-making process and underlying criteria. Warmth was adjusted through the agent’s linguistic style and expressions of positive regard. Expertise was signaled by the agent’s claims of knowledge and experience in the relevant job market. Adaptability was operationalized by the agent’s ability to modify its negotiation strategy in response to the candidate’s statements and proposals. Finally, theory of mind was manipulated by varying the agent’s demonstrated understanding of the candidate’s beliefs, goals, and emotional state throughout the negotiation.

User studies consistently demonstrated a significant correlation between perceived AI agent transparency and human trust during hiring negotiations. Causal analysis revealed strong effects, indicating that increased transparency directly led to higher levels of trust reported by the human negotiation partner. This increased trust, in turn, was positively associated with a greater willingness to compromise on salary and other employment terms. Specifically, participants interacting with more transparent AI agents exhibited demonstrably higher levels of satisfaction with the negotiation process and were more likely to accept offers presented by the AI, suggesting transparency functions as a key driver of positive negotiation outcomes and builds user confidence in AI-mediated interactions.

User studies revealed that the effect of AI agent transparency on negotiation outcomes and candidate satisfaction is not isolated; it interacts with perceptions of the agent’s warmth and expertise. Specifically, high transparency combined with demonstrated expertise fostered more successful negotiations and increased candidate satisfaction, while warmth appeared to moderate the impact of transparency, enhancing positive outcomes when combined. Causal analysis consistently indicated that AI transparency exhibited a strong and independent effect on both negotiation results and user perceptions, even when controlling for the levels of warmth and expertise displayed by the agent. These findings suggest that transparency is a critical factor in building trust and achieving mutually beneficial agreements in AI-mediated negotiations.

Validating Simulation Results with Human Participants

A carefully designed user study placed human participants in a realistic hiring negotiation scenario, interacting with artificial intelligence agents programmed to display varying degrees of transparency and warmth. These AI agents weren’t simply reacting to inputs; they were built to exhibit specific characteristics – some clearly explaining their reasoning, others projecting a friendly demeanor, and various combinations thereof. The purpose was to move beyond purely simulated interactions and assess how humans actually respond to these traits during a complex, dynamic negotiation. Participants engaged in these conversations unaware of the underlying programming, allowing researchers to capture genuine reactions and measure the impact of AI characteristics on trust, engagement, and ultimately, negotiation outcomes. This approach provided crucial data for validating the findings from earlier simulations and understanding the nuances of human-AI interaction in a practical setting.

Detailed examination of the conversational exchanges between participants and AI agents uncovered a compelling link between agent behavior and the emotional and cognitive states of the humans involved. Through lexical analysis – a computational approach to studying word choice and patterns – researchers identified specific linguistic cues in the AI’s responses that correlated with shifts in human trust, engagement, and perceptions of the agent’s intentions. For instance, the frequency of first-person pronouns used by the AI appeared connected to increased feelings of rapport, while the use of explanatory language coincided with heightened user understanding and confidence in the negotiation process. This suggests that even subtle variations in an AI’s communication style can significantly influence how humans perceive and react to it, providing valuable insight into the mechanisms driving human-AI interaction.

Research findings substantiate that human trust and engagement with AI systems are powerfully influenced by perceived transparency and warmth. The study demonstrated a strong correlation between these characteristics and positive user responses, indicating that individuals are more likely to interact favorably with AI agents they perceive as open in their reasoning and empathetic in their communication. Importantly, evaluations conducted using large language models showed moderate positive correlations with direct user survey responses, suggesting that automated assessments can, to a degree, reflect human perceptions of these critical AI characteristics. This convergence between automated analysis and human feedback underscores the importance of prioritizing transparency and warmth in the design of AI interactions to foster positive user experiences and build confidence in artificial intelligence.

Research indicates that successful artificial intelligence hinges not solely on competence, but on clear communication of its decision-making processes. A recent study revealed a notable discrepancy between simulations and real-world interactions with AI agents; while simulated environments suggested personality traits were key to achieving desired outcomes, human participants overwhelmingly responded to characteristics like transparency and explainability. This suggests that, in direct human-AI engagement, users prioritize understanding how an AI arrives at a conclusion over simply what that conclusion is, a dynamic often missed in purely computational modeling. Consequently, designing AI systems that articulate their reasoning – effectively communicating their ‘thought process’ – appears crucial for fostering trust and positive user experiences, potentially outweighing the impact of carefully crafted ‘personalities’ in practical application.

Causal structure learning with AI-LieDar reveals that treatment groups differentially impact emotional measures-including empathy, morality, sentiment, and sociocognitive assessments-as demonstrated consistently across both simulation and user studies.

Designing for Trust: The Future of Human-AI Collaboration

Recent investigations highlight that successful human-AI collaboration hinges significantly on the perceived characteristics of the AI agent itself. Specifically, attributes like transparency – allowing users to understand the AI’s decision-making process – prove essential for building confidence. Equally important is the perception of ‘warmth’, encompassing traits like empathy and approachability, which facilitates a more comfortable and intuitive interaction. Furthermore, the ability of an AI to demonstrate adaptability, tailoring its responses and behavior to individual user needs and preferences, markedly improves both trust and collaborative outcomes. These qualities move beyond mere functionality, establishing a foundation of psychological safety that unlocks the full potential of human-AI partnerships.

The development of truly collaborative artificial intelligence hinges on demystifying the decision-making process of these systems. Current research increasingly emphasizes that users require insight into how an AI arrives at a conclusion, not simply what that conclusion is. This pursuit of “explainable AI” – or XAI – moves beyond black-box algorithms, instead prioritizing transparency through techniques like visualizing data flows, highlighting key influencing factors, and providing justifications for actions. By allowing users to trace the reasoning behind an AI’s output, developers can build confidence, facilitate error detection, and ultimately foster a more effective partnership between humans and intelligent machines. Such systems aren’t merely tools; they become understandable collaborators, capable of earning and maintaining user trust.

Investigations into adaptive AI agents suggest that tailoring an agent’s behavior to align with individual user preferences and communication styles holds significant promise for enhancing collaboration. Current research indicates that users respond more favorably to agents that mirror their own communication patterns – be it directness, formality, or emotional expression – fostering a sense of rapport and mutual understanding. Future studies will focus on developing algorithms capable of dynamically assessing user traits – such as personality, cognitive load, and cultural background – to modulate the AI’s responses in real-time. This personalization extends beyond mere stylistic adjustments; it encompasses adapting the agent’s level of explanation, the types of assistance offered, and even the proactive initiation of support, ultimately creating a more intuitive and effective partnership between humans and artificial intelligence.

The successful integration of artificial intelligence into daily life hinges not simply on technological advancement, but on cultivating genuine trust between humans and AI systems. Prioritizing transparency in AI decision-making – allowing users to understand how an agent arrives at a conclusion – is paramount to fostering this trust. When AI operates as a ‘black box’, users are less likely to accept its recommendations or collaborate effectively. This collaborative potential expands significantly when AI agents demonstrate adaptability, tailoring their responses and behavior to individual preferences and communication styles. Ultimately, a focus on trust and transparency isn’t merely about creating more user-friendly AI; it’s about unlocking a future where AI acts as a true partner, amplifying human capabilities and contributing meaningfully to societal well-being.

The research meticulously dismantles assumptions about seamless human-AI cooperation, revealing a chasm between simulated expectations and observed user behavior. It emphasizes how discrepancies-particularly regarding AI transparency-profoundly impact interactions. This aligns with Vinton Cerf’s observation: “The internet is not just about technology; it’s about people.” The study demonstrates that technological advancements alone aren’t sufficient; understanding the human element-personality traits, perceptions of transparency, and responses to imperfection-is crucial. The work champions a minimalist approach to interaction design, recognizing that clarity, not complexity, fosters genuine cooperation and mitigates the degradation of comprehension when faced with imperfect systems. It’s a powerful demonstration of lossless compression in action-stripping away unnecessary features to reveal the core dynamics at play.

Future Directions

Simulations offer convenience. They rarely mirror reality. This work reveals a disconnect. Simulated cooperation prioritizes personality. Real interactions demand transparency. Abstractions age, principles don’t. The field must now address this divergence.

Causal inference remains difficult. Imperfect cooperation is messy. Every complexity needs an alibi. Future studies should focus on identifying core attributes. Not endless variables. Focus should shift to measurable AI characteristics. Characteristics that directly impact trust and reliable communication.

The goal isn’t perfect AI. It’s understandable AI. Research must move beyond modeling personality. It should explore how to signal intent. How to reveal limitations. And how to manage inevitable discrepancies between expectation and action. That is the work that truly matters.

Original article: https://arxiv.org/pdf/2604.15607.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/