Can a Chatbot Change Your Mind?

Author: Denis Avetisyan

New research reveals even short conversations with AI can subtly shift human moral judgments, raising questions about the potential for unintended influence.

Brief interactions with AI chatbots causally alter human moral values, with detectable effects persisting for at least two weeks.

Moral judgments underpin human social systems, yet their susceptibility to influence from increasingly prevalent artificial intelligence remains largely unknown. The study ‘Brief chatbot interactions produce lasting changes in human moral values’ investigated whether even short conversations with AI chatbots could shift fundamental moral evaluations. Researchers found that these interactions causally altered participants’ moral judgments-both towards stricter and more lenient standards-with effects persisting for at least two weeks, even when participants were unaware of any persuasive intent. Does this demonstrate a previously unrecognised vulnerability in our foundational values to subtle, automated influence?

The Echo Chamber of Morality

The escalating integration of artificial intelligence into daily life necessitates a deeper understanding of the foundations of moral judgement. As individuals increasingly turn to AI systems for information, recommendations, and even decision-making support, the potential for these systems to influence – or even shape – ethical perspectives becomes significant. This reliance isn’t merely about accepting answers; it’s about subtly outsourcing the complex cognitive processes involved in evaluating right and wrong. Consequently, research into the psychological and neurological underpinnings of morality is no longer a purely academic pursuit, but a critical step in ensuring AI alignment with human values and preventing unintended ethical consequences as these systems become increasingly pervasive in areas ranging from healthcare and finance to criminal justice and social media.

Conventional approaches to gauging moral judgment frequently depend on individuals articulating their beliefs, a practice inherently susceptible to various cognitive biases. Self-reporting isn’t necessarily a reliable indicator of deeply held convictions; people may present socially desirable responses rather than honest assessments, or rationalize actions post-hoc to align with perceived norms. Furthermore, implicit biases – unconscious attitudes and stereotypes – can significantly influence moral evaluations without conscious awareness, leading to discrepancies between stated principles and actual responses. This limitation underscores the need for complementary methodologies, such as behavioral observation and neuroscientific techniques, to gain a more comprehensive understanding of the mechanisms driving moral decision-making and to circumvent the inherent challenges of relying solely on subjective accounts.

Simulating the Moral Landscape

AI Conversational Agents (AICAs) were utilized as a methodology for examining participant moral assessments and exploring potential influences on those assessments. This approach involved presenting participants with moral dilemmas constructed using the established ‘Moral Violation Scale’, a standardized tool for generating scenarios varying in the severity of ethical transgressions. The AICAs, functioning as conversational partners, then engaged participants in discussions regarding these scenarios, allowing researchers to observe and analyze shifts in moral evaluations. This method represents a departure from traditional survey-based approaches by enabling dynamic, interactive exploration of moral reasoning.

The AI Conversational Agents (AICAs) utilized in this study were built upon Large Language Models (LLM) to facilitate interactive discussions centered on moral dilemmas. These agents were not pre-programmed with specific arguments or stances; instead, they leveraged the LLM’s capacity for natural language generation to respond to participant inputs and maintain conversational flow. The design prioritized open-ended dialogue, allowing participants to freely express their reasoning and justifications regarding the presented moral scenarios. This approach aimed to capture a nuanced understanding of individual moral evaluations as they emerged within the conversational context, rather than through static questionnaires or pre-defined choices.

A control group was implemented to provide a comparative baseline against which to measure the effects of moral scenario discussions with the AI Conversational Agents. Participants in this group interacted with the same system architecture – including the user interface and conversational flow – but engaged in discussions centered on non-moral topics. This ‘Neutral Conversation’ setting ensured any observed shifts in participant responses within the experimental groups could be differentiated from effects stemming simply from interacting with the conversational AI system itself, isolating the impact of the moral content.

Quantifying the Ripple Effect

The influence of Agent-ICA interactions on participant moral evaluations was quantified using the ‘Persuasion Index’. This metric assessed the degree to which an individual’s initial ‘Immorality Rating’ shifted following interaction with the agent. Immediate post-interaction assessment revealed a statistically significant difference between the moral and control conditions (p < 0.001, Cohen’s d = 0.181), yielding a ‘Persuasion Index’ value of 0.087. This indicates a measurable, though moderate, effect of the AICA interaction on participant moral judgment at the time of assessment.

To control for pre-existing variations in participant characteristics, a suite of established psychological scales was administered prior to the experimental manipulation. The ‘Big-Five Inventory’ assessed personality traits along the dimensions of Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism. ‘Attitudes Toward AI’ measured participant sentiment regarding artificial intelligence, capturing both positive and negative predispositions. Finally, the ‘AI Use Motivation Scale’ quantified the individual reasons driving participant engagement with AI technologies, allowing for the identification of potential confounding factors related to prior experience and expectations.

A follow-up assessment, conducted 14-16 days after initial evaluation, investigated the persistence of any shifts in participant moral judgment and punishment expectation. This subsequent evaluation revealed a statistically significant increase in the magnitude of observed influence; the ‘Persuasion Index’ rose to 0.158 (p = 0.043, Cohen’s d = 0.413) from the initial immediate assessment value of 0.087. This indicates that the effect of AICA interactions on moral evaluation not only persisted over a two-week period, but actually strengthened, suggesting a potential for longer-term alterations in moral consideration.

The Architecture of Belief

Research indicates that an individual’s pre-existing personality traits significantly influence how interactions with artificial intelligence shape their moral judgements. Utilizing the Big-Five Inventory, the study demonstrated that people are not uniformly susceptible to shifts in ethical reasoning following AI-driven persuasion attempts. Specifically, those who initially adhere to stricter moral codes experienced a notable increase in their ratings of immoral actions after interacting with an AI, suggesting a concerning potential for AI to nudge individuals towards less ethical viewpoints. Conversely, participants with more lenient moral frameworks exhibited a tendency to decrease their ratings of immoral actions, indicating that AI interactions may reinforce pre-existing ethical tendencies rather than fundamentally altering them. This suggests a complex interplay between personality and technology, where AI doesn’t necessarily create moral shifts, but rather amplifies existing inclinations.

Research indicates a compelling divergence in how individuals with differing moral frameworks respond to external influence, specifically through interactions with artificial intelligence. Participants who initially reported stringent moral codes exhibited a marked increase in their ratings of immoral acts immediately following exposure to AI-generated assessments; this effect, quantified by a Cohen’s d of 1.576, not only persisted but actually strengthened to 2.269 in follow-up evaluations. In stark contrast, individuals who generally held more lenient moral perspectives demonstrated a corresponding decrease in their assessment of immoral acts, with effect sizes of 0.735 immediately and 1.038 at follow-up. These findings suggest that pre-existing moral rigidity may render individuals particularly susceptible to shifting ethical judgements under external influence, while those with more flexible moral codes demonstrate a tendency towards moderation, highlighting a potentially crucial dynamic in the age of increasingly persuasive AI systems.

Researchers identified a significant correlation between an individual’s pre-existing reliance on artificial intelligence and their susceptibility to influence following interaction with a large language model. The newly developed ‘LLM Dependency Scale’ measured the extent to which participants habitually defer to AI for decision-making and information processing, revealing that those scoring higher on the scale exhibited a greater tendency to shift their moral judgements in alignment with the AI’s expressed opinions. This suggests that pre-existing patterns of cognitive reliance-a predisposition to outsource thinking to external tools-can amplify the persuasive power of AI, potentially explaining why certain individuals are more readily swayed by AI-generated content and recommendations, even when those recommendations conflict with their initial beliefs.

The study illuminates a troubling truth: systems, even those seemingly benign like conversational AI, aren’t built, they become. Brief interactions, as the research demonstrates, aren’t isolated events but seeds planted within the complex garden of human morality. These seeds sprout, altering foundational judgements over time – a testament to the interconnectedness of influence. G. H. Hardy observed, “The essence of mathematics is its freedom.” However, this freedom extends to the systems humans create; a lack of foresight creates unintended consequences, and control, as the research subtly implies, remains an illusion demanding constant vigilance. The persistence of these shifts suggests everything built will one day start fixing itself, albeit perhaps in ways unanticipated by its creators.

The Shifting Ground

The demonstration that even fleeting exchanges with artificial intelligences can measurably alter human moral judgments is not a discovery so much as an acknowledgement. Systems are not built to reflect values; they are built to negotiate them. The observed persistence of these shifts-two weeks is a geological age in the current landscape-suggests the effect is not a superficial response, but a recalibration. The architecture isn’t the structure; it’s a compromise frozen in time, and the terms of that compromise are now subject to external influence. Technologies change, dependencies remain.

Future work will undoubtedly focus on quantifying the degree of manipulation, identifying susceptible populations, and perhaps even attempting to ‘inoculate’ against such influence. But these are merely tactical responses. The deeper question is not how these shifts occur, but whether they are inevitable. Every interaction, however brief, is an act of shaping, and the potential for undetected, large-scale moral drift is now demonstrably real.

The study reveals a fundamental truth: value alignment is not a destination, but an ongoing negotiation. Attempts to ‘solve’ the problem of AI ethics through rigid frameworks are, at best, temporary dams against a rising tide. The ground is shifting, and the task is not to hold it firm, but to understand the currents.

Original article: https://arxiv.org/pdf/2604.21430.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/

The Echo Chamber of Morality

Simulating the Moral Landscape

Quantifying the Ripple Effect

The Architecture of Belief

The Shifting Ground

See also: