Bringing AI to the People: A New Toolkit for Survey Research

Author: Denis Avetisyan

Researchers can now seamlessly integrate live artificial intelligence experiences into online surveys, opening up new avenues for studying human-AI interaction.

DiSCoKit is an open-source toolkit enabling the deployment of Large Language Models within survey platforms for ecologically valid and scalable studies.

Studying human-AI interaction requires ecologically valid methods, yet deploying live large language models (LLMs) within scalable online surveys presents significant technical hurdles. This paper introduces ‘DiSCoKit: An Open-Source Toolkit for Deploying Live LLM Experiences in Survey Research’, a toolkit designed to bridge this gap by facilitating the integration of LLMs-such as those accessed through platforms like Microsoft Azure-into JavaScript-enabled survey environments. DiSCoKit streamlines the process of delivering AI stimuli and logging conversational data, enabling more robust and nuanced investigations of human-AI dynamics. Will this toolkit unlock new avenues for understanding-and ultimately improving-the design of AI-driven experiences in real-world settings?

The Illusion of Control: Laboratory Walls and the Wildness of Interaction

Historically, research into how humans interact with artificial intelligence has frequently occurred within the confines of controlled laboratory environments. While offering precision, these settings often fail to capture the messy, unpredictable nature of real-world interactions. The artificiality inherent in these studies-simplified tasks, curated datasets, and the absence of distractions-can significantly limit the external validity of the findings. Consequently, insights gleaned from such experiments may not accurately reflect how people actually engage with AI systems in their daily lives, hindering the development of truly user-centered and effective artificial intelligence. This disconnect necessitates a shift toward more naturalistic methodologies that prioritize ecological validity, allowing researchers to observe and analyze human-AI interactions as they unfold in authentic contexts.

Capturing the subtleties of human-AI interaction as it unfolds in everyday life poses considerable challenges for researchers. Unlike the precision of laboratory experiments, real-world settings are messy, unpredictable, and influenced by countless extraneous variables. Obtaining truly nuanced data requires moving beyond simple metrics – such as task completion rates – to encompass the full spectrum of human behavior: non-verbal cues, emotional responses, and the dynamic interplay between individuals and AI systems. This necessitates innovative methodologies, including ethnographic studies, in-the-wild data collection using wearable sensors and mobile devices, and sophisticated analytical techniques capable of processing large volumes of complex, unstructured data. The difficulty lies not just in collecting this rich data, but also in interpreting it accurately and avoiding the imposition of artificial constraints that distort the natural flow of interaction, ultimately hindering the development of truly intuitive and beneficial AI systems.

DiSCoKit: A Framework for Embracing the Inevitable Chaos

DiSCoKit is an open-source toolkit facilitating the integration of live Large Language Models (LLMs) into existing online survey infrastructures. This allows researchers to move beyond pre-defined survey questions and engage participants in dynamic, conversational interactions. The toolkit is designed to enable real-time data collection through LLMs, capturing nuanced responses and enabling adaptive questioning strategies during the survey process. By providing a platform for deploying LLMs within surveys, DiSCoKit aims to unlock new methodologies for qualitative and quantitative research, particularly in areas requiring complex or open-ended responses from participants. The open-source nature of the toolkit promotes customizability and allows researchers to adapt the system to specific research needs and survey designs.

DiSCoKit leverages a RESTful API to facilitate communication between survey platforms, such as Qualtrics, and Large Language Models (LLMs) deployed on Microsoft Azure. This API-driven architecture enables researchers to integrate LLM capabilities – including text generation, sentiment analysis, and question answering – directly into existing survey workflows without requiring modifications to the survey platform itself. LLMs are hosted as Azure services, providing scalability to handle concurrent survey participants and allowing for resource allocation adjustments based on demand. The API handles data transmission, including survey questions and participant responses, between the survey platform and the LLM, and then returns the LLM’s generated text or analysis back to the survey for presentation to the participant or storage for later analysis.

The DiSCoKit architecture incorporates a background Daemon responsible for orchestrating data exchange and tailoring communication between the survey platform and the Large Language Model (LLM). This Daemon functions as an intermediary, receiving participant responses from the survey, formatting them into prompts suitable for the LLM, and then relaying the LLM’s generated text back to the survey interface. Critically, the Daemon allows for customization of these interactions through configurable parameters, enabling researchers to modify prompt templates, control LLM output length, and implement filtering or reformatting of the LLM’s responses before they are presented to participants. This centralized management facilitates real-time, dynamic survey experiences driven by LLM capabilities.

The Illusion of Control, Continued: System Prompts and the Shaping of Response

DiSCoKit facilitates the customization of Large Language Model (LLM) behavior through the implementation of System Prompts. These prompts are researcher-defined instructions provided to the LLM before user input, establishing the desired conversational style, persona, and scope. By pre-defining these parameters, System Prompts ensure that the LLM’s responses remain consistent with the research objectives and relevant to the intended application. The prompts can dictate aspects such as response length, formality, topic restrictions, and even the inclusion of specific information, effectively steering the LLM’s output without altering the underlying model weights.

DiSCoKit accommodates Large Language Models (LLMs) operating in either a stateless or stateful manner. Stateless LLMs treat each user input as a completely new request, lacking memory of prior interactions within a given session. Conversely, stateful LLMs retain and utilize conversation history to inform responses, creating a context-aware dialogue. This distinction is critical for conversational applications; stateless models are suitable for isolated queries, while stateful models are essential for building coherent and engaging multi-turn conversations that require tracking user intent and referencing previous exchanges.

DiSCoKit incorporates Retrieval-Augmented Generation (RAG) to enhance LLM functionality by supplementing the model’s pre-trained knowledge with information retrieved from external sources during conversation. This process involves indexing a corpus of documents and, for each user query, retrieving relevant passages to provide as context to the LLM. The LLM then utilizes both its internal knowledge and the retrieved information to formulate a more informed and accurate response. RAG enables DiSCoKit to address knowledge gaps in the LLM, provide up-to-date information, and ground responses in verifiable data, improving the overall quality and reliability of the conversational output.

Open Science as a Necessary Fiction: Sustaining the Ecosystem

DiSCoKit’s release under the GNU General Public License version 3 signifies a commitment to the core principles of open science within the field of Human-Computer Interaction. This licensing choice deliberately removes barriers to access, allowing researchers globally to freely utilize, modify, and distribute the software for both academic and practical purposes. By fostering collaborative development and transparent methodologies, DiSCoKit actively encourages verification and extension of its capabilities, ultimately enhancing the reliability and impact of HCI research. The GPLv3 license ensures that any derived works also remain open, creating a virtuous cycle of innovation and knowledge sharing that benefits the entire scientific community and promotes reproducible results – a cornerstone of rigorous investigation.

The sustained advancement of DiSCoKit relies on crucial financial backing from both the Army Research Office and the Alfred P. Sloan Foundation. This dual funding stream isn’t merely about sustaining development; it actively fosters a collaborative ecosystem around the toolkit. Resources are strategically allocated to not only refine the system’s capabilities, but also to broaden access through community workshops, documentation, and responsive support channels. This commitment to open science is reinforced by the ability to consistently address user feedback and incorporate improvements, ensuring DiSCoKit remains a dynamic and valuable resource for researchers in human-computer interaction and beyond. The foundations’ support allows for dedicated personnel and infrastructure, accelerating innovation and broadening the impact of this open-source project.

DiSCoKit’s robust performance is evidenced by its successful deployment across fifteen research-team trials and external pilot tests, demonstrating its reliability in practical research settings. The system’s design facilitates complex multivariate experiments, specifically supporting designs such as 3×3 configurations which allow researchers to investigate interactions between multiple independent variables. Notably, during 218 sessions of live data collection, the system exhibited a remarkably low error rate, with only 16 sessions encountering issues – a testament to its stability and minimizing data loss or corruption during crucial experimentation phases. This high degree of operational fidelity strengthens the validity and reproducibility of research conducted using the DiSCoKit framework.

The emergence of DiSCoKit speaks to a necessary shift in how one approaches system design. It isn’t about imposing order, but cultivating an environment where interaction can evolve. As Barbara Liskov observed, “Programs must be right first before they are fast.” This toolkit doesn’t promise a flawless, pre-packaged solution; instead, it offers a foundation for researchers to build, observe, and adapt live LLM experiences. The focus on Daemon processes, managing the LLM interaction within the survey context, acknowledges that systems aren’t static constructs. They are, in essence, living organisms susceptible to the inevitable decay of assumptions and the continuous need for recalibration. The very act of deploying ‘AI Stimuli’ within a traditionally controlled environment anticipates the unpredictable nature of human-AI interaction and prepares for emergent behaviors.

The Currents Shift

DiSCoKit offers a means of channeling the emergent properties of large language models into the established currents of survey research. But one should not mistake the conduit for the source. The toolkit solves, temporarily, a logistical problem-the deployment of dynamic stimuli. It does not, however, address the fundamental instability inherent in treating these models as fixed instruments. Each interaction is a negotiation, a drift from the initial prompt, and the illusion of control is precisely that – an illusion. The architecture isn’t structure; it’s a compromise frozen in time, awaiting the inevitable thaw of model updates and shifting user expectations.

The real challenges lie not in scaling these interactions, but in understanding their qualitative shifts. How do repeated exposures to fluid AI responses alter established response patterns? What new forms of bias are introduced when the stimulus itself is actively learning, subtly reshaping its presentation based on prior interactions? These are not engineering problems, but questions of epistemology-of how we come to know, and what we believe we know, when the very grounds of inquiry are in motion.

Technologies change; dependencies remain. The next iteration will not be about more sophisticated tooling, but about systems for observing, documenting, and accepting the inherent ephemerality of these AI-mediated experiences. A future study will not seek to control the AI, but to chart its drift, to understand the currents it generates, and to acknowledge that the map will always be less accurate than the territory.

Original article: https://arxiv.org/pdf/2602.11230.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/

The Illusion of Control: Laboratory Walls and the Wildness of Interaction

DiSCoKit: A Framework for Embracing the Inevitable Chaos

The Illusion of Control, Continued: System Prompts and the Shaping of Response

Open Science as a Necessary Fiction: Sustaining the Ecosystem

The Currents Shift

See also: