Animating Animals from Words: A New Era of Motion Control

Author: Denis Avetisyan

Researchers have developed a framework capable of generating realistic and diverse animal movements directly from text descriptions, regardless of skeletal structure.

The system achieves text-driven motion transfer, generating biomechanically plausible movements-even for markedly different skeletal structures like those of a meerkat and a cassowary-from a single textual instruction, such as “scanning surroundings.”

A topology-agnostic autoregressive model trained on a large-scale dataset enables text-to-motion generation for animals with heterogeneous skeletons.

Despite advances in computer animation and robotics, generating realistic and controllable animal motion remains challenging for skeletons with varying structures. This limitation motivates the work ‘Topology-Agnostic Animal Motion Generation from Text Prompt’, which introduces a novel framework capable of synthesizing diverse animal locomotion from text prompts, regardless of skeletal topology. Central to this approach is a large-scale dataset, OmniZoo, and a topology-aware autoregressive model that effectively fuses textual semantics with skeletal geometry. Could this generalized framework unlock new possibilities for cross-species animation, robotic control, and virtual creature design?

The Art of Motion: Capturing the Essence of Animal Locomotion

The creation of convincing animal motion for computer graphics and animation has persistently challenged researchers, stemming from the inherent intricacies of biological movement itself. Unlike the predictable mechanics of inanimate objects, animal locomotion is a multifaceted interplay of skeletal structure, muscular forces, neural control, and environmental interaction. Each species exhibits a unique repertoire of gaits and postures, further complicated by individual variations and contextual factors like speed, terrain, and intent. This natural variability-a subtle shift in weight, a momentary hesitation, the asymmetrical ripple of muscles-is crucial for realism, yet exceedingly difficult to replicate computationally. Consequently, achieving truly believable animal animation demands not simply mimicking a single instance of movement, but capturing and reproducing the full spectrum of possible motions, a task that has driven innovation in biomechanical modeling, motion capture technology, and procedural animation techniques.

Current approaches to simulating animal movement frequently falter when applied to species outside of their initial training data. This limitation stems from the inherent diversity of locomotion-a giraffe’s gait differs dramatically from that of a snake or a spider-and the difficulty in capturing these nuances with a single, generalized model. Consequently, animators and researchers often face the daunting task of painstakingly adjusting parameters and retraining systems for each new animal they wish to simulate. This per-species customization is not merely a refinement; it frequently demands a complete overhaul of the underlying motion framework, representing a significant investment of time and computational resources. The lack of generalization hinders the creation of truly versatile animation tools and limits the scalability of motion synthesis for large-scale simulations or virtual ecosystems.

The creation of universally adaptable animal motion systems hinges on both the breadth of available data and the sophistication of the underlying framework. Current approaches frequently falter when applied to species outside of their training set, highlighting the need for a dataset encompassing a vast array of animal morphologies and locomotion styles. However, data alone is insufficient; the framework must be capable of discerning and replicating subtle variations in gait, posture, and dynamics – the nuances that distinguish a cheetah’s sprint from a sloth’s crawl. A robust system requires not simply cataloging movements, but understanding the biomechanical principles and behavioral contexts that govern them, allowing for the generation of plausible and diverse motions even for animals not explicitly represented in the training data. This demands a flexible architecture, potentially leveraging techniques like physics-based simulation or machine learning models capable of generalization and adaptation, ultimately unlocking the potential for truly realistic and versatile animal animation.

Our text-to-motion system realistically animates animals reconstructed from real-world images, as demonstrated by a falcon performing aerial maneuvers and a horse trotting with coordinated leg movements.

An Autoregressive Symphony: Orchestrating Believable Motion

The motion generation system employs an autoregressive model, processing sequential data to predict subsequent frames. Given a history of $n$ previous motion frames and accompanying conditioning signals – such as text prompts or audio – the model estimates the parameters of the next frame in the sequence. This prediction is then fed back into the model, along with the updated history, to generate further frames. This iterative process allows the system to synthesize extended motion sequences, maintaining temporal coherence by explicitly modeling the dependencies between successive frames. The conditioning signals provide external control, influencing the generated motion and enabling the creation of diverse and contextually relevant movements.

The system employs a Generalized Motion Residual Vector Quantized Variational Autoencoder (VQ-VAE) to discretize continuous motion data into a sequence of discrete ‘Motion Tokens’. This process involves encoding motion frames into a latent space, followed by vector quantization to map latent vectors to a finite set of learned embeddings, effectively creating a codebook. Representing motion as discrete tokens enables several advantages; it reduces the dimensionality of the input space, facilitates efficient processing, and allows the model to learn long-range dependencies more effectively compared to directly modeling continuous motion data. The resulting sequence of Motion Tokens then serves as the primary input for the autoregressive model, streamlining the learning process and improving the quality of generated motion sequences.

The motion generation process is conditioned using dense text embeddings created by a SigLip2 Encoder. This encoder processes input text prompts and transforms them into a high-dimensional vector representation, capturing semantic information about the desired movement. The resulting text embedding serves as input to the autoregressive model, influencing the generated motion sequence. Specifically, the SigLip2 architecture facilitates the translation of linguistic input into a format suitable for controlling the generated motion, allowing users to guide the system with natural language prompts and achieve semantic control over the output.

This pipeline generates generalized conditional motion by extending a residual VQ-VAE to handle diverse skeletal topologies and fusing topology-aware skeleton embeddings with text prompts via a two-stage Transformer.

Decoding the Skeleton: A Topology-Aware Representation of Pose

The Topology-aware Skeleton Embedding Module addresses the problem of differing skeletal structures across species by converting variable-length skeletal data into fixed-size, compact embeddings. This module encodes both the topology – the connectivity of the skeleton’s joints – and the geometry – the 3D positions of those joints – into a latent vector representation. Specifically, the module utilizes graph convolutional networks to process the skeletal topology, extracting relational features between joints. Simultaneously, point cloud encoding techniques capture the geometric information. These features are then fused and projected into a lower-dimensional embedding space, resulting in a fixed-size vector that represents the skeletal structure regardless of species or individual variation. This allows for consistent input to downstream motion prediction models.

Skeletal motion prediction utilizes a two-transformer architecture following the encoding of skeletal data. Specifically, topology-aware skeletal embeddings, representing the skeletal structure, are combined with discrete ‘Motion Tokens’ – a quantized representation of movement – and inputted into a Residual Transformer. This transformer processes the combined data to establish initial predictions of future motion states. Subsequently, these predictions are fed into a Masked Transformer, which is trained to reconstruct potentially incomplete or corrupted motion sequences, thereby improving the robustness and realism of the generated motion predictions. This sequential processing allows the model to leverage both structural information and discrete motion representations for accurate and coherent motion forecasting.

The masked transformer component operates by randomly masking portions of the input motion sequence during training. This forces the network to learn to predict the missing data – effectively ‘inpainting’ the sequence – based on the contextual information present in the unmasked frames. This process enhances robustness by enabling the model to generate plausible motion even when faced with incomplete or noisy input data. Furthermore, the inpainting capability contributes to the realism of generated sequences by promoting temporal coherence and reducing artifacts that might arise from abrupt transitions or discontinuities in the motion data. The masked transformer thus learns a distribution over plausible motion states, allowing it to generate diverse and natural-looking animations.

The topology-aware skeleton embedding module captures skeletal structure to create a meaningful representation of pose.

The Symphony Extends: Cross-Species Generalization and Impact

The innovative ‘Cross-Species Motion Transfer’ technique represents a significant advancement in motion synthesis, allowing for the creation of realistic and varied movements across a diverse range of species from a single textual description. This approach bypasses the need for species-specific training data, instead leveraging a generalized understanding of motion principles to translate commands – such as “a joyful leap” or “a cautious stalk” – into appropriate animations for creatures as different as a cat, a bear, or even a fantastical dragon. The system effectively decouples the description of motion from the embodiment of the actor, showcasing a remarkable capacity to generalize learned behaviors and apply them to previously unseen species, opening doors for more flexible and efficient character animation in fields like robotics, virtual reality, and entertainment.

The generation of realistic and consistent motion sequences benefits significantly from the incorporation of a ‘Motion Summary’ into the model’s architecture. This summary acts as a condensed representation of the entire intended movement, providing the system with crucial global contextual information beyond simply interpreting frame-by-frame inputs. By distilling the overarching narrative of the motion – such as the general direction, speed, and style – into a compact vector, the model maintains coherence across longer sequences and avoids the common pitfalls of drifting or inconsistent movements. This is particularly impactful when transferring motion across species, as it ensures the generated actions remain plausible and physically grounded, even when applied to anatomies drastically different from those in the training data. The result is a substantial improvement in the fidelity and naturalness of the synthesized motion, allowing for more believable and compelling animations.

The training process benefits significantly from the implementation of Classifier-Free Guidance, a technique that enhances both the quality and the degree of user control over the generated motion sequences. Traditionally, conditional generation models relied on classifier guidance, requiring a separate classifier to steer the generation process. However, this approach can be computationally expensive and less flexible. Classifier-Free Guidance streamlines this by training a single model to function both with and without the conditional input-in this case, the text prompt. During inference, the model’s output is then intelligently scaled based on the difference between conditional and unconditional predictions, effectively amplifying the influence of the prompt and leading to more accurate and nuanced motion generation. This innovative approach not only improves the fidelity of the animations but also empowers users with greater precision in shaping the desired movement characteristics across different species.

Our model surpasses existing methods in generating realistic and text-aligned motions across diverse skeletal structures.

From Observation to Animation: Bridging the 2D-3D Divide

The conversion of two-dimensional images into realistic three-dimensional motion relies heavily on accurate geometric reconstruction, and this is achieved through the utilization of Hunyuan3D 2.0. This advanced system meticulously analyzes input imagery to generate detailed 3D meshes – essentially digital skeletons outlining the shape and form of the subject. By effectively translating visual data into a tangible, three-dimensional representation, Hunyuan3D 2.0 provides the foundational geometry necessary for subsequent animation and motion simulation. The system’s ability to discern depth and spatial relationships from flat images is crucial for creating convincing and lifelike movements, serving as the critical first step in bringing static visuals to dynamic life.

Following the reconstruction of 3D meshes from 2D imagery, the resulting geometry undergoes a crucial process of rigging with skeletal structures facilitated by ‘UniRig’. This rigging establishes a digital framework, analogous to the bones and joints of a living creature, that allows for realistic and controllable motion. By defining how different parts of the 3D model connect and move in relation to each other, ‘UniRig’ enables animators – or, in future iterations, automated systems – to pose and animate the model convincingly. The skeletal structure serves as a control system, translating intended movements into deformations of the 3D mesh, ultimately bringing the reconstructed form to life with dynamic and believable motion.

The ultimate aim of this research extends beyond current capabilities, envisioning a fully automated system for animal motion synthesis. Future development will concentrate on seamlessly integrating 3D reconstruction and skeletal rigging into a cohesive pipeline, accepting diverse inputs – including single images, video footage, or even artistic sketches – and autonomously generating realistic and varied animal movements. This automated process promises to significantly reduce the time and expertise required for creating animal animations, with potential applications spanning entertainment, scientific visualization, and behavioral studies. The anticipated outcome is a versatile tool capable of producing compelling and believable animal motion with minimal human intervention, opening new avenues for creative content generation and research.

Our motion dataset provides a comprehensive overview of diverse movement patterns.

The research meticulously details a system capable of translating textual descriptions into believable animal movements, irrespective of skeletal differences. This pursuit of adaptable motion generation echoes a design philosophy where functionality doesn’t preclude elegance. As Fei-Fei Li aptly states, “AI is not about replacing humans; it’s about augmenting our capabilities and helping us solve problems we couldn’t solve before.” This framework, by enabling the creation of diverse and realistic animal locomotion from simple text prompts, exemplifies that augmentation. The topology-aware autoregressive model demonstrates how a deep understanding of underlying structure – in this case, skeletal geometry – can unlock a poetic and fluid interface between intention and action, creating a harmony between form and function.

Beyond the Bone Structure

The capacity to sculpt motion from mere textual description, even across the bewildering diversity of animal forms, represents a considerable stride. However, a truly elegant solution should not require explicit skeletal geometry as input. The current framework, while demonstrating impressive results, still operates with a subtle dependency-a reliance on pre-defined topology. Future work must strive for a system that infers structural constraints implicitly, deriving form from function and movement, rather than the other way around. A good interface is invisible to the user, yet felt; similarly, a superior motion generator should abstract away from the underlying mechanics, presenting only the seamless illusion of life.

Furthermore, the limitations inherent in any autoregressive model-the tendency towards repetition, the occasional drift into implausibility-remain. A compelling avenue for exploration lies in integrating principles of physics-based simulation, not to replace the learned behavior, but to constrain it, to subtly nudge the generated motions towards greater physical realism. Every change should be justified by beauty and clarity; simply increasing the scale of the training dataset, while helpful, is not a substitute for fundamental algorithmic refinement.

Ultimately, the true measure of success will not be the fidelity with which this system mimics existing animal locomotion, but its capacity to invent new forms of movement-to generate motions that are both plausible and surprising, that reveal previously unimagined possibilities within the realm of biomechanics. The challenge, then, is not merely to replicate life, but to expand its definition.

Original article: https://arxiv.org/pdf/2512.10352.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/