Robots That Feel: Understanding Hardness Through Touch and Sight

by

Author: Denis Avetisyan

Researchers have developed a new framework enabling robots to assess object hardness – like determining the ripeness of fruit – and articulate their reasoning in human-understandable language.

TactEx combines multimodal sensing with explainable AI to achieve human-like hardness estimation and language grounding in robotic manipulation.

Accurately perceiving object properties like hardness remains a challenge for robots lacking human-like tactile intelligence. To address this, we present ‘TactEx: An Explainable Multimodal Robotic Interaction Framework for Human-Like Touch and Hardness Estimation’, a novel system that integrates vision, tactile sensing, and language to estimate object hardness and provide understandable explanations for its decisions. This framework achieves statistically significant hardness discrimination-demonstrated through fruit-ripeness assessment-by fusing tactile data with visual cues and guidance from large language models. Could this approach pave the way for more intuitive and reliable human-robot collaboration in complex manipulation tasks?

The Deficit of Feeling: Beyond Vision in Robotic Manipulation

Robotic manipulation frequently falters not due to a lack of powerful actuators, but a deficit in feeling. Traditional robotic systems, reliant on vision and pre-programmed movements, struggle with the subtle interplay required for nuanced object interaction. Unlike humans, who effortlessly adjust grip force based on tactile feedback, robots often apply either insufficient or excessive pressure, leading to dropped objects or damage. This limitation stems from the difficulty in replicating the density and complexity of human skin – the vast network of mechanoreceptors that provide information about texture, pressure, and slippage. Consequently, tasks requiring delicate handling, assembly of fragile components, or even simple exploration of an object’s properties remain significant hurdles, highlighting the crucial need for advancements in robotic tactile sensing and the development of algorithms capable of interpreting that data to achieve truly dexterous manipulation.

Determining an object’s hardness is paramount for robotic systems tasked with delicate manipulation and precise quality control, but achieving accurate estimations presents a formidable engineering challenge. Unlike vision, which can be easily fooled by appearance, reliably gauging material properties through touch requires sophisticated sensors and algorithms capable of discerning subtle variations in deformation and force. This is particularly crucial when handling fragile items – imagine a robot assembling electronics or sorting produce – where excessive force could cause damage. Current approaches struggle with inconsistencies arising from surface texture, object geometry, and the inherent variability in material response; a seemingly firm apple, for instance, might yield unexpectedly to pressure. Overcoming these limitations necessitates advancements in tactile sensor design, coupled with machine learning models trained to interpret complex tactile data and predict material hardness with a level of precision approaching human sensitivity.

Current robotic systems frequently depend on visual data to assess an object’s properties, a strategy proving increasingly fallible in real-world scenarios. Reliance on cameras and image processing introduces vulnerabilities to fluctuating light levels, where shadows and glare can distort perceived textures and shapes. More critically, visual systems struggle with occlusion – when parts of an object are hidden from view – leading to incomplete or inaccurate assessments of material characteristics. This limitation hinders a robot’s ability to confidently interact with objects in cluttered environments, particularly when delicate manipulation or precise quality control is required, as the system may misjudge an object’s hardness or fragility based on incomplete visual information. Consequently, researchers are actively exploring alternative sensing modalities, like tactile sensors, to overcome the inherent unreliability of purely vision-based approaches.

Synergy of Senses: Integrating Vision and Touch

This research introduces a multimodal framework designed to enhance object understanding by integrating data from both visual and tactile sensors. The system capitalizes on the complementary strengths of each modality; vision, implemented with Grounded-SAM (GSAM), provides global contextual awareness and object segmentation for targeted manipulation. Simultaneously, a GelSight-Mini sensor captures high-resolution tactile data, delivering detailed information regarding contact forces and surface characteristics. This fusion allows for a more robust and accurate perception of objects, overcoming limitations inherent in relying on a single sensory input and enabling more reliable robotic interaction.

Grounded-SAM (GSAM) facilitates precise robotic manipulation by providing accurate object segmentation in robotic environments. This system builds upon the Segment Anything Model (SAM) by grounding its predictions with 3D scene information obtained from a RealSense camera. Specifically, GSAM projects 2D SAM segmentations into 3D space, enabling the robot to identify and locate target objects even with partial visibility or occlusion. The resulting 3D segmentation masks are then used to generate collision-free trajectories for the robotic end-effector, guiding it towards the desired object for interaction. Evaluations demonstrate GSAM consistently outperforms standard SAM-based approaches in cluttered scenes, enabling successful grasping and manipulation rates exceeding 95% in controlled experiments.

The GelSight-Mini sensor utilizes a high-resolution camera and diffuse illumination to capture detailed tactile information during object contact. This sensor provides data representing surface normals, contact forces, and slippage, enabling the system to discern subtle variations in surface properties like texture and curvature. The sensor’s resolution allows for the detection of features at the millimeter scale, providing a dense representation of the contact interface. Captured data is then processed to estimate key parameters, including contact location, force magnitude, and direction, which are critical for robust object manipulation and identification.

The tactile perception module utilizes a ResNet50 convolutional neural network combined with a Long Short-Term Memory (LSTM) recurrent neural network to process sequential data from the GelSight-Mini sensor. This ResNet50+LSTM architecture is specifically trained to estimate object hardness based on tactile input. Performance metrics demonstrate a root mean squared error (RMSE) of 4.3 when predicting hardness values, indicating a quantifiable level of accuracy in the system’s tactile perception capabilities. The resulting output of this module provides critical information for object understanding and manipulation tasks.

TactEx: An Explainable Framework for Robotic Interaction

TactEx is a robotic interaction framework designed to estimate the hardness of objects by integrating data from vision, tactile sensing, and natural language processing. The system employs a multimodal approach, combining visual data regarding object appearance with force and texture information gathered through tactile sensors. This fused sensory input is then processed to determine an object’s hardness, and the resulting estimation is communicated to the user via natural language. The framework’s architecture allows for the correlation of visual features and tactile measurements, improving the robustness and accuracy of hardness assessment beyond what is achievable with single-modality approaches.

The TactEx framework utilizes a Large Language Model (LLM) to convert quantitative hardness estimations into human-readable explanations, facilitating improved user comprehension and confidence in robotic assessments. Evaluation of these generated explanations, based on metrics of accuracy, completeness, and clarity/coherence, yielded average scores of 4.19, 4.94, and 4.92, respectively. This indicates a high degree of fidelity and understandability in the LLM’s translation of data into natural language, supporting effective communication of robotic perception to human users.

The TactEx framework incorporates interactive guidance, enabling users to influence the hardness estimation process in real-time. This functionality is implemented through a command-based interface where users can issue intuitive instructions to the robotic system. These commands allow for focused data acquisition – for example, requesting additional tactile exploration of a specific region or specifying a desired level of pressure during contact. The system then integrates this user input to refine its internal hardness estimate, effectively creating a collaborative estimation process. This iterative refinement loop improves both the accuracy of the hardness assessment and the user’s confidence in the results.

Experimental results indicate that the TactEx framework achieves a high degree of accuracy in hardness estimation, as evidenced by a Spearman correlation coefficient of 0.88. Furthermore, the system demonstrates statistically significant performance in ripeness ranking across five distinct fruit types (p < 0.01). This statistically significant result suggests the framework can reliably differentiate ripeness levels based on tactile and visual data, providing a quantifiable measure of its performance beyond simple hardness assessment.

From Precision to Sustainability: Expanding the Scope of Tactile Robotics

Automated quality control benefits significantly from the precision of tactile sensing, and the TactEx framework offers a robust solution for assessing the hardness of manufactured parts. By accurately measuring material properties through touch, the system identifies inconsistencies or defects that may not be visible to the naked eye. This capability is crucial for maintaining product consistency across large-scale production runs, reducing the likelihood of faulty items reaching consumers. Unlike traditional methods reliant on manual inspection or destructive testing, TactEx provides a non-invasive and repeatable process, minimizing waste and increasing efficiency. The framework’s sensitivity allows for the detection of subtle variations in hardness, enabling manufacturers to proactively address potential issues and optimize their production processes, ultimately leading to improved product reliability and customer satisfaction.

Beyond industrial quality control, the TactEx framework presents a compelling solution for automating tasks within agriculture. Robotic systems, guided by this technology, can now determine fruit and vegetable ripeness through the precise assessment of hardness. This capability moves beyond simple visual cues, allowing for the identification of produce at peak maturity – even when external appearances are misleading. By gauging the subtle changes in firmness, robots can selectively harvest crops, minimizing damage during picking and significantly reducing food waste. The framework’s application promises increased harvesting efficiency and a more sustainable approach to food production, paving the way for automated farms and optimized yields.

The ability to precisely assess produce ripeness through tactile sensing directly addresses significant challenges in agricultural harvesting. Traditional methods often result in substantial damage to fruits and vegetables during collection, leading to considerable post-harvest waste. By employing a system capable of identifying optimal ripeness – and thus firmness – robots can gently select produce, minimizing bruising and extending shelf life. This not only reduces economic losses for growers but also contributes to a more sustainable food supply chain. Furthermore, the automation of harvesting through tactile feedback enables faster and more efficient collection, particularly crucial for time-sensitive crops, ultimately increasing overall yield and reducing labor costs.

The TactEx framework’s versatility is underscored by its successful performance across a range of testing scenarios – Sc1 through Sc4 – demonstrating a robust ability to generalize beyond controlled laboratory conditions. This adaptability stems from its unified approach to multimodal interaction, effectively integrating tactile sensing with other data streams to create a comprehensive understanding of the environment. By consolidating these inputs into a single platform, TactEx equips robots with the capacity to navigate complexity and respond to unforeseen circumstances, ultimately enhancing their reliability and allowing them to execute intricate tasks – from industrial quality control to delicate fruit harvesting – with increased precision and consistency.

The pursuit of robotic understanding, as demonstrated by TactEx, necessitates a rigorous distillation of input into actionable insight. The framework’s ability to correlate visual and tactile data with linguistic explanations embodies a commitment to clarity over complexity. This echoes Blaise Pascal’s sentiment: “The eloquence of angels is not in the words they use, but in the silence they maintain.” TactEx strives for that same elegant efficiency-a system that ‘speaks’ through accurate hardness estimation and transparent reasoning, minimizing superfluous data or opaque processes. The core concept of language grounding, allowing robots to articulate their assessments, is not merely about communication, but about revealing the essential logic behind the perception.

What Remains?

The pursuit of robotic touch invariably circles back to the elusiveness of ‘understanding’. TactEx offers a valuable reduction – a framework for correlating sensory input with linguistic description. Yet, the core problem isn’t solved, merely shifted. Accurate hardness estimation, even when explained, remains a proxy. The robot doesn’t know ripeness; it recognizes a pattern. The true challenge lies not in adding more modalities, but in distilling a functional definition of ‘understanding’ within a purely mechanistic system. The framework’s current reliance on pre-defined hardness scales, for instance, subtly begs the question of inherent human subjectivity.

Future work will likely focus on active learning strategies. Allowing the robot to formulate its own questions-to probe the world rather than merely react to it-could yield more robust and generalizable models. However, a more radical path exists: embracing the inherent ambiguity. Perhaps the goal isn’t to eliminate uncertainty, but to quantify and represent it. A robot that acknowledges its own limitations may ultimately be more reliable-and more useful-than one that feigns omniscience.

The elegance of any system is best measured by what it leaves out. TactEx begins this necessary paring. The next iteration should not strive for completeness, but for essentiality. The question is not how much can be added, but what can be discarded – leaving only the necessary scaffolding for functional interaction.

Original article: https://arxiv.org/pdf/2602.18967.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/

See also:

2026-02-24 17:30