Researchers from the State Key Laboratory of Automotive Safety and Energy at Tsinghua University, led by Professor Shichun Yang, have developed a novel framework called the Synesthesia of Vehicles (SoV) to enhance the safety of autonomous vehicles (AVs) by predicting tactile excitations from visual inputs.
Autonomous vehicles rely on a combination of visual and optical sensors to navigate and make decisions. However, these sensors often fail to detect critical road-induced excitations, such as bumps or potholes, which are essential for a vehicle’s dynamic control and safety. Inspired by human synesthesia—a phenomenon where one sensory experience triggers another—the researchers proposed the SoV framework to address this limitation.
The SoV framework employs a cross-modal spatiotemporal alignment method to tackle the temporal and spatial disparities between visual and tactile data. This alignment ensures that the visual inputs accurately correspond to the tactile excitations experienced by the vehicle. Additionally, the researchers developed a visual-tactile synesthetic (VTSyn) generative model using latent diffusion. This model enables unsupervised high-quality tactile data synthesis, allowing the AV to predict tactile feedback from visual inputs without the need for extensive labeled data.
To validate their approach, the researchers collected a multi-modal dataset using a real-vehicle perception system. This dataset encompassed diverse road and lighting conditions, providing a robust testing ground for the SoV framework. Extensive experiments demonstrated that the VTSyn model outperformed existing models in temporal, frequency, and classification performance. This enhancement in tactile perception enables AVs to proactively respond to road conditions, thereby improving overall safety.
The practical applications of this research for the energy sector are significant. As the energy industry increasingly integrates autonomous vehicles into its operations—such as for transportation of goods, personnel, or equipment—the enhanced safety provided by the SoV framework can reduce the risk of accidents and downtime. This can lead to more efficient and reliable operations, ultimately contributing to the sustainability and productivity of energy sector activities.
The research was published in the journal IEEE Transactions on Intelligent Vehicles.
This article is based on research available at arXiv.