Towards Secure and Robust Vision-Language Models in Autonomous Driving: A Survey for Perception-Oriented and Decision-Oriented Attacks

Vision-Language Models (VLMs) have been widely adopted in autonomous driving (AD) for their strong generalization capabilities and interpretability. However, they remain inherently vulnerable to robustness challenges, particularly adversarial examples, which pose significant safety risks. This paper presents a comprehensive survey of attacks on AD VLMs, introducing a novel taxonomy that categorizes attacks into two types: Perception-oriented Attacks, which manipulate input data to mislead the perception process, and Decision-oriented Attacks, which exploit vulnerabilities in decision-making and interaction processes. Following this classification, the paper systematically reviews existing attack methods, their limitations, and the key challenges they present. It also explores future directions and potential applications to enhance AD VLM robustness and performance. This work aims to establish a foundation for developing more secure and reliable AD VLM systems.