A Multimodal Learning Approach for Protecting the Metro System of Medellin Colombia Against Corrupted User Traffic Data

abstract

Highlights: What are the main findings? The novel concepts of Self-Supervised Tabular Learning and Large Multimodal Models are integrated to create a multimodal learning solution for auditing the metro system of Medellin, Colombia. On publicly available data, in an offline process, corrupted user traffic is detected, explained, and corroborated using SHAP values and the image understanding process of a Large Multimodal Model. What are the implications of the main findings? A visibility layer is added for smart proper policy making, also shedding light on the pros and opportunities of the current publicly available data. Each abnormal passenger behavior is not only flagged, but a thorough justification is also provided to enhance the robustness of the detections. A critical task in infrastructure security is to model user traffic in transportation systems to alert whenever anomalous behavior is observed. Discerning those abnormal samples is possible by auditing the available data, which then enables proper policy making to guarantee fair tariffs and the design of strategies to tackle problems such as passenger congestion. In this paper, we present an offline cybersecurity approach for the multimodal modeling of user traffic for the Colombian metro. To identify the anomalies, we design custom Deep Autoencoders based on the embeddings produced by the Self-Supervised Learning TabNet architecture. Additionally, we provide explainability through a SHAP-based component and the analysis of external image data using LLaVA as the selected Large Multimodal Model. The results indicate that most problems that occur on one metro line also affect the other, demonstrating the interconnectivity of the metro system, a crucial aspect that motivates the coordinated emergency response to improve the passenger travel experience. Although the detected problems might already have been identified and reported on social media, the transparency provided helps create confidence when an abnormality is observed, and in case there is no backup information on our official external data sources, it represents an alert to examine it more deeply, becoming an intelligent assessment tool for the metro. This article also sheds light on the potential of the publicly available dataset used and the importance of expanding its existing variables and information. © 2025 by the authors.