Cross-Modal Federated Learning Security: Extending Prototype Consistency for Backdoor-Resilient Multimodal Representation Learning - Henri C. Berry; Davide Allen; Sunil Jha (Computational Intelligence Systems, 2026)
Multimodal federated learning represents a paradigm shift in distributed machine learning, enabling collaborative model training across heterogeneous data sources that span images, text, audio, and sensor streams. However, the integration of multiple modalities introduces novel attack surfaces, particularly backdoor poisoning, where an adversary embeds hidden triggers across one or more modalities to cause targeted misclassification at inference time. Existing defenses rooted in unimodal settings often fail to generalize to cross-modal environments due to the complex interplay between representation spaces. This paper proposes a systematic extension of prototype consistency mechanisms—originally developed for single-modal and split learning contexts—to secure cross-modal federated learning architectures. We examine the structural trade-offs between robustness and utility when enforcing alignment constraints on multimodal prototype clusters, and we analyze how such constraints interact with the inherent heterogeneity of client data distributions and modality-specific encoders. Beyond technical design, we discuss the governance and fairness implications of deploying backdoor-resilient multimodal systems in critical infrastructures such as healthcare diagnostics and autonomous navigation. Our analysis draws on recent advances in prototype-based defenses, including the ProtoGuard framework for vertical split learning, and situates these within the broader landscape of secure representation learning. The paper concludes with a forward-looking discussion on sustainability, policy requirements, and the need for standardized evaluation benchmarks for cross-modal federated security.