Light SAED: A Robust, Lightweight, and Culturally Adaptable Cross-Modal Transformer for Sarcasm-Aware Emotion and Intensity Detection in Multimodal Tweets

Abstract

Detecting emotions on social media is crucial for applications such as mental health monitoring and brand analytics. However, existing models often overlook inter-modal interactions, disregard cultural variations, and rely on computationally expensive architectures. We propose LightSAED, a lightweight cross-modal transformer that fuses textual, visual, and emoji data to detect emotions, sarcasm, and emotional intensity in tweets. LightSAED introduces three key innovations: (1) a dynamic cross-modal attention mechanism for effective multimodal fusion, (2) a dedicated sarcasm detection sub-layer trained with explicit supervision, and (3) a hierarchical cultural adaptation layer leveraging region-specific embeddings based on sociolinguistic features. We also present TwemoInt++, a curated dataset of 50,000+ tweets, annotated for emotion, sarcasm, and intensity, stratified into ten culturally defined regions. Extensive experiments show that LightSAED outperforms state-of-the-art baselines, improving emotion accuracy by 6.2% and sarcasm detection F1-score by 9.8%. Robustness tests against noisy data and adversarial examples further validate its reliability. To enhance efficiency, pruning and 8-bit quantization reduce inference time by 42% and model size by 63%, enabling real-time edge deployment on resource-constrained devices. Despite its advancements, challenges remain in handling ambiguous cultural cues and low-resource languages, paving the way for future enhancements