abstract
- Empathy in human-human interactions encompasses nonverbal behaviors that are often overlooked by state-of-the-art empathic perception models, despite extensive progress in the text domain. In this article, we investigate empathy classification in dyadic interactions using audiovisual data, specifically arousal and valence values extracted from facial expressions, head gestures, and textual content derived from speech. We propose a novel cue integration framework that augments precomputed text-based features with video-derived emotional cues and applies them to a pattern-based empathy classifier. Furthermore, we introduce a ranking module that provides interpretability by revealing the relative influence of empathy cues. Our approach is evaluated in two distinct settings: a controlled laboratory environment in which multiple pairs of participants engage in face-to-face conversation, and a real-world school setting involving group interactions among children. Results show that our multimodal empathy classifier generalizes better than deep learning baselines and on par with state-of-the-art large language models across both dyadic and group interaction scenarios, without requiring retraining.