Engagement isn't one behavior — it shows up as gaze, backchannels, facial responsiveness, and more, all varying across people and settings. Instead of predicting engagement labels directly, we pretrain a two-stream encoder to tell active listening apart from silence in paired interaction video — a task that needs no manual labels but forces the model to capture how the speaker and listener respond to each other. That single frozen encoder then transfers to three downstream behaviors across three datasets, reaching CCC 0.742 on NoXi engagement (matching audiovisual methods with vision alone), 80.24% on clinical focus-of-attention, and above-chance backchannel detection it was never trained for.
Gaze, listener feedback, and perceived engagement are usually studied in isolation, each with its own model and labels. We hypothesize they all arise from the same thing: inter-participant dynamics — how one person's behavior is shaped by the other's. Learn that, and it transfers.
One video encoder processes the speaker's face — expressions, motion, and timing as they hold the floor.
A weight-tied encoder processes the listener — nods, gaze, and micro-responses shaped by the speaker's presence.
Bidirectional attention lets each stream attend to the other, capturing the coupling that defines a real interaction.
The same frozen weights transfer to focus, backchannels, and engagement — no architectural changes.
The speaker and listener are encoded by a weight-tied backbone, coupled through bidirectional cross-attention into a single fused representation, then read out by three lightweight task heads. Toggle the pretext state to trace what flows through.
The pretrained two-stream encoder is compared against a randomly-initialized twin and a single-stream variant, then against published task-specific methods.
| Method | Mode | In-house FOVA | AMI FOVA | AMI BC | NoXi Eng. |
|---|---|---|---|---|---|
| Random init | FT | 68.76 | 52.16 | 52.66 | 0.521 |
| Single-stream | FT | 74.21 | 55.37 | 54.37 | 0.616 |
| Ours (SSL) | LP | 73.63 | 60.31 | 56.67 | 0.652 |
| Ours (SSL) | FT | 80.24 | 62.68 | 57.98 | 0.742 |
Accuracy for in-house FOVA; balanced accuracy for AMI FOVA and backchannel (BC); CCC for engagement. LP = linear probe, FT = fine-tune. The same encoder weights are used for every task — only the head differs.
| Method | Modality | CCC ↑ |
|---|---|---|
| CLIP + MLP | V | 0.556 |
| OpenSMILE + MLP | A | 0.691 |
| w2v-BERT + MLP | A | 0.701 |
| MM'23 Baseline | A+V | 0.710 |
| Yang et al. | A+V | 0.724 |
| DCTM | A+V | 0.745 |
| Ours (LP) | V | 0.652 |
| Ours (FT) | V | 0.742 ± 0.002 |
Our visual-only model matches the audiovisual DCTM (0.745) using no audio. Result is the mean over 3 seeds; the ±0.002 spread is far smaller than our margins over single-stream (+0.13) and random init (+0.22), so the gain is not seed-driven. All methods use the same NoXi validation split (test labels are held by the organizers).
| Method | Balanced Acc. ↑ |
|---|---|
| Head pose + HMM | 55.6 |
| ICAF | 56.8 |
| Ours (LP) | 60.31 |
| Ours (FT) | 62.68 |
The linear probe alone already exceeds both geometric-feature baselines — the pretrained features encode gaze-relevant structure with no task-specific adaptation.
Each design choice, isolated. Pretraining and inter-participant modeling matter most; face crops beat full frames; the backbone choice is secondary.
The encoder is pretrained entirely on the AMI meeting corpus — adults, multi-party, seated. We then test it on an in-house dataset of 114 participants (ages 8–49) in dyadic clinical interactions, including participants with psychiatric conditions that meaningfully change social behavior.
Despite that domain shift, fine-tuning reaches 80.24% FOVA accuracy (vs. 74.21% single-stream, 68.76% random init), and the linear probe alone (73.63%) nearly matches the single-stream fine-tuned result. The inter-participant dynamics learned during pretraining reflect general properties of face-to-face interaction, not just the meeting domain.
@inproceedings{chappa2026ssl,
title = {Self-Supervised Representation Learning for Heterogeneous
Behavioral Expressions of Social Engagement},
author = {Chappa, Naga VS Raviteja and Yankowitz, Lisa and
Nair, Gokul M and Sariyanidi, Evangelos and
Zampella, Casey J. and Campbell, Kathleen and
Guthrie, Whitney and Herrington, John D. and
Schultz, Robert T. and Tun\c{c}, Birkan},
booktitle = {Proceedings of the 28th ACM International Conference
on Multimodal Interaction (ICMI)},
year = {2026}
}Partially supported by the Office of the Director (OD), NICHD, and NIMH under grants R01MH122599, R01MH118327, P50HD105354, and R21HD102078; and the IDDRC at CHOP/Penn.