An ADOS-2-Aligned Multimodal Architecture for Clinical Autism Assessment

Abstract

The ADOS-2 groups its observations into domains and combines them into domain totals before it applies a cutoff (Lord et al., 2012), so I built a pipeline that models each behavioral domain separately and combines the domains only at the end. Three domain-specific classifiers train on the social, communication, and repetitive-behavior feature subsets, and a logistic-regression meta-learner combines their probabilities into a consensus prediction. The output layer writes two reports from a single analytical run; a research report gives an ML team performance, cross-validation, feature importance, and dataset diagnostics, and a clinical report gives a clinician item-level behavioral scores, model consensus, uncertainty, session quality, and an explicit “requires clinical review” flag when the ensemble disagrees. The pipeline runs entirely on synthetic data that I generated to overlap the way clinical groups overlap, and a runtime check flags any held-out AUC above 0.95 as more likely simulation drift than model skill.

The clinical reality a simulator has to carry

The ADOS-2 is a standardized, examiner-administered observation, typically 40 to 60 minutes with a child who may or may not cooperate, scored by a clinician whose training may differ from the last examiner’s (Lord et al., 2012). An automated pipeline would add video with face detection that may not hold throughout the session. Features that come out of a process like that are noisy, and the groups they describe overlap.

Labeled schematic of the multimodal feature-extraction pipeline. Video and audio input streams flow through computer-vision and NLP modules, producing behavioral features that map to the pipeline's ADOS-2-style domains.

FIG. 01 Multimodal feature-extraction design. In a deployed system, video would feed computer-vision modules (MediaPipe gaze and face tracking, skeletal pose estimation) and audio would feed NLP modules (Whisper speech-to-text, prosodic features). In this pipeline, simulator classes generate the features those modules would produce; no real video or audio is processed. The item labels are a simplified scheme I defined for the pipeline, modeled on ADOS-2 codes.

I wrote a ClinicallyRealisticDataGenerator to build that ambiguity in on purpose. I drew gaze-to-face percentage from N(52, 22) for classic ASD profiles, N(68, 18) for high-functioning ASD, N(58, 16) for typical development with false-positive risk factors, and N(73, 15) for typical development, so the distributions overlap heavily. I drew spontaneous vocalizations from N(32, 15) for ASD and N(38, 15) for non-ASD profiles, scaled by age. I added gaze measurement noise with a standard deviation of 17 percentage points and then 25% to 35% multiplicative noise on every feature. Session quality acts as a global confound; a low-quality session lowers face detection, audio clarity, and session completeness together. Each feature loses 8% to 15% of its values at random, and in 5% of sessions two or three features go missing together, because a problematic session tends to affect several measurements at once. In 10% of audio recordings, a simulated technical failure cuts detected vocalizations to 60% of the generated count.

I overweighted the cases that clinicians who run ADOS assessments regularly will recognize, making 35% of profiles challenging:

High-functioning ASD with subtle presentation
Borderline ASD that meets some but not all diagnostic criteria
Masked ASD, where the individual compensates to hide traits
Typical development with false-positive risk factors
Developmental delay that resembles ASD on surface features
Anxiety-driven behavior that produces ASD-like surface features

Simulation parameter	Value	Rationale
Participants	1,200	Large enough to train and cross-validate eight model families
ASD prevalence	22%	Resembles a specialty referral clinic
Measurement noise	25% to 35% per feature	Sensor error, poor lighting, and background audio
Missing values	8% to 15% per feature, plus 5% of sessions with correlated gaps	Occluded camera angles and dropped audio
Challenging profiles	35%	Keeps the models from training on textbook cases alone

The condensed excerpt below shows the generator’s logic for one feature, gaze-to-face percentage; every other feature follows the same pattern of profile-specific distributions followed by noise.

# Condensed from ClinicallyRealisticDataGenerator (gaze-to-face percentage).
# Group distributions overlap by design; noise is added afterward.
GAZE_MEAN_SD = {
    'asd_classic':          (52, 22),
    'asd_high_functioning': (68, 18),
    'typical_fp_risk':      (58, 16),
    'typical':              (73, 15),
}
mean, sd = GAZE_MEAN_SD[profile_key]
features['gaze_to_face_percent'] = np.clip(np.random.normal(mean, sd), 5, 98)

# Every feature then receives 25-35% multiplicative measurement noise.
noise_level = np.random.uniform(0.25, 0.35)
features[name] = value + np.random.normal(0, abs(value) * noise_level)

On the generated data, the groups overlapped the way I intended for most features; the mean gaze-to-face percentage was 52.6 for ASD and 67.7 for non-ASD profiles, and the mean vocalization count was 28.7 against 35.9. Phrase repetition separated the groups far more than any other feature (means 0.14 against 0.03), a property of how I parameterized that feature, and that separation shapes the feature-importance results below.

Feature Extraction and Domain Mapping

The simulator produces 10 behavioral features, and I assigned each to one of three domains modeled on the ADOS-2 structure. The clinical report converts each simulated feature into an item code from 0 (typical) to 2 (markedly atypical), following the pipeline’s item scheme.

Hierarchical tree diagram showing video and audio input streams branching into computer-vision and NLP feature extractors, each leaf node labeled with the pipeline's ADOS-2-style item label.

FIG. 02 Hierarchical extraction tree mapping the simulated video and audio streams to the pipeline's ADOS-2-style item labels.

Domain group	Feature	Measurement
Social	Gaze-to-face	% of session with gaze directed at a face
Social	Mutual gaze	Count of mutual gaze episodes
Social	Social smiles	Count
Social	Joint attention	Count of successful bids
Communication	Response to name	Latency in seconds
Communication	Spontaneous vocalizations	Count
Communication	Phrase repetition	Rate
Repetitive behavior	Hand flapping	Duration in seconds
Repetitive behavior	Functional play	% of session using toys functionally
Repetitive behavior	Close visual inspection	Duration in seconds

The Machine Learning Architecture

The ADOS-2 algorithm combines selected item codes into a Social Affect total and a Restricted and Repetitive Behavior total, and the cutoffs apply to scores built from that domain structure (Lord et al., 2012). I built the model to mirror that structure. Three domain-specific classifiers train independently, a Random Forest on the social features, an XGBoost model on the communication features, and a regularized logistic regression on the repetitive-behavior features, and each emits a probability of ASD class membership from its own domain alone. A logistic-regression meta-learner takes those three probabilities as input and produces the consensus prediction. The base models never see each other’s feature subsets, and the meta-learner never sees the raw features.

# Late fusion over the pipeline's three domain groups.
# Each base classifier sees only the features in its own domain.
feature_groups = {
    'social':        ['gaze_to_face_percent', 'mutual_gaze_episodes',
                      'social_smiles_count', 'joint_attention_bids'],
    'communication': ['response_to_name_latency', 'spontaneous_vocalizations',
                      'phrase_repetition_rate'],
    'repetitive':    ['hand_flapping_duration', 'functional_play_percent',
                      'close_visual_inspection'],
}
domain_models = {
    'social':        RandomForestClassifier(n_estimators=80, max_depth=6, class_weight='balanced'),
    'communication': XGBClassifier(n_estimators=80, max_depth=4, eval_metric='logloss'),
    'repetitive':    LogisticRegression(C=0.3, class_weight='balanced'),
}
for domain, model in domain_models.items():
    model.fit(X_train[feature_groups[domain]], y_train)

# The meta-learner combines the three domain probabilities.
meta_train = np.column_stack([m.predict_proba(X_train[feature_groups[d]])[:, 1]
                              for d, m in domain_models.items()])
meta = LogisticRegression(C=0.5).fit(meta_train, y_train)

The design pays off in interpretation. When the meta-learner produces a high consensus probability, the per-domain probabilities show which domain drove it; a case with a high social probability and middling communication and repetitive-behavior probabilities is a different clinical picture from one in which all three domains agree, and the clinical report shows that decomposition directly. A single classifier over all features would produce a consensus number with no domain-level explanation.

Around the late fusion I ran conventional comparison models with conservative settings. The supervised baselines are logistic regression with an elastic-net penalty (C = 0.1), a Random Forest, and XGBoost with depth-4 trees, a learning rate of 0.05, and L1 and L2 regularization. The semi-supervised models are self-training with a confidence threshold of 0.9 and label spreading with 10 nearest neighbors and alpha 0.5. A Bayesian ensemble of five independently seeded Random Forests with 70% bootstrap subsampling supplies the uncertainty estimate, and the pipeline flags any case whose ensemble standard deviation exceeds 0.2 for clinical review. I trained on the 75.7% of sessions that passed the simulated quality check, as a clinic would set aside a session it could not score.

What realistic performance looks like

I evaluated every model on a held-out 25% test split. Test AUCs ranged from 0.931 to 0.974, and the realism check flagged the two models above 0.95 with the warning “Suspiciously high AUC, check for overfitting.” I wrote that check because, on synthetic data built to overlap, an AUC above 0.95 more likely means the simulation drifted toward separability than that the model got that good, and I encoded the assumption as a runtime check so that every run applies it.

Model	Test AUC	F1	Sensitivity	Specificity	Realism check
Random Forest	0.933	0.696	0.80	0.859	passes
XGBoost	0.946	0.625	0.50	0.972	passes
Logistic Regression	0.974	0.844	0.95	0.915	flagged
Self-Training	0.945	0.744	0.80	0.901	passes
Label Spreading	0.970	0.667	0.50	1.000	flagged
Bayesian Ensemble	0.932	0.681	0.80	0.845	passes
Bayesian Calibrated	0.931	0.732	0.75	0.915	passes
Late Fusion	0.932	0.683	0.70	0.901	passes

Five-fold cross-validation tempered the single-split picture; label spreading cross-validated to AUC 0.891 (SD 0.022) and XGBoost to 0.953 (SD 0.017). The label-spreading result shows how much a single favorable split can flatter a model, since the same model scored 0.970 on the held-out split.

Schematic ROC curve arching into the warning zone above AUC 0.95, annotated with the logistic regression model's held-out AUC of 0.974.

FIG. 03 Schematic ROC curve for the logistic regression model, whose held-out AUC of 0.974 tripped the 0.95 realism check. The curve shape is illustrative; the AUC is the measured value.

Aggregated feature importance ranked phrase repetition rate first by a wide margin (0.514), followed by hand-flapping duration (0.129), close visual inspection (0.126), response-to-name latency (0.068), and social smiles (0.042). The ranking reflects the simulator more than clinical truth; phrase repetition was the feature I parameterized with the least group overlap, and the models found it. I read the dominance of one feature, together with the two flagged AUCs, as a signal to add overlap to phrase repetition in the next version of the generator.

Horizontal bar chart of aggregated feature importance for the top five features, with phrase repetition rate far ahead of hand-flapping duration, close visual inspection, response-to-name latency, and social smiles.

FIG. 04 Top-five aggregated feature importance across the supervised models. Phrase repetition dominates because the simulator gave it the least group overlap.

One analytical run, two stakeholder-specific reports

I built the output layer to write two reports from the same analytical run, a research report for the ML team that owns the model and a clinical report for the clinician who ordered the assessment. The two audiences ask different questions; an ML reviewer needs to audit the model, and a clinician needs to weigh the evidence for one child.

The research report gives model performance across all eight models, the spread of AUC and F1, cross-validation with confidence intervals, aggregated feature importance, dataset diagnostics (participants, prevalence, quality pass rate, and missing values), group differences with effect sizes and overlap scores, and the realism check’s verdict. I generate it from a fixed HTML template, so successive runs are directly comparable.

The clinical report answers a different question, because a probability of 0.67 is not a clinical finding. For one participant, it gives an assessment summary, each model’s prediction, an uncertainty analysis, item-level codes on the 0 to 2 scale grouped by domain, the features that deviated from typical ranges, areas of strength, data limitations, and recommendations. The Bayesian ensemble reports whether its five Random Forests agreed; when their standard deviation exceeds 0.2, the report flags the case as requiring clinical review and gives no classification.

The report also shows data quality beside the prediction (video clarity, face detection rate, audio clarity, and session completeness), since a prediction from a session with 60% face detection deserves less weight than one from a session that passed every quality check. The clinician, who holds the training, the context, and the legal and ethical responsibility for the diagnosis, sees the prediction and the quality of the evidence behind it, and the ML team sees the diagnostics it needs; I keep the two reports separate so neither audience has to read past the other’s material.

Limitations

The pipeline runs on synthetic data, because real ADOS-2 data is protected and scarce; every performance number on this page describes behavior on data I engineered to overlap and to carry noise. The simulator also generates the outputs that MediaPipe and Whisper would produce, so the pipeline has never processed real video or audio. Out-of-sample validation on real ADOS-2 data would have to come before any conclusion about clinical utility.

I fit the meta-learner on the domain models’ predictions for the same training data the domain models learned from, which can overstate how much the meta-learner trusts each domain; out-of-fold domain probabilities would be the cleaner design, and I would make that change next. Cross-validation covered XGBoost and label spreading only.

The pipeline reduces each session to aggregate counts (total social smiles, average response latency, percent of time with gaze on a face), and aggregates lose temporal sequence. A social smile one second after a clinician’s joint-attention bid is clinically different from a smile at a random moment, and the ADOS-2 is built around the timing of behavioral contingencies. A temporally aware version would extract event sequences, and the architecture would have to change to handle them.

Timeline diagram contrasting aggregate-count representation with event-sequence representation, showing that a social smile occurring one second after a clinician's joint-attention bid is clinically distinct from one occurring at a random time.

FIG. 05 The temporal contingency that aggregate-count features cannot see. Frequencies preserve how often a behavior occurs; they lose whether it occurs in response to a clinician's bid, which is the relationship the ADOS-2 administration is designed to elicit.

The session-quality confound covers plausible failure modes but not every kind of real-world noise (examiner bias, cultural variation in expressive behavior, and comorbid presentations), and the model treats ASD as a binary label where the clinical category is a spectrum. The 22% prevalence matches a specialty referral clinic, so precision and specificity would shift in a general screening population. Within those limits, I built the pipeline as decision support for a clinician, with uncertainty and data quality reported beside every prediction.

References

Chen, T., & Guestrin, C. (2016). XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 785-794). ACM. https://doi.org/10.1145/2939672.2939785
Lord, C., Rutter, M., DiLavore, P. C., Risi, S., Gotham, K., & Bishop, S. L. (2012). Autism Diagnostic Observation Schedule, Second Edition (ADOS-2) manual. Western Psychological Services.
Zhou, D., Bousquet, O., Lal, T. N., Weston, J., & Schölkopf, B. (2004). Learning with local and global consistency. In Advances in Neural Information Processing Systems 16 (pp. 321-328). MIT Press.