The Helper's Language as an Early Indicator

Abstract

When a service conversation starts to fail, the person helping tends to say so first, in apologies and stated limits, while most dissatisfied customers leave without a word of complaint; anyone who waits for the customer to object starts late and misses most of the cases they set out to find. I first built a version of this method for a conversational assistant answering customer requests, and the part that stayed with me was how early the helper’s side of the transcript started apologizing. Here the method runs end to end on 5,000 synthetic claim chats between claimants and claim professionals, with attorney representation within 90 days as the outcome, so every step can be shown on data anyone may inspect. You will see seven dictionary categories in the LIWC tradition, three for the professional and four for the claimant; how a whole-conversation flag picks up the outcome through chat length, and how a rate inside a fixed first-contact window closes that route; a first-two-turn model that predicts representation (AUC .65, calibration slope .96); a weak-supervision label model built from three evidence streams and measured against 300 survey responses, 20 of them negative; and the same pipeline rerun on a second synthetic population, where the proxy falls to chance. The method ends in a pre-registered field experiment, because an indicator earns its keep once it points at something a claim organization can test.

2. A dictionary for both speakers

The counting method comes from the Linguistic Inquiry and Word Count (LIWC) tradition, in which researchers assign words to psychologically meaningful categories and report each category as a percentage of the words in a text (Pennebaker et al., 2015; Tausczik & Pennebaker, 2010). The finding from that tradition most relevant here is that much of the social and psychological information in a text sits in its small, frequent words; pronouns, negations, and other function words vary with attention, self-focus, and stance in ways speakers seldom monitor (Chung & Pennebaker, 2007; Pennebaker et al., 2003). First-person singular is the clearest case. Higher rates of “I,” “me,” and “my” accompany distress and depression, modestly but consistently across many samples (Rude et al., 2004; Tackman et al., 2019), and in conversations between people of unequal standing the lower-status speaker tends to use more of them (Kacewicz et al., 2014), a description that fits a claimant waiting on a decision from the person across the chat.

Seven categories do the work here, three for the claim professional and four for the claimant, and each one is counted only in its own speaker’s turns, with that speaker’s own word count as the denominator. The same word means different things on each side of the exchange; “can’t” from a professional states a limit on what the company will do, while “can’t” from a claimant is negation, often about their own situation (“I can’t keep paying for this myself”). Four categories have close LIWC2015 analogues. Apology, limitation, and correction are specific to service conversations, so they are custom lists, and LIWC users can load lists like these as user dictionaries. The lexicon below is a small illustrative one of my own; for a real deployment it helps to take the validated LIWC-22 categories where they exist (Boyd et al., 2022) and to validate the custom lists against human coding before counting anything at scale.

Category	Speaker	Nearest LIWC2015 category	Example entries	Role in the simulation
Apology	Claim professional	none (custom)	sorry, apologize, apologies, regret	raises representation risk
Limitation	Claim professional	none (custom)	unfortunately, unable, cannot, can’t, not able, not authorized	raises representation risk
Hedging	Claim professional	tentat	maybe, perhaps, possibly, might, probably, I think, I believe	planted null
Negation	Claimant	negate	no, not, never, nothing, nobody, any word ending in n’t	raises risk modestly
Correction	Claimant	none (custom)	already, I said, I told, I meant, actually, twice	raises risk modestly, arrives late
Anger	Claimant	anger	angry, furious, ridiculous, unacceptable, fed up, sick of	raises risk modestly, arrives late
First-person singular	Claimant	i	I, me, my, mine, myself, I’m, I’ve	raises risk modestly

In the last column sits the one thing you can never know in a real dataset, the truth. I generated every chat from fixed sentence banks under a planted process, with a latent friction score for each chat. Higher friction means more professional apologies and limitations, slightly more claimant negation and first-person singular, more correction and anger only for the 30 percent of claimants who voice complaints at all (and more so in later turns), and a higher probability of representation. Hedging I planted at a constant rate per sentence, unrelated to friction or outcome, so hedging is a null category; a method that finds hedging predictive has found an artifact. Because the truth is known, every analytic step can be scored against it, which is the whole reason to build the example synthetically.

ApologyLimitationHedgingNegationCorrectionAngerFirst-person singular

1. Turn 1 · Claimant

   I feel like I'm on my own with this.

2. Turn 2 · Claim professional

   Sorry about the mix-up with the rental. Unfortunately the policy does not cover that item.

3. Turn 3 · Claimant

   A tree from next door fell on the garage. The adjuster came out to look at the roof on Monday.

4. Turn 4 · Claim professional

   I regret that the callback took this long. I think the shop might have it by then. My apologies for the delay on the payment.

5. Turn 5 · Claimant

   I don't know what I'm supposed to do now. Evenings are better for a call. I can't keep paying for this myself.

6. Turn 6 · Claim professional

   Hopefully we can close this out soon. I'm sorry, that request was missed on our end.

7. Turn 7 · Claimant

   That's not what I said earlier. I uploaded the photos of the damage this morning. What do you need from me to move this forward?

4. Prediction from the first contact

At first contact only the first two turns exist, so the features are the seven category rates in turns 1 and 2, the claimant’s opening message (14.7 words on average) and the professional’s first reply (17.9 words). The model is a logistic regression on standardized rates with a light L2 penalty, scored by five-fold stratified cross-validation so every chat is scored by a model fit without it, with a 95% interval for the AUC from 2,000 bootstrap resamples of those out-of-fold predictions (Efron & Tibshirani, 1993; Hanley & McNeil, 1982). Resampling the evaluation set alone leaves out the smaller variation from refitting the model, which is worth knowing when you read the interval.

The cross-validated AUC was .65 (95% CI .62 to .67), a modest level of discrimination, and in my experience the level a first-message model tends to reach on real service conversations as well.

Discrimination is how well the model ranks claims; calibration is whether its probabilities mean what they say, and a measure read as rates by team or by week is only as good as its calibration (Van Calster et al., 2019). The model is fit without class reweighting, since with a reweighted minority class the AUC stays the same and every predicted probability rises. Mean predicted risk was 11.6 percent against an observed rate of 11.6 percent, the calibration slope was .96 (1.00 is ideal), and the Brier score was .098 against .102 for predicting the base rate for everyone. The top decile of predicted risk had an observed representation rate of 28.8 percent and the bottom decile 9.2 percent.

The coefficients recover the planted structure, with one instructive exception.

First-two-turn category	Odds ratio per SD	95% bootstrap CI
Apology (professional)	1.50	1.40 to 1.61
Limitation (professional)	1.33	1.23 to 1.42
Hedging (professional)	1.11	1.01 to 1.21
Negation (claimant)	1.13	1.03 to 1.21
Correction (claimant)	0.98	0.86 to 1.05
Anger (claimant)	1.06	0.97 to 1.12
First-person singular (claimant)	1.04	0.95 to 1.14

Apology and limitation have the largest odds ratios. Negation has the modest effect I planted; correction and anger sit near 1.0 because they seldom appear in the first claimant turn; and first-person singular, planted at the same strength as negation, loses its signal in an opening message of about fifteen words, a reminder that dictionary rates on very short texts are noisy. Hedging, the planted null, comes out at 1.11 with an interval that just clears 1.0, and the cause is a feature of the generator. Each sentence has at most one category, so a hedge occupies a slot an apology could have filled; once apology is held constant in the regression, the hedging coefficient rises above one, a suppression effect created by the way the text was built. The same kind of artifact appears in real text whenever categories compete for the same words or slots, and the univariate rate (r = .01) is the safer reading of hedging here.

6. Measuring when ground truth is thin

Representation is a hard outcome, and claim professionals record it in the file. The claimant’s experience is softer and matters on its own terms, and here the only explicit evidence is a post-close survey that few people answer. In the simulation, 300 survey responses were drawn at random, and 20 of them were negative, about the size of anchor a team usually has when it first asks whether its implicit measures mean anything.

Three evidence streams combine into one label. The first is helper language (the professional’s apology and limitation rate over the whole chat, flagged for the top fifth); the second is claimant language (negation, correction, and anger, flagged the same way); the third is behavioral (a repeat contact within seven days). Rates per word keep the length trap out of the two language streams, and reading the whole chat is appropriate here, since the question concerns how the conversation went once it was over. A two-class latent class model fit by expectation maximization (Dawid & Skene, 1979), the model family behind data programming and weak supervision (Ratner et al., 2016, 2017), estimates each stream’s sensitivity and false-positive rate without any labels and yields a posterior probability of a dissatisfied experience for every chat. A simulated LLM-judge label stays out of the model and serves as an independent cross-check, the way a judge pass is often used to audit a proxy (Zheng et al., 2023).

Against the 300-response anchor, the label model’s AUC is .64 (95% CI .51 to .77). In the simulation you can do something no real program allows and score the same label against every chat’s true state; that AUC was .76. Drawing 1,000 alternative surveys of 300 from the same population, the middle 90 percent of their AUC estimates ran from .61 to .82. The anchor that happened to be drawn sits low in that spread, and with any single anchor of this size the answer would stay just as open; by the Hanley and McNeil (1982) approximation the AUC on the current anchor is known only to within about ±.13, and narrowing that to ±.05 at this prevalence takes about 2,190 responses, roughly 146 of them negative.

Fitting the label model means assuming the streams are independent once the true state is known, and dropping each stream in turn is a practical check on that assumption. Without helper language, the true AUC fell from .76 to .71; without repeat contact it fell to .70; without claimant language it rose to .78. Both language streams rise with the same friction, and among the satisfied survey respondents their flags still correlated (phi = .19), so the shared friction enters the posterior twice and gets too much weight. None of this was visible on the anchor, since its estimates for the four versions (.64, .63, .66, .62) differ by far less than its own interval. The judge’s labels and the proxy agreed only weakly, at a Cohen’s (1960) kappa of .16, although the judge’s own AUC against the true state was .71; two imperfect readers of the same construct can disagree substantially while each contains real signal.

8. From indicator to experiment

From an association between apology language and representation you learn that apologies mark breakdowns, and nothing yet about whether a different way of handling the breakdown would change the outcome. A field experiment answers that question, and the indicator is how you choose what to test. Experimental work on apologies in legal settlement suggests that an apology that accepts responsibility and one that only expresses sympathy may move claimants differently (Robbennolt, 2003), and the limitation category points at a second target, the moment a professional tells a claimant what the company will not do. A pre-registered test, filed before any claim enters it (Nosek et al., 2018), would carry these elements.

• Intervention. Coaching and a short script for claim professionals on delivering a limitation, pairing each stated limit with its reason, the next step, and the date of that step, and closing with a check that the claimant understood.
• Unit of randomization. Teams of claim professionals, since coaching is delivered to people and spreads within a team; claims inherit the arm of the team that handles them.
• Primary KPI. Attorney representation within 90 days of first notice of loss, analyzed by intention to treat with a mixed-effects logistic regression and a random intercept for team.
• Guardrail. The helper apology rate per 100 words in turns 1 and 2, held within a two-sided band around its baseline. A sharp rise would mean the script turned into rote apology, the Population B failure that erased the signal; a sharp fall would mean professionals learned to avoid the words. Once the coaching is about the language, the language can no longer serve as the measure of success (Campbell, 1979), so it stays a guardrail and representation stays the outcome. Cycle time and the post-close survey score serve as secondary guardrails.
• Exclusions and stopping. Claims already represented at first contact are excluded in advance, and the analysis runs once, at a fixed horizon 90 days after the last claim enters.

The sample size follows from the outcome. Detecting a drop from 11.6 to 10.15 percent (a 12.5 percent relative reduction) at α = .05 with 80 percent power requires 7,236 claims per arm if claims were independent. Claims are clustered within teams, so the requirement grows by the design effect, one plus the intraclass correlation times the cluster size minus one.

Claims per team	Assumed ICC	Design effect	Claims per arm	Teams per arm
200	.01	2.99	21,636	109
200	.02	4.98	36,036	181
400	.01	4.99	36,108	91

The ICC values are assumptions, and it pays to estimate them from historical claims before committing to a design, since the required sample at an ICC of .02 is about two thirds larger than the sample at .01.

References