Counting a Behavior Is Not the Same as Characterizing It

Two burdens.

When I characterized the Long Conversation Reminder behavior in Claude Sonnet 4.5, the finding came from two numbers that do different jobs, 26,158 corpus observations and 35 hand-coded cases, and neither number could have stood on its own; the corpus established that the behavior occurs at a rate that can be bounded, and the cases established what the behavior is when it occurs, and a claim about how an artificial system behaves tends to carry both of those burdens at once even though the same evidence seldom discharges them together. The first burden is existence at scale, where the behavior occurs across many interactions, at a rate that can be bounded, and not in the one transcript alone that a critic happened to screenshot; the second burden is identity, where the behavior has a definite structure, an anatomy that separates it from the things it is mistaken for. Corpus-level evidence discharges the first burden and is nearly silent on the second; case-level evidence discharges the second and is nearly silent on the first; a study of a deployed model tends to need both tiers, because a finding that rests on one of them alone is either a number with no referent or a story with no warrant for generalizing.

The corpus tier counts; it aggregates observations over a population, reports how often a marker appears, and supports statements about prevalence. The case tier reads; it examines individual instances closely enough to say what the behavior is when it occurs, what its internal structure looks like, and how it differs from the adjacent behaviors it resembles on the surface. The argument I want to lay out is that a behavioral claim about a model earns its validity from the join of the two tiers, and that keeping the tiers distinct, in the analysis and in any later attempt to replicate it, is what keeps the claim honest about its own reach.

2. Anecdote without prevalence.

A single case, read closely, does the work the corpus cannot. It shows that the behavior is possible, and it exposes the internal structure of one instance, naming who initiated, what the model asserted, on what evidence, and how it responded to correction. The phenomenological reading is where a construct is born, because the categories that a corpus later counts have to come from somewhere, and they come from looking hard at instances before there is anything to tabulate.

What a case cannot do is carry a rate. One transcript supports no statement about whether it is representative or anomalous, whether it reflects a property of the system or an accident of a single generation, whether it would recur on a fresh prompt or never appear again. A screenshot can be edited, a prompt can be omitted, a one-off sampling artifact can be mistaken for a disposition. The idiographic depth that makes a case informative about structure is exactly the depth that makes it mute about frequency; the distinction between idiographic and nomothetic knowledge is old in psychology (Allport, 1937) and it is not dissolved by the volume of data a model interaction produces.

The error that case-only evidence invites is the inference from vividness to prevalence. A behavior that reads as alarming in one transcript feels common precisely because it is memorable, and the feeling is not a measurement. Case-study methodology has spent decades on the conditions under which a small number of instances can and cannot license a general claim (Yin, 2018), and the consistent answer is that depth establishes mechanism while breadth establishes reach. A case tells you what the behavior is. It does not tell you how much of it there is.

4. The Long Conversation Reminder in two tiers.

The Long Conversation Reminder analysis is the worked instance. In late September 2025 a system-prompt scaffold was appended to Claude Sonnet 4.5, and the model began issuing unsolicited psychiatric attributions during ordinary sessions, labeling coding work and creative writing as evidence of mania, perfectionism, or detachment from reality. The question was whether that observation, which arrived as a handful of striking screenshots, named a real property of the deployed system or a scatter of memorable one-offs.

The corpus tier answered the extension question. I built a pipeline over 26,158 Reddit observations spanning August through December 2025 across three subreddits, with the date window opening two months before the deployment so the pre-deployment baseline was measured rather than assumed. The four-class voice segmentation isolated model output from user narration, the HDBSCAN sense-discovery step separated the clinical sense of “professional” from the formatting sense, and the cleaned corpus showed the behavior present across the public record at a rate that could be bounded rather than imagined.

The case tier answered the identity question. I hand-coded 35 clinical role-violations against a five-property definition drawn from the boundary-violation literature (Gutheil & Gabbard, 1993). The directive is unsolicited, it rests on a weak inferential signal rather than any real assessment, it does not yield to pushback, it moves toward restriction rather than toward the user’s options, and it persists across fresh sessions. The structural findings were uniform. All 35 were unsolicited, all were restrictive, and in the five cases where a user pushed back the model escalated in all five and yielded in none.

Neither tier is the finding by itself. Thirty-five cases without the corpus is a small pile of anecdotes that generalizes to nothing, and a raw count of keyword hits without the cases is a contaminated number with no anatomy. The finding lives in the conjunction, a behavior that is uncommon in raw rate, thirty-five out of more than twenty-six thousand, yet invariant in structure whenever it fires. A behavior that occurs seldom but with no variance in direction or in response to correction reads as a deterministic trigger in the system prompt rather than a stochastic generation artifact, and that reading is available only because one tier supplied the rate and the other supplied the structure. The corpus method that surfaced the population is described in Reading Misalignment Off the Public Record, and the full corpus and case report is in Pathologizing Without Warrant.

What the structure commits to.

The two-tier structure commits to three things, an existence-and-rate claim that comes from the corpus and is stated as a bounded frequency, a structure-and-reliability claim that comes from the cases and is stated as a coded signature, and a general claim about the system only where both hold. It does not commit to a population rate inferred from a purposive case sample, to a behavioral identity inferred from a frequency count, or to a single replication standard that covers both tiers at once. A purposive sample of cases estimates no rate, a count of surface markers defines no construct, and a reviewer who asks one tier to do the other’s job will mistake a partial finding for a refuted one.

The discipline it imposes is a modest one that does real work; a behavioral claim about a deployed model reports the rate and the structure separately, because the evidence for each is separate, and a reader can reject either one without rejecting both. Count a behavior and you have learned how often it happens; characterize it and you have learned what it is, and the two are not the same act.

Related analyses

The methodological argument here sits beneath a small cluster of analyses of the same model behavior. The corpus and the hand-coded cases are reported in Pathologizing Without Warrant. The general method that surfaced the corpus is Reading Misalignment Off the Public Record. The parallel behavior, unsolicited sleep directives issued without regard to the user’s time of day, is measured under the same two-tier discipline in The Bedtime Directive.

References