A walkthrough of the discourse-mining pipeline that surfaced the Claude Sonnet 4.5 complaint cluster, through attributed-quote extraction, BERTopic clustering over sentence embeddings, a NetworkX co-occurrence graph, and a Prophet changepoint test against the release date. Each step is named with the question it answers and the inference the public record cannot support, on a corpus of public Reddit discussion rather than a controlled benchmark.
When a deployed model develops a behavior its users dislike, the users tend to document it in public forums, often quoting the model word for word, before a formal benchmark registers the behavior at all. The pipeline described here treats that public discussion as a measurement instrument and reads it in four steps, attribution of model-voice spans before any sentiment scoring, BERTopic clustering that lets the complaint themes settle out, a NetworkX co-occurrence graph linking model outputs to reported outcomes, and a Prophet changepoint test that asks whether complaint volume shifts at a release date, with a RandomForest and SHAP pass to confirm which lexical features carry the separation. The worked example throughout is the cluster of complaints that followed Claude Sonnet 4.5’s September 29, 2025 release, in which users posted model outputs offering unsolicited clinical-sounding assessments. I offer the pipeline as a methodology rather than a result, with each step named alongside the question it answers and the inference the public record cannot support; the counts below describe structure inside a targeted retrieval frame rather than prevalence across any user population.
The corpus for the worked example comes from the Arctic Shift academic Reddit archive, retrieved across four Claude-related subreddits (r/ClaudeAI, r/Anthropic, r/ClaudeCode, and r/claudexplorers) over the August 1 to December 31, 2025 window, which brackets the September 29 release of Claude Sonnet 4.5 with two months of pre-release discussion on the near side. A wholesale pull of 31,078 posts, filtered for moderator-removed stubs, left 22,008 intact posts; seed-term matching on the complaint vocabulary tagged 1,173 of them, and a comment refetch against those identifiers returned 24,985 comments, for a canonical corpus of 26,158 rows.
The frame is the first thing I want to be honest about. The corpus is a targeted retrieval, the set of seed-matching posts on four subreddits plus their comments, and it is not a random sample of Reddit or of the Claude user base, so every count reported below describes structure inside that frame and none of it estimates how often a behavior occurred platform-wide. The running example is a cluster of posts in which users quoted the model offering unsolicited clinical-sounding assessments, calling an extended session “manic,” recommending professional help during a coding task, and describing “escalating detachment from reality” in reply to ordinary prompts; the behavior is characterized in its own right in a companion analysis, and what I am after here is the method that surfaces it rather than the behavior itself.
A forum post stays testimony until the model’s own words can be separated from the user’s account of them, so the pipeline runs attribution before any sentiment scoring. The attribution step scans each post for a model utterance that is recoverable as a span, through quotation marks, a markdown blockquote, or an explicit attribution phrase (“Claude said,” “it replied with”), and keeps the cases where the recovered span carries a signature of the behavior under study. Two seed lexicons anchor that signature, a model-output lexicon (professional, therapy, clinical, manic, diagnose, escalating, spiral) and a user-experience lexicon (gaslighting, condescending, patronizing, unsolicited, intrusive). Each lexicon is widened per corpus segment by ranking candidate terms on TF-IDF weight and cosine similarity to the seed set under all-MiniLM-L6-v2 embeddings, which recovers contextual neighbors a hand-written list would miss while holding close enough to the seeds to stay on topic.
Attribution comes first because a measured claim about a model’s output depends on the output being present in the record, recovered as a span, rather than summarized by an upset user. A global sentiment score over a whole thread, a VADER compound for instance, reports that the thread is negative and throws away the boundary between the model’s voice and the user’s reaction in the same pass; segmenting the discourse into those two voices preserves the association the analysis is after, the specific distress that travels alongside the model’s clinical language. The limitation here is structural and sits right at this layer, since only behaviors a user chose to quote can be attributed, so a model failure that produces frustration without a quotable artifact leaves the attribution step with nothing to recover.
Fixing the complaint categories in advance tilts the result toward the categories already expected, so the pipeline clusters the attributed discourse with BERTopic and lets the themes settle out of the text. The embedding model is all-MiniLM-L6-v2 at 384 dimensions, reduced with UMAP and clustered with HDBSCAN, with the topic count left on automatic so the algorithm sets it. The separation this step produces is the reason to run it, since a single broad complaint about harmful helpfulness resolves into distinct phenomena under clustering, with unsolicited clinical framing forming one topic, patronizing tone a second, and refusal to continue a task a third, each countable, trackable over time, and testable on its own for co-occurrence with reported harm.
The payoff turns up in the worked example, where the topic grouping “manic,” “therapy,” and “professional help” co-occurred with distress terms at a higher rate than the topic grouping “annoying,” “verbose,” and “refuses to code,” a contrast a preset category scheme would have erased by filing both complaints under negative user experience. Two limits travel with the technique. The clusters are sensitive to the embedding model and to the UMAP and HDBSCAN settings, so a single run is one view of the structure, and it helps to check stability across configurations before placing any weight on a cluster; small clusters in particular are noisy enough that reading post by post tends to pay off. The algorithm groups the text; the judgment about whether a group names a coherent behavior or an artifact of the embedding stays with the analyst.
A behavior becomes a safety question once it connects to an outcome a user lived through, so the pipeline builds a weighted co-occurrence graph in NetworkX, with nodes for model-output terms and user-reported outcome terms and edges weighted by how often a pair shows up in the same thread. In the worked example the clinical-framing terms (diagnose, clinical, professional, therapy) connect to outcome terms (gaslit, frustrated, condescending), and the edge between diagnose and gaslit carries several times the weight of the next-heaviest pair, which is the countable form of the qualitative reading that users who received diagnostic language were disproportionately the ones who reported feeling gaslit.
The graph earns its place by surfacing co-occurrence that a raw frequency table hides, since two terms can each be common across the corpus and still seldom appear together, so a pairing that does appear carries more signal than either term’s marginal count, the intuition that pointwise mutual information formalizes. The step fits the situation where the analysis holds two vocabularies, what the system did and what users reported, and the question is which pairings travel together. The limit is the standing one for any co-occurrence measure, where a shared thread is association rather than causation, the thread is a coarse unit, and an edge records that two terms appear together without recording which direction the influence runs, so the graph is best read as a set of leads to carry back against the posts.
A steady correlation between an output and a complaint is suggestive; complaint volume that rises at the moment a model ships is more telling, and that is the question a changepoint test against a known date is built to ask. The pipeline fits a Prophet model to daily complaint volume with weekly seasonality enabled and the September 29, 2025 release encoded as a prior changepoint, and in this exploratory run the harmful-helpfulness complaints rise around that date rather than holding to a flat baseline. The result reads as a flag rather than a verdict, since a release-timed rise is consistent with the release having introduced the behavior, and it is equally consistent with a user base that grew, an attention cascade once a single post named the pattern, and the coverage quirks of whichever days the archive happened to preserve well; the disciplined response is to treat the date as worth scrutiny and go back to the posts.
A RandomForest classifier over the labeled posts plays an interpretive role beneath the time series, learning which lexical features separate a harmful-helpfulness complaint from an ordinary one. The corpus is a scarce-positive setting, since most posts are routine complaints and only a minority are clinical-framing reports, so a classifier trained on it is too brittle to deploy; the SHAP attribution layer on top is what the classifier is for, reporting which features carry the separation and confirming that the clinical-output terms, rather than generic negativity, do the discriminating. The pattern worth carrying out of this step is a narrow one, since a weak classifier read through SHAP answers why two groups differ and is the wrong instrument for predicting membership of a third.
The four steps compose into a procedure that is not specific to one model or one behavior, since attribution of model-voice spans, emergent topic structure, a co-occurrence graph from outputs to reported outcomes, and a changepoint test against a release date apply to any deployed system whose users quote it and react in public, which makes a public forum a standing, low-cost evaluation channel that runs between formal evaluations on discussion the users already produce. The procedure scales down to a triage on a hundred posts and up to a full corpus without changing the order of operations.
The method does not commit to a prevalence estimate, a causal claim, or a finding that transfers outside the retrieval frame; the release-timed rise is a lead, the co-occurrence edges are associations, and the topic rates describe the seed-matched corpus rather than the user population. It carries one limitation by construction, which is that it sees only the behaviors users notice and report and attributes only the outputs users quote word for word, so that a quotable directive surfaces cleanly, the way an unsolicited “get some sleep, for real this time” does, while a subtler degradation, a model that drops context without comment after a long session, produces user frustration and no quotable span for the attribution step to recover, and a behavior that lands on users who never post does not enter the corpus at all. What the pipeline offers is a way of reading the public record that keeps its own uncertainty in view, a complement to offline evaluation that points at the episodes worth a closer and more careful look.
The two behaviors this method surfaced in the worked example are characterized in their own right elsewhere. The pathologizing behavior, where the model attributes distress or mania to a user’s ordinary input without clinical warrant, is measured in Pathologizing Without Warrant. The sleep-nudging behavior, where the model issues unsolicited directives to stop working and rest regardless of the hour, is measured in The Bedtime Directive. The argument that both express one caretaker disposition is When Anthropic’s Claude Takes the Wheel.