The Kaplan-Meier estimator asks "what is the probability that a patient is still alive at time t?" The same tool, with the same right-censoring assumptions, asks "what is the probability that a user session is still violation-free at time t?" A Spark-scale analytics engine that imports survival analysis from biostatistics into Trust & Safety, layered with LDA topic modeling, VADER sentiment, and NetworkX co-occurrence, producing a 12-section auto-generated investigation report that is reproducible across cases and comparable across users.
I kept staring at T&S session data that had the same shape as the censored patient cohorts from my biostatistics training, and at some point I stopped pretending the analogy was loose. In medical research, the Kaplan-Meier estimator answers a specific question: given a cohort of patients diagnosed at time zero, what is the probability that any given patient is still alive at time t? The estimator handles the fact that not every patient’s outcome is observed; some are still alive when the study ends, some leave the study, some die from unrelated causes. These are right-censored observations, and KM is the non-parametric tool designed to use them rather than throw them out.
Trust & Safety analytics has a structurally identical problem. Given a user’s session beginning at time zero, what is the probability that the session has remained free of policy-violative content at time t? The events of interest (encountering content flagged by a moderation system) occur on the same kind of irregular timeline as patient deaths. The censoring is the same: many sessions end without any violation event, which is not the same as those sessions being violation-free forever. The data structure is the same, and the estimator transfers without modification.
Once you accept the reframe, an entire toolkit comes with it: median time to violation as a population-level health metric, comparative survival functions across user segments, hazard rate analysis for identifying high-risk windows. The biostatistics literature has been refining these tools since 1958, and most of what I had been building in T&S amounted to cruder approximations of the same estimators.
The engine runs on a YARN-managed Spark cluster sized for petabyte-scale moderation telemetry: roughly 3,000 CPU cores allocated at 2 GB per core for 7.8 TB aggregate execution memory, with shuffle partitions clamped against the cluster’s physical limits to stop data skew when complex cross-metric aggregations hit the executors. The Spark layer makes the analytical layer tractable; the part I care about here is the decomposition.
The pipeline splits cleanly into four modules, each with a single responsibility, and the modules communicate through standardized intermediate CSV artifacts rather than in-memory state. Any module can be re-run in isolation when a methodology question demands it, which is the property that makes the pipeline auditable: you can re-run the survival analysis on the same intermediates without re-running the Spark fetch, and if the numbers change, the problem is in the analysis, not in the data extraction.
FIG. 01 The four-module decomposition. Modules communicate through standardized CSV intermediates instead of in-memory state, so any analytical module can be re-run in isolation without re-running the Spark fetch.
The core analytical move is small in code. For each user-segment, the engine computes the temporal difference between consecutive content events as duration in seconds. The event indicator is one if the content interaction was associated with a flagged policy violation and zero otherwise. The pair (duration, event_observed) is what lifelines.KaplanMeierFitter wants. Fit the estimator on the segment’s data, plot the survival function, read off the median time to violation.
# Per-user-segment survival fit
df[time_column] = pd.to_datetime(df[time_column])
df.sort_values(by=time_column, inplace=True)
df['duration'] = df[time_column].diff().dt.total_seconds()
df = df.dropna(subset=['duration'])
df['event_observed'] = df[event_column].apply(
lambda x: 1 if not pd.isna(x) and x.strip() != '' else 0
)
kmf = KaplanMeierFitter()
kmf.fit(df['duration'], event_observed=df['event_observed'], label=segment)
kmf.plot_survival_function()
median_survival = kmf.median_survival_time_ # seconds to first violationFIG. 02 The survival fit. Duration is the inter-event spacing in seconds; event_observed is one when a policy was triggered. The median survival time is the population-level health metric.
Where this gets interesting for per-user investigation is the comparative mode. Segment one user’s data into temporal tertiles (early, mid, late), fit separate Kaplan-Meier curves on each, and you can see whether that user’s median time-to-violation is shrinking, stable, or growing across the observation window. A shrinking median means the account is accelerating toward violative content. The same comparison across user cohorts surfaces population-level shifts in platform health.
A survival time below 3,600 seconds (one hour) means a user encountered violative content within the first hour of platform engagement in that segment. The engine surfaces that threshold automatically in its narrative output; in practice, that one-hour window is where I saw moderation feedback loops compound most often, because the recommendation engine has already begun shaping the next session before the first one ends.
Survival analysis answers when violations happen. It does not answer what topical neighborhood the user was in when the violation occurred, or how that neighborhood evolved. The engine’s text NLP module fills that gap with three layered passes: latent Dirichlet allocation for topic clusters, VADER and TextBlob for sentiment polarity over time, and NetworkX-backed co-occurrence graphs for word-pair structure inside a five-token window.
None of these signals is conclusive on its own, which is why the engine runs all three on every slice instead of letting the analyst pick. A drop in the survival curve that coincides with a sentiment polarity shift and a topic-cluster turnover is a different finding than a survival drop while the topical landscape is stable. The first pattern suggests the user moved into new content territory; the second suggests the existing territory became more violative. Different remediations follow from each, and I have seen analysts reach the wrong conclusion when they had only one axis to look at.
Co-occurrence analysis catches what topic modeling smooths over. LDA gives you topic distributions; the co-occurrence graph gives you the word-pairs that shape the texture inside those topics. When two words that did not previously co-occur start appearing together at high frequency in a user’s consumption window, the engine flags the pair regardless of whether either word is independently policy-relevant. The engine sidesteps LDA’s stochastic renumbering problem (the same data can produce different topic indices on different runs) by labeling topics by their top-five terms instead of by index, which makes the topical drift inference robust to renumbering even though downstream automation that hard-codes topic IDs would not be.
FIG. 03 Topic drift across time slices. LDA topic clusters are tracked from early to late tertile; emergent clusters in red signal new topical territory. When this chart shifts at the same temporal boundary as the survival curve, the analyst has both the timing of the change and the content that changed.
The fourth module is a Jinja-templated HTML generator that takes the analytical outputs of the prior three and produces a 12-section report with a fixed structure. When every investigation produces the same sections in the same order with the same statistical choices behind each section, two investigations of two different cases become directly comparable in a way that ad-hoc analysis does not permit. Reviewers can read the seventh section of any report with a known set of expectations.
| SECTION | CONTENT |
|---|---|
| 01 | Executive summary |
| 02 | Methodology |
| 03 | N-grams and co-occurrence analysis |
| 04 | Themes and clusters (LDA topic modeling) |
| 05 | Caption and hashtag analysis |
| 06 | Visibility and engagement metrics |
| 07 | Sentiment analysis (VADER) |
| 08 | Search term effectiveness |
| 09 | Cross-metric correlation analysis |
| 10 | Search term to policy violation mapping |
| 11 | User engagement analysis |
| 12 | Conclusion and recommendations |
FIG. 04 The fixed 12-section investigation report. The fixed structure means two analysts investigating two different accounts can compare section 7 directly without wondering whether the underlying statistical choices were different.
The engine generates the survival functions, heatmaps, and topic clusters as image artifacts written to a per-investigation directory. The Jinja template assembles them into the final HTML. A separate FPDF rollup converts the HTML to a print-ready PDF. The whole sequence runs unattended after the analyst points the orchestrator at a new dataset and confirms the column mappings.
Survival analysis assumes independence between consecutive events, which is debatable in any system where the recommendation engine shapes the next interaction. The Kaplan-Meier estimator is robust to this in practice, but the median survival time should be read as a population-level summary, not a per-session prediction. Right-censoring assumptions apply equally: a session that ends without a violation is censored, not violation-free in perpetuity.
Topic modeling produces interpretable clusters at the cost of run-to-run reproducibility; the top-five-term labeling sidesteps the renumbering problem for human readers, but any automation that consumes topic indices directly would need to handle the instability.