Cyber Threat Intelligence
From OSINT Collection to Actionable Cyber Threat Intelligence
An eight-stage pipeline from public sources to SOC detection — and the specific way each stage fails. Most pipelines do not break where they are monitored; they break at the handoffs, silently, in the direction of false confidence.
Most descriptions of an intelligence pipeline are architecture diagrams: boxes, arrows, a happy path. They are accurate and almost useless, because a pipeline’s behaviour is determined by its failure modes, not its stages — and the failure modes live in the handoffs between boxes, where nothing is monitored.
This is the pipeline I work from, stated as eight stages, with the failure mode of each stated alongside it. The through-line is a single property: every stage must preserve the ability of a later stage to reason about the evidence. A stage that produces correct output while destroying provenance has damaged the pipeline, and it will pass every test that only inspects its output.
Stage 1 · Public Sources
What it is. The universe of open material: vendor and CERT reporting, malware repositories and sandbox output, certificate transparency logs, passive DNS, WHOIS and registration data, paste sites and code hosting, leak sites and extortion portals, forums and messaging channels, government advisories, academic publication, and news.
What it is not. A uniform population. Every source carries a bias that propagates downstream unless it is recorded at the point of collection.
Failure mode — the source population is never characterised. Teams enumerate sources and never ask what each systematically omits. Vendor reporting over-represents adversaries who attacked vendor customers. Sandbox corpora over-represent malware that someone chose to submit. Leak sites show only victims who did not pay quietly. Any frequency claim derived from these — “this technique is rising” — is a claim about the collection, not about the world, and the two diverge precisely where it matters.
The discipline. For each source, record in writing: what it covers, what it structurally cannot cover, its update cadence, its licence and terms, and its known biases. This document is the only thing that lets a later analyst say “we did not see X because we do not look there” instead of “X did not happen.”
Stage 2 · Collection
What it is. Scheduled, rate-disciplined acquisition of raw material, with an immutable record of what was retrieved, from where, and when.
Failure mode — provenance is attached late. Collection stores content and reconstructs its origin afterwards. Open sources are mutable and disappear; a page rewritten or deleted six months later leaves an assessment that can never be re-examined. Provenance reconstructed later is provenance invented later.
Failure mode — collection failure is silent. A source that stops returning results looks identical, downstream, to a source with nothing new to report. Unless collection emits a positive “ran, retrieved N, N may legitimately be zero” record, the pipeline cannot distinguish a dead feed from a quiet week — and a dead feed will be read as good news for as long as it takes someone to notice by accident.
The discipline. Every collected artefact carries source identifier, canonical URL, retrieval timestamp, content hash, and a retained raw copy. Every collection run emits a record whether or not it produced content. Rate limits and terms of service are enforced in code, not in policy documents.
Stage 3 · Validation
What it is. Assessing whether collected material is what it appears to be, and grading it, before it is allowed to influence anything.
This stage does two separable jobs that are routinely merged:
- Technical validity — is this a well-formed indicator? Is the domain resolvable? Is the hash a real artefact? Is the IP in a range that makes the claim plausible?
- Evidential grading — how reliable is the source, and how credible is this specific content? These are two axes, not one. The Admiralty Code grades them separately precisely because a reliable source can report something improbable and an unreliable source can be right.
Failure mode — circular reporting counted as corroboration. One original claim, republished across five aggregators, arrives as five apparently independent sources agreeing. Confidence rises with no new evidence in existence. This is the most common way open-source confidence inflates, and it is entirely mechanical to prevent: record, for every claim, the earliest origin reachable, and give downstream copies zero additional weight.
Failure mode — validation strips what it validates. A record arrives with source, first-seen timestamp and reliability grade, and leaves as a bare string in a list. It is now unassessable: it cannot be aged out, disputed, weighted, or attributed. The pipeline did not lose data; it lost the capacity to reason about data. Everything downstream will run normally and produce output that cannot be defended.
Stage 4 · Entity Resolution
What it is. Deciding when two records refer to the same real-world thing — the same infrastructure, the same tool, the same actor, the same campaign.
This is where a collection of observations becomes a structured picture, and it is the stage with the worst ratio of consequence to attention.
Failure mode — over-merging. Two distinct actors sharing a bulletproof hosting provider get merged into one. Every subsequent assessment about the merged entity is now an assessment about a thing that does not exist. Over-merges are self-reinforcing: the merged entity has more evidence, appears more significant, and attracts more collection.
Failure mode — under-merging. The same actor tracked as three entities means each looks minor and none triggers escalation. Splitting is quieter than merging and therefore survives longer.
Failure mode — the merge decision is not recorded. The single most damaging variant. Once two records are merged, if the basis for the merge is not stored, the decision can never be reviewed, and the resulting entity’s history is unreconstructable. Merges must be reversible, which means the pre-merge records survive and the justification is attached to the link, not applied to the data.
The discipline. Merge on evidence that is expensive for an adversary to fabricate. Shared hosting is not evidence. Shared custom code lineage is. Record the criterion used, and treat every merge as a hypothesis carrying the confidence of its weakest link.
Stage 5 · Attribution
What it is. Associating activity with an actor. Three distinct questions get this name, in increasing difficulty:
- Clustering — are these events the same activity? Answerable from technical evidence.
- Naming — is this the group already tracked as X? Inherits every assumption in someone else’s cluster.
- Responsibility — which organisation or state directed this? Generally not answerable from open sources.
Failure mode — the transition between the three is silent. A report presents clustering evidence and concludes at the responsibility level, with the escalation happening inside a single sentence. The reader receives a level-3 claim supported by level-1 evidence and has no way to see the substitution.
Failure mode — evidence weighted by availability rather than by cost to forge. Language strings, timezone stamps, compile timestamps and registrant names are nearly free for an adversary to choose, and all four have been deliberately planted in real operations. Sustained operational tempo aligned to a working calendar across two years is not free. Weight accordingly.
Failure mode — two findings from one source counted as two. If infrastructure analysis and malware analysis both trace to the same sandbox report, they are one line of evidence displayed twice. Independence must be established at origin, not at analysis.
The discipline. State which of the three claims is being made, in the assessment sentence itself. Enumerate competing hypotheses before assembling evidence, including deliberate misdirection as a standing candidate. Score evidence by what it rules out, not by what it is consistent with.
Stage 6 · Confidence Assessment
What it is. Attaching a stated, standardised confidence to each judgement — and recording what would overturn it.
Failure mode — invented numerics. A “confidence score” of 0.87 produced by an unstated method is decoration. It conveys precision the underlying reasoning does not have, and because it is a number, it survives summarisation better than the caveats attached to it.
Failure mode — collapsing two axes into one. Source reliability and content credibility answer different questions and drive different actions. Merging them destroys exactly the information that tells an analyst whether to seek corroboration.
Failure mode — no third state. Tooling supports confirmed and not confirmed, which is one state short. There are three:
| State | Meaning |
|---|---|
| Assessed true | We looked, and formed a judgement, with confidence c |
| Assessed false | We looked, and the evidence is against it, with confidence c |
| Unable to assess | We did not look, could not look, or the evidence does not bear on it |
“Unable to assess” is not low confidence. Low confidence is a judgement; this is the absence of one. A schema with no field for it will store it as a negative finding, and “we did not look” and “we looked and found nothing” are opposite claims. Every pipeline that conflates them is generating false negatives at a rate it cannot measure — because the records that would reveal the rate are the ones being converted.
The discipline. Use published bands — Admiralty Code for source and content, ICD 203 for estimative language. Require a falsifier on every assessment: the observation that would overturn it. An assessment that cannot name its own refutation was not reasoned to.
Stage 7 · Cyber Threat Intelligence
What it is. The product. A statement addressed to a named consumer, answering a declared requirement, carrying evidence, confidence, and a stated implication.
Failure mode — the product has no consumer. Produced because the pipeline produces, not because anyone asked. Without a Priority Intelligence Requirement behind it, relevance has no criterion, and the function’s only defensible metric becomes volume.
Failure mode — wrong altitude for the reader. Strategic, operational, tactical and technical intelligence have different consumers, lifespans and formats. A CISO handed a packet capture and a SOC analyst handed a geopolitical essay have both received something correct and useless.
Failure mode — no expiry. Technical indicators decay in hours to days; operational assessments in weeks to months. Intelligence without a stated lifespan accumulates into blocklists that generate false positives long after the threat moved, and into strategic assessments cited as current years after the ground shifted.
The discipline. Six questions, answered in the product: who asked · what we assess · on what basis · how confident and why · what would change this · what decision this enables.
Stage 8 · SOC Detection
What it is. The conversion of intelligence into operating controls — detection rules, hunt hypotheses, blocking decisions, alert enrichment. The stage where the whole pipeline either earns its cost or does not.
Failure mode — the loop does not close. Very few functions can say which detections trace to which intelligence product, or which products changed a decision. Without that link the pipeline cannot be tuned, because it has no signal about which of its stages produced value.
Failure mode — pushing the cheap end of the pyramid. Hashes, IPs and domains are the easiest to automate and the cheapest for an adversary to discard. A pipeline optimised for indicator volume will deliver the tier that costs the adversary least, at scale, and the SOC will correctly observe that more intelligence produced no more detection.
Failure mode — the detection is never proven to fire. A rule deployed and never observed to trigger is in one of two states: nothing matched it, or it does not work. These are indistinguishable from the outside and the pipeline reports the same green either way. A control never seen to go red provides exactly as much assurance as no control, while consuming the confidence budget of a real one.
The discipline. Every detection references the intelligence that motivated it. Every rule is validated against a known-positive before deployment and re-validated on a schedule. Detections are aged and retired on the same clock as the intelligence behind them.
The property that has to survive all eight stages
Stated once, because it is the whole design:
Every stage must preserve the ability of a later stage to reason about the evidence.
A stage can be individually correct and still break the pipeline, by producing accurate output that has been stripped of the context a later stage needs. This failure is invisible to any test that only inspects a stage’s output, which is why it survives review and why it is the normal condition of pipelines in production.
Three consequences follow, and they are the things I would check first in any deployment:
Unknown must be representable end to end. If any stage lacks a way to say “could not determine”, it will emit a default, and the default will be read as a finding by everything downstream. The most confident output in the pipeline will be produced exactly where it knows least.
Coverage must be reported alongside every result. A measurement that reports what it examined without reporting what it could not examine systematically overstates. Worse: the number can be improved by examining less, and any optimising process will find that gradient. The general form — a denominator derived from the measured data’s own span cannot detect the data that never arrived — is why so many pipelines report perfect coverage indefinitely.
Controls must be observed to fail. For each gate, the useful question is not “is it running” but “when did it last reject something, and can I make it reject something now?” A gate with no recorded red is not evidence of a clean pipeline. It is an untested claim wearing the uniform of a passed test.
Status
This is a working framework rather than a finished publication — a description of how I structure the problem, refined against practice. Individual stages are treated at more length in the Articles section; the confidence and attribution material in particular is developed further there.