The product looks right. Two pages, structured sections, a named threat actor, confidence language calibrated to the judgment — "we assess with moderate confidence" — and a clean analytical conclusion. A consumer reads it, flags the actor for priority tracking, and routes it to the team owning that threat cluster. Nothing in the artifact signals that anything is missing.
This is the problem. The assessment isn't wrong, but a consuming analyst has no way to determine which sentence rests on which source. The form of finished intelligence and the tradecraft of finished intelligence were always separable in principle. They rarely separated in practice because the same analyst produced both: the person who wrote the judgment was also the person who graded the sources, weighed the competing hypotheses, and could be asked — by a reviewer, an evaluator, or a downstream consumer — to show their work.
Generation removes that person. The artifact remains. It carries the register and structure of finished intelligence, and none of the sourcing record that made the tradecraft layer enforceable.
What the tradecraft layer was enforcing
The tradecraft requirements on intelligence products are not conventions. Under Intelligence Community Directive 203, all IC analytic products must be consistent with five Analytic Standards that encompass nine Analytic Tradecraft Standards, the first of which requires a product to properly describe the quality and credibility of its underlying sources, data, and methodologies — identifying which sources and methodologies support which judgments, and using source descriptors that convey the factors affecting credibility (Source: ODNI Intelligence Community Directive 203, Analytic Standards as amended, https://www.dni.gov/files/documents/ICD/ICD-203.pdf).
ICD 203 further requires each IC element to maintain a product evaluation program using those Analytic Standards as its core assessment criteria — the standards are the foundational basis for audited product review, not stylistic guidance (Source: ODNI Intelligence Community Directive 203, Analytic Standards as amended, https://www.dni.gov/files/documents/ICD/ICD-203.pdf). A product can be evaluated against them because a producer can be asked to justify how each was met.
This auditability was the enforcement point. A human analyst who wrote a judgment could be asked: which source supports this? How was that source graded? What competing explanation did you consider and why did you set it aside? The standards attached to a person. The product evaluation program could reach through the artifact to the reasoning behind it.
Generated summaries break that chain while leaving the artifact intact. A product that reads as if it carries sourcing, confidence calibration, and alternative hypothesis consideration may carry none of those things in any recoverable form. The form of compliance survives. The substance that made compliance meaningful does not.
Why this is not the SOC case record problem
The generated case narrative relocated a decision — a generated case summary moved a judgment earlier in a workflow that had no formal sourcing standard governing it. The consequence was a decision architecture question.
An intelligence product operates under a different condition. The nine Analytic Tradecraft Standards in ICD 203 enumerate specific obligations — source quality description, confidence expression with its basis stated, alternative hypothesis consideration, information versus judgment distinction — and an evaluation program audits products against them. A generated intel assessment does not relocate a decision. It defeats a standards regime by satisfying its form: the product looks like it met the standards, and there is no producer to verify that it did. That is a different failure mode, and it requires a different remediation. The case record precedent is the closest analog in this corpus; the analogy stops at the point where formal tradecraft requirements begin.
What a generated intel product must carry
The table below maps each of the four core tradecraft obligations to what a generated product presents in their absence, why the omission is invisible in the artifact, and what field in the product record would restore it.
| Tradecraft Requirement | What It Obliges The Producer To Record | What A Generated Summary Presents Instead | Why The Omission Is Invisible | What Field Would Restore It |
|---|---|---|---|---|
| Describe source quality and credibility | Per-claim source descriptors indicating the basis, access, and reliability of each source supporting each judgment | A bibliography or footnote block at product end, or no citation at all | A bibliography carries the appearance of sourcing without linking any source to any specific judgment; the reader cannot tell which claim it supports | Per-claim source attribution field: each judgment tagged with the source identifier and its grading at time of production |
| Express and explain confidence | A confidence level paired with the specific basis for that level — what would raise or lower it, and what drives it toward high or low | A bare confidence adjective — "moderate confidence," "high confidence" — with no stated basis | Calibrated language reads as tradecraft; nothing in the artifact reveals whether a basis was evaluated or the term was inherited from a template | Confidence basis field: a structured entry stating the evidence type, its completeness, and the factor or gap most responsible for the current level |
| Consider alternative hypotheses | An enumerated record of competing explanations considered and the reasons each was accepted or set aside | A single analytical conclusion stated as the assessment | A declarative conclusion is indistinguishable in form from a conclusion reached after hypothesis enumeration | Alternative hypothesis log: at minimum, a field listing hypotheses considered and the dispositive factor for each rejection |
| Distinguish information from judgment | Explicit marking of which statements report collected information versus which represent analytic inference from that information | Declarative sentences throughout, with no structural differentiation between reported fact and analytic conclusion | The declarative register of finished intelligence applies equally to collected facts and inferred conclusions; both read the same way | Information and judgment provenance marker: a field or inline tag distinguishing retrieved content from generated inference at the claim level |
Three of the four obligations reduce to fields a consumer can check; the fourth does not. A hypothesis log is producer-side tradecraft — a consuming analyst cannot validate that the enumerated alternatives were the right ones, or that the dispositive factors were weighed honestly, in the way they can check whether a judgment carries a source and a confidence basis. It remains an obligation on the producer and an audit item for an evaluation program; it is not a verifiability field. So the minimum structural requirements for a generated product to be consumable against tradecraft standards are three: source attribution per claim rather than a bibliography; confidence expressed with its basis stated rather than a bare adjective; and a record distinguishing what the generator retrieved from source material versus what it asserted as inference. When any of these three is absent, a consuming analyst cannot verify sourcing, cannot calibrate confidence, and cannot identify which conclusions are retrieved and which are generated — making compliance with the ICD 203 evaluation program a matter of form rather than substance.
A bibliography attached to a generated summary is the absence of per-claim attribution wearing its clothes. It satisfies the appearance of sourcing while providing no mechanism to link any judgment to any source. In environments where generated summaries feed into an evaluation program, treating a bibliography as equivalent to per-claim attribution creates a gap that an auditor can expose but a consumer cannot.
What has not changed
Collection gaps still bound what any product can say. A generated summary that retrieves from a well-structured collection produces a more complete artifact than one retrieving from a sparse or unvalidated set — but neither generator nor consumer can exceed the limits of what was actually collected. A model cannot compensate for a collection gap; it can only surface, structure, or infer from what was gathered. Where collection is thin, a generated product may assert with the same register and confidence language as a product grounded in deep collection, and the artifact will not distinguish them.
A requirement that was never stated still produces intelligence nobody needed. Generation does not change the dependency on tasking: a summary of the wrong question is still the wrong answer, at higher production velocity.
Confidence has always been a judgment rather than a measurement. Analytic confidence reflects the quality and completeness of the underlying information and the degree of analytic uncertainty — it is an evaluative act, not a calculation (Source: ODNI Intelligence Community Directive 203, Analytic Standards, Section D.6.e, https://www.dni.gov/files/documents/ICD/ICD-203.pdf). Generation can reproduce the vocabulary of confidence expression; it cannot substitute for the evaluative act that assigns a basis.
The ICD 203 standards themselves have not changed. The nine Analytic Tradecraft Standards remain the criteria against which IC products are evaluated (Source: ODNI Intelligence Community Directive 203, Analytic Standards as amended, https://www.dni.gov/files/documents/ICD/ICD-203.pdf). What changed is whether there is a producer who can be held to them.
The question to ask of your last ten products
Take the last ten intelligence products your team consumed — finished assessments, not raw feeds or case records. For each, identify the judgment that most directly influenced a decision: a priority call, a resource allocation, a detection configuration, a policy exception.
Then ask two questions of that judgment. First: which specific source supports it, and how was that source graded at the time the product was written? Second: is the confidence level accompanied by a stated basis — what the analyst weighted most heavily, and what gap prevented a higher confidence rating?
Where the product cannot answer either question, the consumer acted on the register of finished intelligence — the confident declarative sentences, the structured format, the named actor — rather than on recoverable sourcing. That is not a failure of the analyst who read the product. It is a failure of what the product carried. The analyst cannot verify sourcing that was never recorded; they can only trust or distrust the artifact as a whole.
If fewer than half of those ten products carry per-claim source attribution and a confidence basis statement, the consuming team's decisions in that period were grounded in analytical form rather than analytical substance — regardless of whether the products were human-produced, generated, or hybrid. The tradecraft requirements exist to prevent exactly that condition. Whether they are being met is a question the product record should be able to answer without asking the analyst who wrote it.
Sources
- ODNI Intelligence Community Directive 203, Analytic Standards (as amended): https://www.dni.gov/files/documents/ICD/ICD-203.pdf
