OPINION — The US Intelligence Community (IC) is experiencing tumult and facing critique, and all things point to significant reform, creating an opening for the analytic community to rethink how it articulates probability in a way that is useful for customers and usable for analysts. DNI Gabbard has begun reducing her office’s budget and staff, and recently announced revocation of security clearances for a number of current and former intelligence officers. The Heritage Foundation, which initiated Project 2025, has called for analytic reform. Even British intelligence experts are noting an opening to reconsider our intelligence practices.
- Senator Cotton, during DCIA Ratcliffe’s confirmation hearing in January, opened his comments by pointing to recent missed calls: In just the last few weeks, members of this committee—and, I presume, the president—had no forewarning of the New Orleans terrorist attack or the collapse of the Assad tyranny in Syria. The same goes for Hamas’s October 7 atrocity against Israel in 2023. I could give other examples, but suffice it to say we’re too often in the dark.
- Cipher Brief authors in March offered a roadmap to rebuild the IC. The Heritage Foundation’s Robert Greenway—who also served on the landing team to prepare DCIA Ratcliffe for his confirmation early this year—has called for comprehensive reform of how the IC produces and presents analysis to the President of the United States.
- While some of the tenor now is new, intelligence professionals are used to reform. Indeed, it has been a constant for those of us who spent our careers in the IC. For example, since 9/11 some select changes include the USA PATRIOT Act (2001), Creation of the Office of the Director of National intelligence (2004), the WMD Commission Report (2005), issuance of ICD 203 Analytic Standards (2007), the National Strategy for Information Sharing and Safeguarding (2012), the USA FREEDOM Act (2015), and a host of ODNI modernization initiatives since 2020.
The current calls for change are looking for a grand strategy. My focus here is admittedly small and more practical – we need a simple-to-apply and understand gauge for weighing and communicating probability. Intelligence professionals, and analysts in particular, have long been focused on their own self-reflection and analytic biases; this moment of change presents an opportunity for our analytic community to refine their measurement of, and language around, probability. Rest assured (and I am now just giving an educated guess as a former analyst who knows our culture of self-critique), officers across the IC have scrutinized their analysis of the many misses that Senator Cotton highlighted.
Sign up for the Cyber Initiatives Group Sunday newsletter, delivering expert-level insights on the cyber and tech stories of the day – directly to your inbox. Sign up for the CIG newsletter today.
Part of the challenge is that what analysts get right or wrong is not so binary. It’s based on a spectrum of probabilistic language that they use, often not so rigidly as presented below, but still quite similar. The specturm, as I highlight below, is prone to miscalculation, misunderstanding, and analyst apathy. I think we can do better. The old standard is ready for a refresh that analysts use to the benefit of US national security.
- Current Standards – From ICD 203 section 2, a, noted above:
- My Proposed Standards:
I will not claim here that my proposed approach perfect, and I can hear my former colleagues now, pulling and picking it apart, as they should! I purposely shared this draft with no current or former analysts (which is the opposite of the norm) because I wanted to present a starting point for change. I contend that this new standard would be more usable, more often used, and yes, more accurate.
- More Usable – It may come across as flippant, but weighing four categories of probability is just more doable than weighing seven. This assertion becomes less flippant once you look more closely and realize that analysts are being asked to consider probability within 5-percentage point increments. Can any human (or any AI LLM) forecast the likelihood of human behavior to the 55th percentile? No. Are we equipped to give a general range? Yes. The current gauge is asking analysts to split hairs when, instead, we should be bundling those hairs to give a more realistic assessment of whether events will occur.
- More Often Used – Altogether, I have written and edited thousands of analytic assessments; overall, a tiny percentage used the ICD 203 framework. The truth is that analysts frequently use words such as “likely” and “unlikely” because that’s more effective than placing specific percentages in their written work. And so, they are quite unlikely (yes, less than 30 percent of the time) to pull out and use the ICD 203 standards. Congressional Oversight might say this is the problem. I would say we have given analysts a tool that is restrictive and cumbersome. Often, the best intelligence work is the analytic email or phone call to the customer sharing insights on the spot; that’s rarely, if ever, conducive to a seven-point probability gauge. And, when you do have time, the tool should work; I have sat in a few debates about whether something was likely or very likely. If the customer simply knew it was likely, and more than probable, it would have done the job.
- More Accurate – Analysis is both an art and a science, and it is becoming more of a science with advanced technologies that allow the analysts to focus more on the why and “so what” than they do on the what, when, and where. But when you require exactness (like asking for gauging probability to a five percentage points), your result is likely to be less accurate. I once worked for a senior manager who compelled us to make a call but to avoid “hammer judgments” such as “leader x will win the election” or “leader y will not fall from power.” Such judgments are a sort of overconfidence bias, which is sometimes rooted in overprecision. The job of the intelligence analyst is to forecast, not predict. If you give the analysts a tool with broader probability ranges that they are more likely to use, they will be more likely to make a call, and the call is more likely to benefit from the nuance that surrounds it.
Give AI a Chance
Analysts have an opportunity to use AI in their search for probability. The IC is wrestling with how to adopt AI and whether its adoption is existential. It can be helpful without being overwhelming. Examining the probability of an issue is one good example. If an LLM has access to all the same reporting as an analyst, an easy and productive exercise would be to ask the AI platform the likelihood of X happening, and then analyze that result in comparison to the analysts’ original findings. I asked Chat GPT, based on my proposed probability standards, the likelihood of Russia and Ukraine reaching a peace deal; the response was that it was Possible (30–50%). That’s good fodder for an analytic debate to hone your own findings.
The Cipher Brief is committed to publishing a range of perspectives on national security issues submitted by deeply experienced national security professionals.
Opinions expressed are those of the author and do not represent the views or opinions of The Cipher Brief.
Have a perspective to share based on your experience in the national security field? Send it to Editor@thecipherbrief.com for publication consideration.
Read more expert-driven national security insights, perspective and analysis in The Cipher Brief
