Only a Sith Believes in Absolutes [when it comes to Net Assessment]
The only thing more annoying than Star Wars nerds fighting over lightsaber crossguards and scrutinizing every little detail of a tiny teaser trailer is the familiar shuffle of hagiography and hit piece regarding the Office of the Secretary of Defense's Office of Net Assessment (ONA). Either Andrew Marshall is "Yoda" or he is Emperor Palpatine.
The idea of net assessment (as well as the office that implements it) is the subject of both excessive hype and excessively uninformed criticism. While Thomas Mahnken has demolished the most egregious factual errors and most uninformed criticisms, those who favor net assessment, Marshall, and the office do not address potential and actual criticisms more sophisticated than the most prominent net assessment bashing.
I will take some recent discussion of ONA and Andrew Marshall in particular as a launch point to both review commonly misunderstood aspects of the office, offer my own critique, and then talk about what kind of conversation we should be having about net assessment. I am not an unbiased observer, as I'll note in the conclusion, but I also hope to convince readers that I am at least open to looking at problems of the office that most sympathetic to it eschew or undermine.
It goes without saying that there are copious Star Wars gags.
A Long Time Ago, In a 5-Sided Polygon Not So Far Away
The popular image of Andrew Marshall is of a sage ("Yoda," though the main prophet of The Foundation seems to fit the image better) gazing into a crystal ball to divine the future of warfare. If this were indeed true, many of the criticisms of Marshall would be on point. Marshall's particular vision of future warfare, his own ideology and fellow travelers, and his idiosyncratic individual views would be valid targets. They certainly are provocative, to say the least.
Unfortunately for critics, this stereotype could not be further from the truth. Net assessment as an activity is not about making predictions of the future. Nor does the success and failure of the office rest on the sagacity of a single man's talent for predictions and analysis. What follows is an extreme abbreviation of an enormous corpus of documents (some of them decades old) that nonetheless hopes to explain the basics of what Marshall and his colleagues set out to do.
Net assessment arose out of unhappiness with the traditional method of military-strategic analysis. Marshall was an economist and was fond of quantitative analysis. However, much Cold War quantitative analysis amounted to bean counting that did not control for intangibles and/or things that were far more difficult to quantitatively model. In Marshall's own work on analyzing military power, he pointed out that socio-bureaucratic variables were absent from quantitative analyses.
Bean-counting does not work because, as Robert McQuie (see his sadly offline article "Battle Outcomes: Casualty Rates as a Measure of Defeat" if you can find it) and others looking at empirical data of World War II battles noted, even well-developed force planning models did not account for past empirical results. In some models, they even predicted that the French would have won the 1940 Fall offensive (instead of the Germans).
On a less abstract level, analyses also either never directly compared US and Soviet matchups or did not do so in a manner that chose the right analytic measures. Finding good analytic measures, however, is bound up in the more abstract and less tangible issues described in the previous paragraph. Lastly, net assessments are not point estimates but analyses of how a particular area of strategic competition might unfold over a period of time. In peacetime, policymakers would need to understand how the dynamics of a competition might play out. Understanding the structure and dynamic of a competition was more important than a point estimate of what would occur at some future date.
In sum, Marshall's philosophy was not too different from Joshua Epstein's ideas about the purpose of modeling:
One crucial distinction is between explain and predict. Plate tectonics surely explains earthquakes, but does not permit us to predict the time and place of their occurrence. Electrostatics explains lightning, but we cannot predict when or where the next bolt will strike. In all but certain (regrettably consequential) quarters, evolution is accepted as explaining speciation, but we cannot even predict next year's flu strain. In the social sciences, I have tried to articulate and to demonstrate an approach I call generative explanation, in which macroscopic explananda—large scale regularities such as wealth distributions, spatial settlement patterns, or epidemic dynamics—emerge in populations of heterogeneous software individuals (agents) interacting locally under plausible behavioral rules (Epstein 2006; Ball 2007). For example, the computational reconstruction of an ancient civilization (the Anasazi) has been accomplished by this agent-based approach (Axtell et al. 2002; Diamond 2002) I consider this model to be explanatory, but I would not insist that it is predictive on that account. This work was data-driven. But I don't think that is necessary.
You can see why this would be appealing to policymakers, who do not have time or resources to gather all of the necessary data and instinctively distrust predictions from quants in the social sciences. They must also make expensive investments with far-reaching implications (see: the F-35) with little idea of how the platform will perform in real-world combat settins.
If you do not believe me (or Epstein), read Paul Bracken's 2006 paper on how to do net assessment, or my own application of it to counterterrorism. My friend Daveed Gartenstein-Ross's book Bin Laden's Legacy is a great example of net assessment in action -- Daveed looks at a competition (America vs. al Qaeda), makes some claims about what dynamics drive the competition, assesses the two sides, and concludes that al-Qaeda is successfully implementing a cost-imposing strategy against the US.
Of course, competitive strategies are just one of the many innovations produced by ONA. Mahnken details more:
In fact, however, Marshall’s influence has been more far reaching than Lewis admits. Marshall personally and the office that he has led have been responsible for studies on a wide variety of topics, including:
Bureaucratic politics (read the acknowledgements to Graham Allison’s seminal The Essence of Decision).
Military power (to include Marshall’s own monograph, Problems of Estimating Military Power, written while he was still at the RAND Corporation).
Military effectiveness (to include sponsorship of Williamson Murray and Allan Millett’s landmark three-volume study of the same name).
And importing the Soviet concept of Revolutions in Military Affairs.
While Mahnken's write-up may strike some readers as an attempt to turn Marshall into a cross between The Most Interesting Man in the World and Dr. Strangelove ("I don't always do strategic military analysis, but when I do I prefer Net Assessment"), all of what he writes above is true. Marshall and his acolytes were responsible for an enormous amount of contribution to the strategic studies discipline -- in and out of government.
It is not for nothing that Marshall wrote an essay titled "strategy as a profession for future generations." Marshall funded research on ideas with far-reaching (and often unseen) impact on international relations, national security, and comparative politics. Students of all of those disciplines -- including many of Marshall's critics -- benefited. To paraphrase Nas, it is hard to find strategists that Marshall has not influenced one way or another.
The Knights of the [Cold War] Republic?
What I just described sounds a lot like.....strategic thinking. Why do we need an separate office for this? Can't it just be done within the Department of Defense in a less exotic fashion or delegated to the National Security Council? If you put yourself in the mind of many SecDefs, the answer to that question is "no."
The American political-military system has several features that act as a double-edged sword. First, it is driven by present concerns and waves of fads. In the late 1990s, it was all about peer competitors and high-end technology. In the 2000s, it was about counterinsurgency and counterterrorism. In the 2010s, it was (and is) about counterterrorism and high-end technology for peer competitors. The community takes its cues from its perception of the international situation, signals cast by intellectual elites in the defense and public policy community, and where it thinks the money will go.
The plus side of this is that it can allow the community to refocus attention and resources on emerging threats. The downside is a lack of historical memory and a need for a giant helping of ADHD medication to prevent it from chasing the next shiny object down the rabbit hole. The very same people who are just SO SURE that COIN, CT, Anti-Access/Area Denial, Cyber, Robots, etc are the Future of Warfare (TM) will just as soon forget that they were so sure once they jump onto their next hobbyhorse.
Long-term strategic analysis and thinking (along with the fact that the office is peanuts in the DoD budget universe), thus, helped keep Marshall and friends in the mix. But this is not the real reason ONA survived for so long. After all (as I allude to throughout the piece) ONA had its hits and misses as well. So why, despite the inconsistencies, did ONA survive for so long?
Critics would like you to believe that it was in part due to a Dear Leader-like cult of personality that developed around ONA and Marshall. It's certainly true that Marshall and ONA have an air of mysticism to them and are often the subject of breathless articles and analytical boot-licking. But before readers default to their "evil old neocon cold warrior war hawk MANIPULATES TEH CITENZRY AND KILLS TEH KITTENS" ONA piece they no doubt have on their next Firefox browser tab, consider the following about the bureaucratic landscape that ONA existed within.
The US system has many bureaucratic overlaps and a competitive struggle both across and within institutions for resources. These factions do not just struggle over the latest allocation of the rice bowl (give me the money, beggar the Army or the Air Force), they contest ideological visions of foreign policy, national security policy, and military policy. No one (especially not net assessment critics, many of whom come from the arms control and Realist foreign policy communities) is neutral. As a plus, it is a buyer's market for senior executives. They can watch the gladiators fight for the almighty budget dollar and the ear of the Grand Poombah, and give the particular gladiators they like the thumbs up sign.
But how ought the defense executive make that decision? Who can they trust to be *their* advisor as they choose advisors? The purpose of ONA is to report directly to the SecDef and work on what he or she thinks is important -- not the Army, not the Air Force, not the Navy, and not the various other gladiators (war colleges, research institutes, think-tanks, etc) fighting for a piece of the pie. And successive generations of SecDefs retained Marshall and his office.
I will not defend all of Marshall's particular views or those of his more vocal associates. I myself loathe "neocons" (whatever that means -- the term, like "neoliberal" has been weakened by left-liberal overuse to refer to anything to the right of Rachel Maddow). But it is important that criticism of ONA come from a place of knowledge, not stereotype. For example, it is a myth (which Mahnken dispenses with handily) that Marshall had some ideological aversion towards strategic stability. In any event, the office's role and function is so hard to characterize in one go that simplistic criticisms such as this fail to persuade regardless.
Many critics error in torturing the facts and speaking out of a place of gross ignorance in asserting that ONA is simply a retirement home for Cold War neocons ever eager to relive the Fulda Gap days. There is a cottage industry in pieces with this theme going back to the early to late-90s.
But many ONA boosters do not understand that merely correcting the errors of ONA's legion of ideological and bureaucratic enemies is not enough. They have to provide an affirmative case for the institution as rigorous as the net assessment methodology they propose. Just as with the Jedi Knights of the Old Republic, the fact that ONA is not a Sith enclave ought not to grant it a free pass. The Old Republic's institutions (including the Jedi Council) had immense flaws, flaws that not even Yoda, Obi-Wan Kenobi, or Mace Windu could beat back through sheer force of will. Maybe the same holds true for ONA?
I now analyze some flaws and areas of potential critique I have observed over the years since I first came into contact with the ONA literature and mythos.
First, the foundation of the office might be summed up as SecDef knows best. Is this necessarily true? Many during the Rumsfeld era of DoD would strongly disagree. An office that reports directly to the SecDef and handles the SecDef's business may or may not be an optimal institutional arrangement. The office of the Secretary of Defense seems to grow smaller and smaller with each administration, Rumsfeld being an aberration. If true control over resources really lies with a cabal around the President and his network of court advisors, does it still make sense to have the SecDef as a middleman?
Second, few ONA defenders have addressed the real criticism of the Yoda-bashers: the composition of the office itself is not diverse enough to produce useful output. Once the SecDef asks for something, does Yoda and his small staff check their own biases when delivering it? There may be a counterpoint that the biases of the staff are appropriate -- it helps to have a high-end conflict man or woman around if everyone else in the government is going hog-wild over COIN. But in any event the office may need a mechanism other than the happiness of the SecDef to convince that they are doing the right job -- and that begins with analyzing the organizational composition of the net assessment system and how its dynamics deliver particular research and analytical outcomes.
This is extremely important given that much of the office's actual deliverables are contracted out. Everyone in DC is angling for study money, but who gets it and why?
Defenders of the office ought not be as mysterious and enigmatic as the real Yoda in talking about this issue. For those beyond the SecDef to take the office seriously, it cannot afford to mimic the mysticism and doublespeak of actual Jedi. If a principal (or influential media analyst) discovers that ONA misled them in a way akin to the way Yoda and Obi-Wan massaged the truth about Luke's father, then that DoD principal (like Mace Windu evaluating Palpatine's moves) will conclude that ONA has truly gone over to the Dark Side.
Finally, the issue of competitive strategies themselves needs to be more deeply examined. An analysis of a strategic competition is a model. As an academic modeler, I am accountable for verification and validation of every model. All modelers have to demonstrate that they have done due diligence in building the model, and even toy models are validated -- I still have to show my model captures a qualitative carciature of an outcome observed in real life. How are the dynamics of strategic competition verified and validated? Additionally, what role should "out of sample prediction" play in such validation?
Erin Simpson, the woman I believe to be one of Marshall's three spiritual successors in the government space, gave a royal smackdown to people who didn't verify their tools and validate their data. Jay Ulfelder (the other one of my two picks for ONA, I won't tell the third due to his request) makes a similar point about the bullshit-checking function of prediction. Both, I think, get to some similar issues.
If at least one problem is (PROBABILITY_OF_RUSSIANS_PWNING_US | AVAILABLE_EVIDENCE) how is that evidence validated and the analytical process verified? Moreover, a strategic competition analysis (or any other kind of analysis) ought not be static and should be iteratively updated as new information comes into play.
Given the awful record of predictions of complex political-military outcomes (note that very few to little net assessment critics volunteer their own analytical track record, and it is hilarious to read one academic, of all people, critiquing Marshall for hedging), I am hesistant to fully endorse the idea of making predictive power the coin of the realm in net assessment. And contra some who have thrown rocks at Marshall, the fact that he and his office were inconsistent and/or insufficient in applying his own ideas does not constitute evidence that he or the office was fatally flawed. Against whose track record, I question?
Surely not that of the Beltway bandits and professors who have critiqued him (and ONA), who would never survive such criticisms. The mere fact that some can look at ONA can be examined on such a long timeline suggests critics themselves are performing net assessment, in their own flawed and ridiculous manner. ONA, like Yoda, is a very, very old animal in DC years. Hence a substantial portion of his (and ONA's) failings are sitting directly in public view.
However, what I do want is an affirmative case for ONA's existence that does not rely on Saint Andy (TM), addresses the critiques I outline (as well as some of the less inflammatory points offered by more knowledgable critics), and also addresses the biggest strike against ONA and the Marshal model I can think of -- it's relevance. Marshall was, after all, originally a young man slaving away in one of DC's many think shops who found the existing paradigm of analysis lacking. All ideas have a context. Can Marshall's ideas survive outside of his own intellectual context? Does the problem that net assessment seeks to solve exist anymore, and if it does how important is it? What modifications would be needed to modernize and optimize net assessment?
There is probably a future Andy Marshall somewhere in the deep bowels of the mil-industrial complex, raging against the Little OSD Office That Could and its dominant dogmas. Can ONA accommodate him? Or will another analytical revolution need to take place? I hope analysts can speak to that hypothetical young man or woman's frustrations, not spam him or her with hagiography or Cold War neocon bloody shirt waving.
ONA, as fascinating and useful as it has been, is not somehow immune to the laws of bureaucratic gravity. So how do we conduct a net assessment of net assessment?
A Elegant Weapon for a More Civilized Age?
I will lay my own cards out on the table. I am a fan of ONA. I believe its products and the net assessment methodology are highly regarded throughout DC, as evidenced by the diverse array of people (who span service, bureaucratic, ideological, and disciplinary divides) who have paid tributes to the office and its products and its long history of serving the SecDef (who could have axed the office at any point, without having to deal with even half the constituency of any other major DoD program). Many of the military officers I correspond with or have gotten to known -- suspicious of whiz-kid civilians and gizmos to a fault -- endorse the office and its work. I have read the primary source documents and taken graduate coursework from some of Marshall's impresarios as well. These are my priors, and they are nothing if not deeply held.
I also will admit a sense of cynicism in evaluating the spate of ONA hit pieces. DC and the parts of academia that interface with the five-sided funhouse are chock-a-block with rivalries both ideological and material. Whether one views ONA as a threat to a bureaucratic rice bowl or needs Andy Marshall to play the Grand Neocon Warhawk Puppetmaster (TM) role in their pet theory of why the national security establishment did/didn't do X or Y, Marshall and his office have as equally eclectic crew of player haters as they do supporters. And the hateration is indeed quite strong.
Gullible audiences who don't stop to consult primary source materials or think about the historical, ideological, and bureaucratic context of such hateration swallow it uncritically. In other words, we are currently seeing the commetariat treat ONA with the same degree of strong analytical certainty and blinding ignorance that characterizes so many other DC debates of note lately.
Yet I also, if a Marshall fan, am nonetheless not a Marshall fanboy. I have struggled over time to update my own net assessment of net assessment with recourse to a basic fact of life here in DC: government is inefficient, inconsistent, factional, ideological, and often a struggle by organizations to retain funding and authorities even if they no longer serve the mission goals. Why would Marshall and ONA be immune from this?
As a side note, it is hard to find any other government program that many of Marshall's critics have applied such an excessively critical lens toward -- accountability in government is only a thing when the government program in question is that of a perceived ideological adversary. Still, Marshall was not God and ONA is not immune to the things that ail our government and natsec spheres.
We desperately need a net assessment of net assessment, and one done by those who can dig deep into the primary source literature instead of waving a bloody neocon/Cold War hawk shirt. Most important, an assessment that finds what was worth keeping about the office both intellectually and bureaucratically, and what has not aged particularly well or did not work. You will not find this kind of assessment in the writings of Marshall acolytes either. It is hard to dispute that net assessment did some good, and it is even harder to dispute that it served a certain set of bureaucratic and institutional requirements. But beyond that, everything else is hagiography.
Why is this important? After all, why mourn an office with a small budget and staff that barely is mentioned in the memoirs of key officials? Well, DC and natsec academia have a strategy fetish. Everyone is bemoaning that we don't have a strategy (grand or not so grand), that our government cannot think in the long term, blah blah blah blah.
These complaints ignore that the US system is at least partially configured in a manner that mitigates against long-term, deep strategic thinking, and we are at least in one way all the better for it. One man's long-term, knowledgable set of bureaucrats is another man's Philosopher-King Council or Deep State. Getting fresh blood in -- and having a system that is hard for any one faction to dominante for a particular period of time due to the degree of intra-elite fighting -- is a Good Thing.
Moreover, everyone with a Grand Strategy (TM) to sell you really has a Grand Ideology, an individual vision of the particular composition of bureaucratic rents and arrangements and national security actions derived from a particular ideological substrate. At its most farcial, DC and academia's grand strategy dreams becomes a strange, amorphous glob of nothingness that somehow works sustainability and "walkable communities" into national security strategy. So no -- I don't trust your Grand Strategy, and if I ever came up with one you sure as hell would be an idiot for trusting mine.
This leaves us back where we started -- the need for some kind of mechanism of long-term, supra-factional process for giving the top dogs in the military-industrial complex advice about hard decisions they must make.
Even if many articulations of the need for integrated strategy either disregard politics or are example of the kind of DC foodfights and lack of objectivity they critique, the need for strategy and long-term thinking still exists. It need not be perfect -- after all, how do we really know any strategy achieved the desired result? -- it only has to be "good enough." That need will be filled, somehow. ONA was not the only office that played a key role in filling it, nor is it necessarily a foolproof model of how to fill the need.
However, it was one of the few offices that developed ideas that are now widely in use outside DoD by everyone from military historians to political scientists (a cats and dogs pairing if there ever was one), that was popularly perceived by generations of SecDefs as above the fray, and certainly the only place whose longtime executive has been compared to the most sagacious of Star Wars Jedi (despite an marked inability to engage in badass lightsaber duels or lift X-Wings from swamps).
Only a Sith Believes in Absolutes
No matter what happens to ONA, the need for strategic thinking beyond a soundbite lifecycle and above the particular rice bowl or political fixation of a natsec faction will always be there. ONA was a flawed attempt at delivering that institutional package. Critics and hagiographers aside, we need to be serious about meeting this demand.
Even critics ought to care about this - no matter how much you might hate ONA, something worse might take its place. Regardless, you don't have to like Yoda/Marshall to take the ideas of net assessment seriously. Just examine him and his office with the recognition and criticism that any kind of complex institution deserves.
I shouldn't need to tell you that only Sith believe in absolutes.
