Category Research & Analysis

First World War Digital Resources

Informal portrait of Charles E. W. Bean working on official files in his Victoria Barracks office during the writing of the Official History of Australia in the War of 1914-1918. The files on his desk are probably the Operations Files, 1914-18 War, that were prepared by the army between 1925 and 1930 and are now held by the Australian War Memorial as AWM 26. Courtesy of the Australian War Memorial. [Defence in Depth]

Chris and I have both taken to task the highly problematic state of affairs with regard to military record-keeping in the digital era. So it is only fair to also highlight the strengths of the Internet for historical research, one of which is the increasing availability of digitized archival  holdings, documents, and sources.

Although the posts are a couple of years old now, Dr. Robert T. Foley of the Defence Studies Department at King’s College London has provided a wonderful compilation of  links to digital holdings and resources documenting the experiences of many of the many  belligerents in the First World War. The links include digitized archival holdings and electronic copies of often hard-to-find official histories of ground, sea, and air operations.

Digital First World War Resources: Online Archival Sources

Digital First World War Resources: Online Official Histories — The War on Land

Digital First World War Resources: Online Official Histories — The War at Sea and in the Air

For TDI, the availability of such materials greatly broadens potential sources for research on historical combat. For example, TDI made use of German regional archival holdings for to compile data on the use of chemical weapons in urban environments from the separate state armies that formed part of the Imperial German Army in the First World War. Although much of the German Army’s historical archives were destroyed by Allied bombing at the end of the Second World War, a great deal of material survived in regional state archives and in other places, as Dr. Foley shows. Access to the highly detailed official histories is another boon for such research.

The Digital Era hints at unprecedented access to historical resources and more materials are being added all the time. Current historians should benefit greatly. Future historians, alas, are not as likely to be so fortunate when it comes time to craft histories of the the current era.

TDI Reports at DTIC

Just as a quick easy test, I decided to find out which of The Dupuy Institue (TDI) reports are on the Defense Technical Information Center (DTIC). Our report list is here: http://www.dupuyinstitute.org/tdipub3.htm

We are a private company, but most of these reports were done under contract for the U.S. government. In my past searches of the DTIC file, I found that maybe 40% of Trevor Dupuy’s HERO reports were at DTIC. So, I would expect that a few of the TDI would be filed at DTIC.

TDI has 80 reports listed on its site. There are 0 listed on DTIC under our name.

https://publicaccess.dtic.mil/psm/api/service/search/search?&q=%22dupuy+institute%22&site=default_collection&sort=relevance&start=0

There are a significant number of reports listed based upon our work, but a search on “Dupuy Institute” yields no actual reports done by us. I searched for a few of our reports by name (combat in cities, situational awareness, enemy prisoner of war, our insurgency work, our Bosnia casualty estimate) and found four:

https://publicaccess.dtic.mil/psm/api/service/search/search?site=default_collection&q=capture+rate+study

This was four of eight reports we did as part of the Capture Rate Study. So apparently one of the contract managers was diligent enough to make sure those studies were placed in DTIC (as was our Kursk Data Base), but since then (2001), none of our reports have been placed in DTIC.

Now, I have not checked NTIS and other sources, but I have reason to believe that not much of what we have done in the last 20+ years is archived in government repositories. If you need a copy of a TDI report, you have to come to us.

We are a private company. What happens when we decide to close our doors?

Basements

Basements appear to be very important in the world of studies and analysis. That is where various obscure records and reports are stored. As the industry gets grayer and retires, significant pieces of work are becoming harder to find. Sometimes the easiest way to find these reports is to call someone you know and ask them where to find it.

Let me give a few examples. At one point, when we were doing an analysis of Lanchester equations in combat modeling. I was aware that Bob McQuie, formally of CAA, had done some work on it. So, I called him. Turns out he had a small file he kept of his own work, but he had loaned it to his neighbor as a result of a conversation he had. So…..he reclaimed the file, two of our researchers drove over to his house, he gave us the file, and we still have it today. Turns out that much of his material is also available through DTIC. A quick DTIC search shows the following: https://publicaccess.dtic.mil/psm/api/service/search/search?site=default_collection&q=mcquie

Of particular interest is his benchmarks studies. His work on “breakpoints” and comments on Lanchester equations is not included in the DTIC listing because it was published in Army, November 1987. I have a copy in my basement. Neither is his article on the 3:1 rule (published in Phalanx, December 1989). He also did some work on regression analysis of historical battles that I have yet to locate.

Battle Outcomes: Casualty Rates As a Measure of Defeat

So, some of his work had been preserved. But, on the other hand, during that same casualty estimation methodologies study we also sent two researchers over to another “gray beard’s” house and he let our researchers look through his basement. We found the very useful study called Report of the Model Input Data and Process Committee, reference in my book War by Numbers, page 295. It does not show up in DTIC. We could not of find this study without a visit to his basement. He now lives in Florida, where they don’t have basements. So I assume the remaining boxes of materials he had have disappeared.

I am currently trying to locate another major study right now that was done by SAIC. So far, I have found one former SAIC employee who has two volumes of the multi-volume study. It is not listed in DTIC. To obtain a complete copy of the study, I am afraid I will have to contract someone else and pay to have it copied. Again, I just happen to know who to talk to find out what basement it is stored away in.

It is hard to appreciate the unique efforts that go into researching some of these projects. But, there is a sense at this end that as the “gray beards” disappear; reports and research efforts are disappearing with them.

TDI Friday Read: How Do We Know What We Know About War?

The late, great Carl Sagan.

Today’s edition of TDI Friday Read asks the question, how do we know if the theories and concepts we use to understand and explain war and warfare accurately depict reality? There is certainly no shortage of explanatory theories available, starting with Sun Tzu in the 6th century BCE and running to the present. As I have mentioned before, all combat models and simulations are theories about how combat works. Military doctrine is also a functional theory of warfare. But how do we know if any of these theories are actually true?

Well, one simple way to find out if a particular theory is valid is to use it to predict the outcome of the phenomenon it purports to explain. Testing theory through prediction is a fundamental aspect of the philosophy of science. If a theory is accurate, it should be able to produce a reasonable accurate prediction of future behavior.

In his 2016 article, “Can We Predict Politics? Toward What End?” Michael D. Ward, a Professor of Political Science at Duke University, made a case for a robust effort for using prediction as a way of evaluating the thicket of theory populating security and strategic studies. Dropping invalid theories and concepts is important, but there is probably more value in figuring out how and why they are wrong.

Screw Theory! We Need More Prediction in Security Studies!

Trevor Dupuy and TDI publicly put their theories to the test in the form of combat casualty estimates for the 1991 Gulf Way, the U.S. intervention in Bosnia, and the Iraqi insurgency. How well did they do?

Predictions

Dupuy himself argued passionately for independent testing of combat models against real-world data, a process known as validation. This is actually seldom done in the U.S. military operations research community.

Military History and Validation of Combat Models

However, TDI has done validation testing of Dupuy’s Quantified Judgement Model (QJM) and Tactical Numerical Deterministic Model (TNDM). The results are available for all to judge.

Validating Trevor Dupuy’s Combat Models

I will conclude this post on a dissenting note. Trevor Dupuy spent decades arguing for more rigor in the development of combat models and analysis, with only modest success. In fact, he encountered significant skepticism and resistance to his ideas and proposals. To this day, the U.S. Defense Department seems relatively uninterested in evidence-based research on this subject. Why?

David Wilkinson, Editor-in-Chief of the Oxford Review, wrote a fascinating blog post looking at why practitioners seem to have little actual interest in evidence-based practice.

https://www.oxford-review.com/blog-research-problem-evidence-based/

His argument:

The problem with evidence based practice is that outside of areas like health care and aviation/technology is that most people in organisations don’t care about having research evidence for almost anything they do. That doesn’t mean they are not interesting in research but they are just not that interested in using the research to change how they do things – period.

His explanation for why this is and what might be done to remedy the situation is quite interesting.

Happy Holidays to all!

Isolating the Guerilla

The Vietnam was significant in that it was third bloodiest war in U.S. military history (58,000 U.S. killed) and the U.S. Army choose to learn no lessons from it !!! This last point is discussed in my book America’s Modern Wars: Understanding Iraq, Afghanistan and Vietnam.

In 1965 Trevor Dupuy’s HERO (Historical Evaluation Research Organization) conducted a three-volume study called “Isolating the Guerilla.” It was an interesting survey of 19 insurgencies that included on its research team 26 experts. This included General Geoffrey Lord Bourne (British Army, ret.), Andrew C. Janos, Peter Paret, among others.

These guys:

https://en.wikipedia.org/wiki/Geoffrey_Bourne,_Baron_Bourne

http://www.nytimes.com/1975/04/26/archives/col-r-ernest-dupuy-88-dead-publicist-and-military-historian.html

https://en.wikipedia.org/wiki/Trevor_N._Dupuy

https://es.wikipedia.org/wiki/Andrew_C._Janos

https://www.goodreads.com/author/show/2793254.William_A_Nighswonger

https://en.wikipedia.org/wiki/Peter_Paret

http://www.legacy.com/obituaries/northjersey/obituary.aspx?pid=163090077

https://en.wikipedia.org/wiki/Theodore_Ropp

https://en.wikipedia.org/wiki/Gunther_E._Rothenberg

http://www.ur.umich.edu/9495/Oct03_94/29.htm

http://www.andersonfuneralhomeltd.com/home/index.cfm/obituaries/view/fh_id/12343/id/3994242

http://www.nytimes.com/1984/08/31/obituaries/frank-n-trager-78-an-expert-on-asia-dies.html

 

The first volume of the study, although developed from historical sources, was classified after it was completed. How does one classify a study that was developed from unclassified sources?

As such, the first volume of the study was in the classified safe at DMSI when I was there. I was aware of the study, but had never taken the time to look at it. DMSI went out of business in the early 1990s and all the classified material there was destroyed. The Dupuy Institute did not have a copy of this volume of the study.

In 2004 we did our casualty and duration estimate for Iraq. It was based upon a survey of 28 insurgencies. We then expanded that work to do an analysis based upon 89 insurgencies. This was done independently of our past work back in 1965, which I had never seen. This is detailed in my book America’s Modern Wars.

As this work was being completed I was contacted by a Lt. Colonel Michael F. Trevett in 2008. It turns out he had an unclassified copy of the study. He found it in the Ft. Huachuca library. It was declassified in 2004 and was also in DTIC. So, I finally got a copy of a study after we had almost completed our work on insurgencies. In retrospect, it would have been useful to have from the start. Again, another case of disappearing studies.

In 2011, Michael F. Trevett published the study as a book called Isolating the Guerrilla. The book is the study, with many of the appendices and supporting data removed at the request of the publisher. It was a self-publishing effort that was paid by Michael out of his personal/family funds. He has since left the Army. I did write the foreword to the book.

What can I say about this case? We did a study on insurgencies in 1965 that had some relevance to the wars we entered in Afghanistan in 2001 and Iraq in 2003. It remained classified and buried in a library in Ft. Huachuca, Arizona and at DTIC. It was de-classified in 2004 and came back to light in 2008. This was through the effort of a single motivated Lt. Colonel who was willing to take the time and his own personal money to make it happen.

The SAIC Library

The story of the disappearing SAIC research library occurred in the middle of the 1990s, during the same time as the HERO Library was disappearing. SAIC had an “Military Operations Analysis Division” that for a time was a competitor to HERO/DMSI. In particular, around 1990, they hired three former HERO/DMSI employees and used them for studies that normally would have been done by us. Trevor Dupuy was on-the-outs with some people at the U.S. Army Concepts Analysis Agency (CAA). Some time in the mid-1990s, SAIC decided to close down their military operations analysis division.

The early 1990s were a difficult time for defense contractors. The Warsaw Pact and the Soviet Union had disappeared and the defense industry was shrinking. SAIC got rid of the division that did analytical work for DOD as they realized it was a dying business (something that we could never get through our heads). Companies like BDM, one of the stalwarts in the industry since 1959, was sold off in 1990s; with Trevor Dupuy’s old company, DMSI, also going out of business in the 1990s.

Anyhow, SAIC had a library for this division. It was the size of two double offices, maybe 400 square feet or more. It was smaller than the HERO Library. They decided to dissolve the library along with the division. They told the staff to grab what they wanted and dumped the rest. Having never had access to this library, I do not know if there were any holdings of value, but as SAIC had been around since 1969, it is hard to believe that there was not something unique there.

 

This post is related to:

The HERO Library

Missing HERO Reports

 

The HERO Library

The first research library that I was aware of that was broken up and scattered was Trevor Dupuy’s library that was kept at HERO/DMSI (HERO was a division of DMSI at this stage).

One has to talk corporate structure here for a moment. HERO (Historical Evaluation Research Organization) had been established and built up by Trevor Dupuy. In an attempt to expand the business, he created a company called DMSI (Data Memory Systems Incorporated) of which he was just one of the owners (although with about 40% of the stock). In 1987-1989 period, the U.S. defense spending reached it peaked as did DMSI, which had 25 employees. With Glasnost, Perestroika, the Warsaw Pact dissolving and finally the Soviet Union collapsing in 1991, the defense budget collapsed. In the resulting collapse, so to did DMSI. With the business crashing, Trevor Dupuy having a falling out with the other management and quit his own company.

DMSI/HERO had an extensive library and an extensive collection of research files, dating back to its founding in 1962. They even had share library privileges with the Library of Congress due to some unique material in the HERO library. The library took up a large room and there were file cabinets full of research files.

This library was broken up. First, Trevor Dupuy took the report file he kept in his office with him. This was the entire collection of 120+ reports written by HERO. Those eventually ended up at TDI (The Dupuy Institute). The library remained at DMSI, except for those books that the Dupuy family took from the library, which I gather was considerable. These are still in the hands of the Dupuy family. Then DMSI went out of business around 1993, and the remaining files and library were scattered. The employees were invited to take what they wanted out of the files. After that, one employee decided to rescue the files that he thought were important and moved them to his barn in rural Virginia. These included most (but not all) of the Ardennes files and Grace Hayes files (the original VP of research at HERO). A few years later, we arranged with that ex-employee to reclaim the files and he graciously brought them to our office from his barn. After we blew the dust, dirt, hay and mouse droppings from them, we then refiled them at TDI. The files taken by other employees were not recovered. Most of the remaining library was taken by a principal at the company and moved to his basement. They were used for his business for a while. Eventually, he needed to clean his basement and the “HERO Library” ceased to exist.

We were able to save most of the critical files, meaning the reports, most of the Ardennes files and the Grace Hayes files. The rest was lost, which was a shame, although not overwhelmingly critical. Still, the process got my attention and this is potentially a problem with any private company. Unless someone goes through some extra-ordinary process to preserve the files and libraries of their work, then when a private company collapses (which most do at some point), those files are lost. I am aware of several other cases like this.

Missing HERO Reports

Back in 1987 I did a DTIC search for HERO reports (DTIC is the defense library of reports, HERO was Trevor Dupuy’s old company(s) that produced around 130 or so reports). My DTIC search ended up finding something like 40% of HERO reports in their files. As almost of all these reports were done under contract for the government, then the figure should have been something more like 100%.

Now, I guess if I was a responsible citizen, I would have made sure that all the missing reports were identified, copies made, and they were then sent to DTIC. I did not do this, because as I busy running a large project that was behind schedule and over budget (the Ardennes Campaign Simulation Data Base).

But……this little survey got my attention concerning what was preserved in the national report library (like NTIS…National Technical Information Center) and what was not. It raises the question as to whether we are properly preserving the studies and results of all this analysis that various companies have done….or are we loosing some of it. Unfortunately, I have found enough cases over the years of significant and important studies and files disappearing or becoming difficult to find….that I have become concerned. Whether this is indicative of a larger problem I will leave to the reader to determine, but there will be a few blog posts about this subject or the next week or two.

All of our reports and studies are listed on our website here: http://www.dupuyinstitute.org/tdipubs.htm

It is some 130 HERO reports and 80 TDI reports.

It would be possible to someone to search the NTIS site and see how many these reports can be found (and which ones they cannot find). I would be interested in knowing the results of that if anyone wants to spend a few days doing this. If the problem exists for HERO and TDI reports…then it probably exists for work done by a lot of other companies also.

So Why Are Iraqi Records Important?

Last preachy post on this subject:

The Sad Story Of The Captured Iraqi DESERT STORM Documents

So, why does it matter? Well, Iraq has been involved in three wars in recent time that are significant.

We engaged them in the 1991 Gulf War. Even though this was a truly lopsided result, we do not have good two-sided data available for this war (it may exist, but it is classified/closed). As this was our largest conventional operation since the Korean War (1950-1953), then having good two-sided data for this war would be useful. As it is, the last major conventional fight that the U.S. participated in where there is good two-sided data is in Italy through June 1944. After that, German records significantly degrade and we do not have a good collection of opposing force data for Korea, Vietnam or Iraq….although we do have it for Grenada ;).

Then we invaded and conquered Iraq in 2003. This was effectively a three-division operation that went extremely well also. As this was the last major conventional operation we conducted, it would be useful to have good data for the opposing side (hopefully we do, but again, it is classified/closed).

But, probably most significant is the Iran-Iraq War from 1980-1988. This was the largest conventional war since the early 1970s (India-Pakistan in 1971 and the Arab-Israeli War of 1973). This was also the first major war that used chemical weapons since World War I (1914-1918). Chemical weapons and nerve agents were used extensively by Iraq against the Iranians, and also brutally against Kurdish civilians. Iran did make limited used of chemical weapons in response. It is also the only extensive use of nerve agents in warfare. This is probably important to understand and analyze. We do not have records from the Iranian side, so it would be nice to have records from the Iraqi side. Hard to analyze their operations if we have records from neither side. We really have no actual operational data on the effects of chemical weapons in combat since 1918, and we have no operational data on the effects of nerve agents in combat. This could be a major shortfall.

Now, the documents destroyed, according to the article we originally cited, was only the material captured during the 1991 Gulf War, with records dating back to 1978.  Some of it (at least 60%) was preserved on digital tapes. And, perhaps we have preserved all the military records we captured in 2003. So, in fact there may be an extensive collection available (at least 725,000 pages), although it is classified or closed to researchers. Based upon our track record of record preservation from Vietnam and onward, I have reason to be concerned.

Anyhow, a related link on our chemical warfare studies (including links to articles on the Iran-Iraq War):

Survey of German WWI Records

 

P.S. There are over a dozen counties in the world that still have chemical warfare arsenals, including Syria, Iran and North Korea.

 

U.S. Records in the Gulf War (1991)

Of course, there were problems with the U.S. Army record keeping in the Gulf War (1991). There were serious problems with the U.S. Army record keeping in the Vietnam War (1965-1973), so not surprising, the problem had not been corrected, and the same problems existed 20 years later. In the Vietnam War, the 82nd Airborne Division pretty much threw away most of their records. According to Don Hakenson, Director, Center of Unit Records Research, Records Management and Declassification Agency; in the Gulf War, 86% or 87% of the battalion daily journals were not preserved (see War by Numbers, page 146).

This became a big issue when the “Gulf War Syndrome” became an issue. People became suspicious that U.S. soldiers had become exposed to hazardous materials or chemical weapons. Yet, when the Veterans Administration and others tried to figure out where the units were at the time, they found that the records no longer existed for many these units. In many cases, they could not determine where the unit or the people were during operations. Many of the records had simply been thrown out.

The Gulf War Syndrome was not a small issue. It has been estimated that 250,000 U.S. veterans were afflicted. It was a case where record keeping briefly became a major issue. Wikipedia article: https://en.wikipedia.org/wiki/Gulf_War_syndrome

Since the 1960s, there has been serious gaps in U.S. record keeping. There still was in 1998 when we conducted a survey of the subject for the U.S. Army. We have conducted no other surveys since then, but gather that corrective action has been undertaken.

U.S. Army Record Keeping