We do have a section on rules of engagement in my book America’s Modern Wars (Chapter 9). In that effort, I ended up coding by judgment the rules of engagement in five categories (polite, strict, restricted, unrestricted, brutal). The telling chart is here:
As one can see, success tends to be at either end of the spectrum, with the counterinsurgents winning around 75% of the insurgencies fought with strict rules of engagements and the counterinsurgents winning around 75% of the insurgencies fought brutally. Anything in between those two points does not work as well.
We ended up doing this for all “83 insurgencies, interventions and peacekeeping operations” (the only category that had the “polite” cases, and they were 100% successful), for the “62 insurgencies” (show above), for the “36 insurgencies versus foreigners” (same pattern as above), and for the “26 insurgent civil wars” (same pattern as above). See pages 85-86 of America’s Modern Wars.
While the results were not statistically significant (see pages 86-87), the fact that the four different tests were all pretty consistent in results no matter which way you cut the data tends to indicate that there is something there. Clearly more work needs to be done, but we were never able to get back to this issue. On the other hand, lots of people have strong opinions on the subject based upon a lot less data.
This is explained in more detail in this post from 2015: