On RAND’s Recent Report

by David 

Recently, the RAND Corporation released the Third Edition of their “Science of Gun Policy” report. Essentially, the report sets criteria for what studies they choose to analyze, calculates their effect sizes (using incident rate ratios and confidence intervals). Then, based on the direction and size of the effect they judge whether the evidence for a relationship between a given law and a crime is “inconclusive,” “limited,” “moderate,” or “supportive.” 

 

In this post, I want to focus on their evidence and conclusions they present in Chapter 18, their section on concealed carry laws. They present 12 studies on the effect of shall-issue or permitless carry vs. may- or no-issue laws. Only 3 studies find a homicide-increasing effect (Fridel (2021), French and Heagerty (2008), and Doucette, Crifasi, and Frattaroli (2019)); all the other studies find no significant effect (e.g., the CI crosses 1.00). Interestingly, despite 75% of the studies evaluated showing null effects, RAND chose to brand the evidence regarding the relationship between shall-issue laws and firearm and total homicide rates as “supportive” that such laws increase them.

 

 

There have been dozens and dozens of studies analyzing the effect of RTC laws on homicide rates. Before 2008, almost all studies (at least those done by economists and criminologists) published in refereed journals found either null or homicide-decreasing effects. More recently, it seems like more and more studies are finding homicide increasing effects. But, some of those studies have odd findings and fail to explain the plausible mechanisms whereby a RTC law might increase homicide. Nevertheless, RAND decided that these studies did not meet their methodological criteria, and thus chose not to include them. Let’s analyze the studies they chose to include. 

 

 

The first study, Fridel 2021, has already been critiqued by Dr. Gary Kleck, Professor Emeritus of Criminology and Criminal Justice at FSU. He notes that Fridel made several mistakes, which led her to find that RTC laws increase the firearm homicide rate by 10.8% (IRR = 1.108, 95% CI: [1.026, 1.196]). He writes that those mistakes include:

1. Failing to analyze total homicide rates, instead analyzing only firearms homicide rates.

2. Causal order—Failing to do anything to distinguish the effect of gun rates or carry laws on violence from the effect of violence on gun rates and carry laws.

3. Using in a panel study, a measure of gun ownership that has been demonstrated to be invalid for use in panel studies.

4. Failing to control for other gun control laws in trying to isolate the effect of carry laws.

5. Failing to control for other confounders, instead controlling only variables that apparently are not confounders.

She responded to Kleck’s critiques here. Kleck responded back to her critiques here. She responded back here. And so on and so forth. Fridel, like many others who do research on this topic, fails to use the number of permit holders as an independent variable. Instead, she uses dummy variables for states to determine whether they are shall-issue/permitless or may-issue states. William English has shown that some may-issue states such as Massachusetts and Connecticut have a higher % of the adult population who have a CCW permit than many other shall-issue states. Therefore, it shouldn’t be considered appropriate to categorize these states like this. Instead, they should look at permit growth. Fridel also uses a selective time frame, 1991-2016. No explanation is given for this time frame, especially since we have state-level data on firearm homicide going back to at least 1968.  

The last thing I’ll say about this study is the selective reporting of the findings. Fridel’s abstract does not mention the null finding with respect to gun ownership and firearms homicide (IRR=1.046, 95% CI: [0.978, 1.119]), which might have been interesting to highlight. 

The second study that finds a statistically significant homicide-increasing effect is French and Heagerty (2008). This study focuses on trend models (it’s published in a statistics journal) and uses RTC laws as an exercise. RAND says that their study had a “suggestive” effect of a positive relationship between RTC laws and homicide rates. The one that RAND includes in their analysis is the one with the lowest p-value (p=0.059) (and probably the best one, to be honest, since it accounts for state-level heterogeneity in the law effect). The authors also conducted a random effects meta-analysis looking how RTC laws affected changes within states. Their figure, Figure 5, shows that some states saw statistically significant increases, some saw decrease, and most saw changes no different from 0. The p-value from the summary is 0.083, which the authors note is in fact not statistically significant. This study does not actually measure permit growth, and so states that have may-issue regimes with high adult permit rates are not included in the study, since it was only trying to measure the effects in states that changed their concealed carry laws. 

The last study is Doucette, Crifasi, and Frattaroli (2019), which looked only at workplace homicides. It is worth noting that workplace homicides do not account for the bulk of homicide in the U.S., not even close. According to the FBI, there was an average of 19,972 murders per year from 1993-1999. In that same time frame, the Bureau of Justice Statistics (BJS) notes that there were 900 workplace homicides per year, approximately 4.5% of nearly 20,000 annual murders (Yes, I know the two are different, but finding the actual homicide statistics would only strengthen the point that workplace homicides account for a small fraction of homicide in America). Therefore, this study only applies to a small fraction of violence in the U.S. The finding that RTC states had a 34% higher rate of workplace homicides (The result RAND relies on) comes from Table 1. The authors present evidence that some of the covariates they use are correlated with workplace firearm homicides, but no evidence that those variables are also correlated with RTC laws. This means we actually have no idea whether they controlled for any true confounders or not. Indeed, just one of their covariates (region) in the model RAND chose to show is statistically related to workplace firearm homicides, which might suggest that only one real confounder was controlled for. Permit-to-purchase laws, stand-your-ground laws, and laws banning the sale of weapons to violent misdemeanants all had no effect on workplace firearm homicides. This might have been interesting to note in the abstract.

In conclusion, I think RAND missed the mark with this subsection of their analysis. Most studies they showed had null effects. The ones that did were either methodologically flawed, very uncertain, or extremely limited in scope. Therefore, I believe there is no justification for the claim that there is “supportive” evidence that RTC laws increase firearm homicides or homicides in general.