Judging Risk

Judging Risk

Risk assessment plays an increasingly pervasive role in criminal justice in the United States at all stages of the process—from policing to pretrial detention, sentencing, corrections, and parole. As efforts to reduce mass incarceration have led to the adoption of risk-assessment tools, critics have begun to ask whether various instruments in use are valid, and whether they might reinforce rather than reduce bias in the criminal justice system. Such questions, however, have largely neglected how decision-makers use risk assessment in practice. In this Article, we explore the judging of risk assessment and why decision-makers so often fail to consistently use quantitative risk assessment tools.

We present the results of a novel set of studies of both judicial decision-making and attitudes towards risk assessment. In our first study, we find that even in Virginia, whose risk assessment instrument has been hailed as a national model for the use of risk assessment, sentencing data indicates that judicial use of risk assessment is highly variable. In our second study, the first comprehensive survey of its kind, we also find quite divided judicial attitudes towards risk assessment in sentencing practice. Even if, in theory, an instrument can better sort offenders in less need of jail or prison, in practice, decision-makers may not use it as intended.

Still more fundamentally, in criminal justice, unlike in other areas of the law, one does not have detailed regulations concerning the use of risk assessment that specify the content of assessment criteria, the peer review process, and standards for judicial review. We make recommendations for how to better convey risk assessment information to judges and other decision-makers, and how to structure that decision-making based on common assumptions and goals. We argue that judges and lawmakers must revisit the use of risk assessment in practice. We conclude by setting out a roadmap for the use of risk information in criminal justice. Unless judges and lawmakers regulate the judging of risk assessment, the risk-assessment revolution in criminal justice will not succeed in addressing mass incarceration.




America is experiencing a risk-assessment revolution in criminal justice. The role of risk assessment is increasingly prominent at all stages of the criminal justice system, including policing, pretrial detention, sentencing, corrections, and parole.[1] The assessment of offender risk was a central feature of criminal sentencing prior to the mid-1970s, when jurisdictions throughout the United States began rejecting forward-looking risk assessment in favor of backward-looking retributive concerns. As jurisdictions have reconsidered the use of incarceration, risk assessment has returned in new and more sophisticated ways.[2] Efforts to reduce mass incarceration have led to the adoption of risk assessment tools as an alternative to money bail, as a way to determine the length of incarceration and the intensity of probation or parole supervision, and as a tool to decide which mentally ill or substance-abusing offenders might safely be diverted to treatment in the community. For example, in its 2017 revision, the Model Penal Code prominently endorsed consideration of risk.[3] Similarly, the First Step Act, perhaps the most far-reaching federal sentencing reform in a generation, mentions risk or risk assessment no less than one hundred times, and relies on risk assessments to allocate prison programming and determine prisoner release.[4]

As risk-based approaches have multiplied, a growing number of critics have asked whether certain risk instruments are predictively valid and whether they might reinforce rather than reduce bias, including racial bias, in criminal justice outcomes.[5] We argue here that the discussion about the role of risk assessment has neglected a separate and fundamental question: how do judges and others use risk assessment in practice?

That question now has a constitutional dimension. The Fifth Circuit affirmed a federal judge’s order that the cash-bail system in Harris County, Texas, violated the Due Process Clause because it adopted a “flawed procedural framework” in which bail decisions by individual judges were arbitrary in practice.[6] The federal judge found that Harris County judges departed from release recommendations made by the pretrial services department as much as 66 percent of the time.[7] The federal judge has now entered a detailed final consent decree in the litigation.[8] Additional litigation is pending in other circuits, and the Harris County litigation may provide both a constitutional roadmap and a model for remedies.[9] In California, lawmakers enacted noteworthy legislation barring cash bail, but permitting local courts and judges to exercise discretion regarding what policies to adopt.[10]

In light of litigation and innovation across a wide range of jurisdictions, a model for the regulation of criminal justice risk assessment is urgently needed. Because the federal First Step Act mandates periodic risk assessment for virtually all federal prisoners, much attention will be paid to how federal prison staff implement risk assessments pursuant to the Act and exercise their discretion.[11] However, in criminal justice, unlike in other areas of the law, one fundamentally does not have regulations concerning the use of risk assessment, such as rules specifying assessment criteria, the peer review process, and standards of judicial review.[12] A model for the regulation of criminal justice risk assessment is urgently needed.

In this Article, we present empirical evidence on how judges actually use (and fail to use) risk assessment instruments. States have made the use of these instruments advisory rather than presumptive or mandatory, and as a result, the discretion of the decision-maker plays an important role.[13] Nonetheless, very little information is available about how judges actually use these risk assessments.[14] To address this question, we conducted a set of studies of judicial decision-making. We focused on Virginia because, in the words of the Model Penal Code, “[o]n risk assessment as a prison-diversion tool, Virginia has been the leading innovator among American states.”[15] Virginia was the first state to incorporate risk assessment into its sentencing guidelines to permit alternative sentences for “lowest risk” drug and property offenders.[16] We found that in 2016, of the entire population of offenders who qualified for the use of the Nonviolent Risk Assessment (NVRA), 46 percent scored in the category of low risk offenders, and were therefore eligible for an alternative sentence.[17] Of those, 42 percent did receive an alternative sentence and 58 percent did not.[18] We observed extremely wide variation between both individual judges and entire judicial circuits in their use of the NVRA.

Second, we surveyed judges in Virginia and found highly divergent attitudes towards risk assessment. We found that a strong majority of judges endorsed the principle that sentencing eligible offenders should include consideration of recidivism risk. However, a strong majority also reported that in their jurisdictions, they lack adequate alternatives to imprisonment. A sizable minority of judges had great discomfort with the goals and the use of risk assessment at sentencing. Some described risk assessment as just “another tool that aids but does not supplant judicial judgment.”[19] Others expressed extreme distaste for risk assessment. For example, one judge stated, “Frankly, I pay very little attention to the [risk assessment] worksheets . . . I also don’t go to psychics.”[20] That some judges were not fully cognizant of the availability of risk assessment in sentencing was also unsurprising, given the almost complete lack of judicial training on the subject.[21]

These studies of judicial practice and opinion concerning risk assessment produced several important insights into how to better institutionalize use of the risk assessment. To change behavior, it is not enough to adopt a technical tool—attitudes towards the use of risk assessment in decision-making need to be addressed if the tool is to be used well.[22] A new approach is needed that takes account of the interface between general quantitative risk information and the officials, such as judges, prosecutors, and probation officers, who take that information into account in decision-making. That interface must be evidence-informed and based on common goals. In this Article, we provide a roadmap of three such goals: (1) presenting risk information in a more comprehensible way to decision-makers; (2) structuring decision-making to better make use of that information; and (3) accompanying these reforms with ongoing monitoring, through judicial review and independent scientific peer review of data to assess on-the-ground use of risk assessment.

In Part I of this Article, we present an overview describing the growing use of risk assessment instruments in criminal justice, including use encouraged by the development of empirically validated quantitative tools to assess risk. We explain what risk assessment is, how it can be used and in what criminal contexts, the types of risk instruments used, and approaches adopted in recent years. Second, we provide an overview of the reasons why risk assessment has become attractive to so many lawmakers and policy-makers in recent years and concerns critics have raised about the design of instruments. Third, we describe a wide variety of risk assessment tools in use, ranging in complexity from simple checklists or scorecards to algorithms.

In Part II, we present three results of the studies in Virginia: (1) the use of nonviolent risk assessment; (2) the results of a novel survey of judges; and (3) the results of an analysis focusing on local treatment resources. We found highly inconsistent uses of risk assessment and highly divergent attitudes toward risk among judges. The local availability of behavioral health resources is substantially correlated with variation in the use of alternative sentencing.

In Part III, we turn to the experience with risk assessment in other states. That evidence similarly suggests the challenge of integrating risk assessment into judicial and other official decision-making.

In Part IV, we develop a model for how to better regulate and integrate risk assessment into criminal justice decision-making. First, we draw from administrative procedures used in other fields that regulate risk management using accepted criteria, with a public approval process, validation, and ongoing review. Second, we draw from research on decision-making regarding risk and communication of risk. We suggest that risk assessment information should be better explained, in its training and in its presentation, to be understandable in the context in which judges and other criminal justice officials work. Third, we discuss solutions aimed at better structuring the manner in which risk information is incorporated into decisions. Fourth, we recommend ongoing validation not just of risk assessment instruments, but also of their use in practice by the decision-makers. We conclude by offering a six-part roadmap for how to regulate the use of risk assessment in the criminal justice system.

The need to get risk assessment right has never been more pressing. Every year, upwards of ten million people are arrested in the United States.[23] Judges must decide whether to jail people pending trial. Every year, millions of people are convicted in the United States. Incarceration, which had reached record levels, began to modestly decline starting in 2008, but so far we have halted growth rather than substantially reduced the prison population.[24] Efforts to reduce reliance on incarceration have been bipartisan, and have largely focused                                                                                                                                                                                                                                        on efforts to divert offenders, including less risky offenders, from jail and prison.[25] That makes the task of accurately identifying such offenders all the more important. At the same time, mounting criticism has focused on the lack of transparency and potential for bias in the design of risk instruments.[26] Those concerns should extend to how officials use risk assessment in practice. It is not enough to adopt a risk-based tool. To change decision-making, we need to address policy, structure of decision-making, and training. Coherent regulation is needed.

I. The Second Coming of Risk Assessment in Criminal Justice

There has been a movement across many fields of regulation and business to use risk assessment and cost-benefit analysis tools to improve rules and outcomes.[27] Those methods can both involve efforts to measure the frequency and likelihood of adverse outcomes, as well as their severity or cost.[28] Such methods are an everyday aspect of business management, and they are integral to entire fields like insurance and finance, in which there has been longstanding debate regarding the goals of regulation and public policy that affect how risk should be distributed in society.[29] Risk-based methods have been widely incorporated into federal administrative policy-making, but they have had an uneven history in criminal justice.

Beginning in the early twentieth century, risk assessment tools were adopted in criminal justice in the United States. In the 1970s, those experiments with the use of quantitative tools to evaluate criminal cases were largely neglected, as criminal justice focused more on retributive approaches to punishment. The largely retributive approach has changed in the past decade and a half—policy-makers turned back towards rehabilitation—and more quantitative research on recidivism began to inform policy.[30] We are in the midst of a risk assessment revolution in criminal justice. In the pretrial context, New Jersey and Kentucky, for example, along with many local jurisdictions, use the Public Safety Assessment (PSA) developed by Arnold Ventures.[31] State Supreme Courts, such as the Indiana Supreme Court, the Kentucky Supreme Court, the Nebraska Judicial Council, and the New Jersey Supreme Court, have ordered studies or sweeping changes.[32] States are using risk-based instruments to assess conditions of confinement as well.[33] That said, in criminal justice, unlike in other fields, one typically does not have regulations concerning use of risk assessment. Instead, the process has often been ad hoc and has involved conveying risk information to judges and other decision-makers who have retained their traditional discretion. In the sections that follow, we describe three issues: (1) the traditional use of risk assessment in criminal justice; (2) the recent rise in the use of more empirically-informed risk assessment instruments in a variety of criminal justice settings; and (3) the debates regarding the value of risk assessment in criminal justice.

A. The Traditional Use of Risk Assessment

The most widely used definition of risk assessment describes it as the process of using risk factors to estimate the likelihood (i.e., probability) of an outcome occurring in a population.[34] “Risk factors” are simply variables that (1) statistically correlate with recidivism, and (2) precede recidivism in time. In the case of pretrial decision-making, the relevant population consists of persons facing criminal charges. In the case of sentencing, the relevant population consists of convicted offenders.

In the area of pretrial detention, traditionally the focus in the United States was to impose bail to prevent flight and ensure that the defendant was present in court. Beginning in the 1970s, the focus on preventing the commission of additional crimes increased.[35] The 1984 Federal Bail Reform Act and state laws adopted in almost every state changed that to focus on the risk assessment of a defendant’s failure to appear in court or commission of a new crime, but these laws asked judges to predict failure to appear or commission of new crime without the benefit of empirical evidence.[36] The laws typically defined “dangerousness” extremely broadly, giving judges substantial discretion whether to consider defendants dangerous or not.[37] Unsurprisingly, given this virtually unlimited discretion, empirical work has found judges highly variable in making pretrial detention decisions.[38]

Judicial discretion involving an assessment of the likelihood of recidivism was once a feature of indeterminate sentencing. Judges and parole officials shared the assessment of risk. At the time of conviction, the judge would set a sentencing “range” (e.g., three to five years) and later (i.e., within this three-to-five-year range) parole officials would make a determination that a prisoner was sufficiently unlikely to recidivate and that he or she could be released from prison. As indeterminate sentencing gave way to determinate sentencing, the obligation fell increasingly (and later, almost exclusively) on the judge to fix sentence lengths. Risk assessment began to disappear from the practice of sentencing because judges became tasked with assigning backward-looking retributive sentences and not forward-looking sentences based on risk of future crime.[39]

B. The Rise of Modern Risk Assessment

Risk assessments are now commonplace at each stage of the criminal process, from police investigations, pretrial settings, sentencing, corrections, during parole and community supervision, as well as in specialized courts such as juvenile or mental health courts. Risk assessment involves, as noted, a process designed to predict outcomes. What outcome is being predicted may depend on the criminal setting in which it is being used, and the risk may be defined to include types of reoffending or failure to appear, substance abuse, or other outcomes. Risk assessment is distinct from risk management, which tries to reduce risk through supervision or treatment interventions. Further, it matters not just how risk is defined, but for what time frame an outcome is to be predicted. In general, risks increase as time frames increase.

Two types of errors may result whenever predictions are made. The individual, despite the assessment that the person is “low” risk, may commit a crime (or fail to appear in court, or violate probation, etc.) which would constitute a “false negative” prediction. Alternatively, the individual may be assessed as “high” risk, but may not commit the relevant type of violation, which would constitute a “false positive” prediction. False negatives may be particularly salient when, despite a prediction to the contrary, an offender commits a serious crime. In contrast, false positives are hard to detect. If a person is erroneously given a lengthy sentence, they cannot easily show that they would not have reoffended had they been released into the community.

The types of risk assessment tools vary, depending on their designs and the uses to which they are put. There are over four hundred structured risk assessment instruments used around the world.[40] Many are largely interchangeable, however, and involve the same or similar risk factors.[41] The developers of some of these tools claim not only to assess risk, but also to address the rehabilitative “needs” of a person.[42]

Researchers describe how these instruments have evolved from first-generation tools, consisting of clinical judgment and the experience of a decision-maker, to second-generation tools relying on static risk factors (such as criminal history, age, and gender), to third-generation instruments looking at both risks and needs, as well as static and dynamic risk factors such as educational status and employment, and finally to fourth-generation instruments, which provide individualized plans based on assessment of static and dynamic factors. A fifth generation of these tools may use machine learning techniques to predict recidivism in real time, using far more complex analysis. This “generation” terminology should not necessarily be taken to mean that more recent and complex tools necessarily perform better, however.[43]

Jurisdictions have widely experimented with using risk instruments pretrial, as an alternative (or a supplement) to reliance on cash bail. Jails have become increasingly overburdened in the United States, with most of those in jail serving pretrial detention.[44] In response, state supreme courts, such as the Indiana Supreme Court, the Kentucky Supreme Court, the Maryland Supreme Court, the Nebraska Judicial Council, and the New Jersey Supreme Court have ordered studies and changes to pretrial bail decision-making, sometimes accompanied by legislation or changed exclusively through legislation.[45] The pace of change has been rapid. Every state has adopted new pretrial policies, with five hundred enactments from 2012 to 2017, and at least fourteen states have adopted statistical risk assessment pre-trial since 2012.[46] The American Bar Association recommends the use of pretrial risk assessment, as does the National Association of Counties and the Conference of State Court Administrators.[47]

Research has shown that quantitative assessments are more reliable in their predictions than those of individual decision-makers.[48] One study found that 42 percent of people would be released pretrial if New York State used a risk assessment instrument to make decisions concerning pretrial release, rather than bail and subjective judicial assessments.[49] Over two dozen local jurisdictions and the entire state of New Jersey are now using the Public Safety Assessment tool funded by Arnold Ventures.[50] The tool was based on analyses of criminal cases in three hundred American jurisdictions. It is freely available, and it was designed to remove factors associated with racial disparities in pretrial detention, such as arrest history, and instead relies on factors such as conviction history.[51] It relies on static factors, and not on information gleaned from interviews with a subject, because it is designed to be used early on in the criminal process.

Other jurisdictions have relied on risk assessment in sentencing to divert outright certain classes of offenders from criminal punishment. There are many important legal and policy differences between the pretrial and sentencing contexts. In the pretrial context, the question is whether a person will appear in court and whether they might pose a danger of recidivism pretrial. However, while retribution is an (arguably) appropriate concern in sentencing, it is (inarguably) an inappropriate concern pretrial, before a person has been convicted of any crime.

Use of risk assessment during sentencing has increased in recent years. Virginia was an early adopter of this approach.[52] As efforts to reduce incarceration have become more prominent in the states, almost half of the states now use risk-based instruments at sentencing, at least in some types of cases.[53] State supreme courts have approved the use of risk assessment in sentencing.[54] Some states, such as Kentucky, Ohio, Oklahoma, Pennsylvania, and Washington, have required judges to consider risk assessment during sentencing.[55] A 2007 National Center for State Courts report encouraged this movement towards empirically informed sentencing approaches.[56] As part of the adoption of realignment or Justice Reinvestment Initiative legislation, states have increased the use of alternatives to incarceration. In addition, a number of states have enacted bail reform legislation. In California, realignment legislation requiring the reduction of statewide prison populations in order to comply with prison-crowding-related court rulings led to the use of county-specific plans that reduced imprisonment quite dramatically, including through the use of a range of different risk-assessment tools.[57] In New York, lawmakers enacted statewide bail reform legislation that went into effect on January 1, 2020.[58]

The juvenile justice system has similarly seen a shift towards the use of risk assessment instruments to identify high-risk offenders for greater sanctions or for rehabilitation. While in 1990, only one-third of state juvenile justice systems used risk assessments, by 2003, 86 percent used risk assessments.[59] While more work has been done on the validity of risk instruments used to predict violence for adults, a systematic review of instruments used in the juvenile setting suggested, on average, the same predictive ability as in adult settings.[60]

In the area of policing, police officers have long made predictions regarding neighborhoods and individuals that may be more likely to engage in criminal activity, and agencies can deploy resources based on those assessments of risk. Increasingly, police departments have used data analysis to inform predictive decision-making.[61]

Still other jurisdictions focus on risk assessment in the probation stage to better utilize community corrections, including as an alternative to incarceration. The National Institute of Corrections has described how its recommended evidence-informed approach uses risk assessments.[62] Similarly, the federal probation office and many state probation agencies use risk-based instruments to make probation decisions.[63] Probation and reentry generally is an area where risk assessment similarly plays a far greater role than it did in the past.

C. Risk Assessment Debates

Risk assessment in criminal justice has never been more widespread and never more debated among policy-makers and scholars. The Model Penal Code, as revised by the American Law Institute (ALI) in 2017, encourages the use of “actuarial instruments or processes to identify offenders who present an unusually low risk to public safety.”[64] One reason for this endorsement is the following:

In virtually every decision-making situation for which the issue has been studied, it has been found that statistically developed predictive devices outperform human judgments . . . This is one of the best-established facts in the decision-making literature, and to find otherwise in criminal justice settings would be surprising (at best) and suspicious or very likely wrong (at worst).[65]

The Model Penal Code recommends sentencing commissions “develop actuarial instruments or processes, supported by current and ongoing research, that will estimate the relative risks that individual offenders pose to public safety” to be incorporated into sentencing guidelines.[66] The Code also calls for needs assessments to match offenders with rehabilitative interventions.[67]

1. Penal Theory

As noted above, there are particularly serious questions regarding theory of criminal punishment in the sentencing context (as opposed to when risk assessment is used during pretrial proceedings, incarceration, or community supervision). The ALI view of risk assessment in sentencing reflects a hybrid approach towards criminal punishment, incorporating both retributive and utilitarian approaches. Broadly put, the retributive approach focuses backwards on an offender’s moral culpability for crimes committed in the past, while the utilitarian approach focuses forwards on deterring future criminal acts by the offender or other would-be offenders (or by incapacitating the offender). Retributivists believe that an offender’s moral culpability for past criminal acts should be the sole consideration in determining punishment.[68] Offenders should be punished “because they deserve it, and the severity of their punishment should be proportional to their degree of blameworthiness.”[69] Those who follow a utilitarian approach believe punishment is justified principally by its ability to decrease future criminal acts by the offender or to deter other would‐be offenders.[70]

Many scholars argue that any workable theory of criminal punishment should address both retributive and utilitarian concerns.[71] In an influential hybrid approach, Norval Morris developed a theory of “limiting retributivism” in which retributive principles may set an upper (and perhaps also a lower) limit on the severity of punishment, but within that range, utilitarian concerns can be taken into account.[72] The Model Penal Code approach reflects this hybrid view, in which risk assessment does not entirely define sentencing, but in which risk assessment can be used to identify low-risk offenders eligible for reduced sentences.[73]

Those who share a purely retributive approach towards criminal punishment would object to the use of risk assessment (or any other utilitarian considerations) in criminal justice. The use of risk assessment in sentencing involves a normative choice: to make decisions regarding predicted outcomes. As a result, scholars, criminal justice policy-makers, and decision-makers, like judges, may not share that normative preference. Some, such as Bernard Harcourt, have objected to the use of prediction generally on grounds that justice should not be based on future outcomes, explaining that risk assessment is not compatible with traditional theories of just punishment.[74] Our focus here is on critics who have raised a series of concerns that operate within the assumption that some hybrid of retributivism and utilitarianism are acceptable goals of criminal punishment. However, we want to highlight that judges and others who reject the use of risk assessment may do so for some of the same reasons that scholars may criticize it: they view risk assessment, correctly, as incompatible with a wholly retributivist view of criminal punishment.

2. Predictive Accuracy

Another set of concerns has to do with the quality of the predictions that risk assessment instruments provide. For example, it is far more challenging to predict uncommon violent offenses (e.g., homicide) than to predict common property offenses, for which there is far more data. An additional concern is with the time period during which the data relied upon are collected. The Model Penal Code recommends that risk assessment tools be periodically reviewed for their reliability.[75] Indeed, many risk assessment instruments used by parole boards, for example, have never been independently validated, have not been validated on in-state populations, are not regularly updated, and are sometimes altered to add additional factors that are not part of the original instrument.[76]

3. Inequality

Others argue that policy-makers have failed to attend to the real distributional consequences of risk assessment instruments. Expressing that view, as noted in the Introduction, then-Attorney General Eric Holder questioned the use of risk assessment as potentially causing “fundamental unfairness.”[77] The concepts of fairness and accuracy as used in the risk assessment context may themselves need to be clarified and can create difficult trade-offs.[78] However, a central and important fairness concern is that risk assessment can result in racial disparities or disparities based on other invidious criteria.[79] For example, Pennsylvania considered a sentencing approach in which rural offenders were given fewer points in their risk scores than urban offenders,[80] an approach that would have strongly correlated with race, and that was for good reason rejected in 2017.[81]

Gender is a factor explicitly taken into account by some risk assessment instruments. Sonja Starr has argued that doing so violates the Equal Protection Clause because under intermediate scrutiny, such assessments use a gender classification without a substantial state interest to justify doing so.[82] Others have responded that use of gender as a factor in discretionary risk assessment does not constitute a classification, and does not raise constitutional concerns, because (1) gender is empirically a highly predictive risk factor for many types of offenses, and (2) avoiding gender as a risk factor for recidivism works to the disadvantage of female offenders.[83]

4. Transparency

A separate concern is with the transparency of the risk assessment instruments used. Certain private companies have marketed software without making public the factors relied upon in complex algorithms; one such algorithm, COMPAS, marketed by Northpointe, apparently relies on socioeconomic and family factors.[84] Critics argue that, although race is of course not a static factor in the algorithm, these dynamic factors likely correlate closely with race.[85] A ProPublica report called the COMPAS software “biased against blacks.”[86] In the widely discussed Loomis case, the Wisconsin Supreme Court rejected an appeal by a person who objected, at his sentencing, that he could not review the basis for the Northpointe software’s risk assessment, nor whether invidious factors or factors that have a disparate impact based on invidious criteria were relied upon.[87] The U.S. Supreme Court ultimately denied certiorari in the case, but the case raised a serious due process concern that is likely to be litigated in future years.

5. Efficacy

Others fear that using risk assessment is too incremental an approach towards the problem of mass incarceration and that more forceful interventions are needed. Thus, Jessica M. Eaglin has questioned the use of risk assessment, arguing that evidence-based approaches are not the right way to reduce mass incarceration.[88] To date, risk assessment has not been used to reduce incarceration to the extent some advocates prefer.

6. Implementation

The question that has been least explored, in our view, in the literature is how judges use risk assessment. Critics have assumed that instruments, whether desirable or not, are uniformly applied by judges.[89] The Model Penal Code has recommended advisory use of risk assessment.[90] Where that is the case, the defendant can challenge the findings of an assessment “in open court,” and can “contest any adverse findings.”[91] However, research suggests that decision-makers do not always accurately perceive risk and that whether risk estimates are actually used may depend on the manner and format with which risk information is communicated.[92] Whether judges appropriately follow the advisory or even presumptive recommendations concerning risk assessment has been little studied in the literature. That is the problem we turn to in the next Part.

II. Empirical Studies of Risk Assessment in Virginia

If risk assessment is desirable because, “[i]n virtually every decision-making situation for which the issue has been studied, it has been found that statistically developed predictive devices outperform human judgment,” then the question remains how well human decision-makers actually use their judgment to incorporate the information from risk assessment.[93] That question has not been carefully examined in the past. Now that risk assessments are so commonly used by legal decision-makers, it is a question that can and should be examined. We examine that question by focusing first on the experience in Virginia with risk assessment in sentencing, based on several studies that we conducted, and then we turn to evidence from other jurisdictions concerning the use of risk assessment instruments.

A. The Virginia “Nonviolent Risk Assessment” Instrument

The Virginia model for the use of risk assessment in sentencing is important to study because Virginia was the first state to make risk assessment a formal part of sentencing. As a result, the Virginia model has received a great deal of attention from policy-makers, judges, and scholars, including, as noted, the American Law Institute in the revised Model Penal Code.[94] The use of risk assessment in Virginia has been hailed as a “valuable test” case and a model for “ratchet-down” approaches in sentencing that reduce prison populations: it identifies low-risk individuals.[95] The Virginia risk instrument has been lauded for its substance and for the process used to develop it. The instrument is transparent, publicly available, and validated within the state, and more recently revalidated.[96] Virginia’s instrument is simple and easy to understand. It involves “placing risk discretion in the sentencing courtroom.”[97] We sought to assess that discretion: how is this risk instrument used by judges in practice?

In 1994, the Virginia Legislature adopted the so-called truth-in-sentencing legislation and abolished parole in the state. To avert a resulting fiscal “collapse”[98] of the state’s prison system, the legislature simultaneously adopted risk assessment “to reduce the use of incarceration for nonviolent criminals, in order to offset the increased prison stays for violent offenders that were built into the original Virginia guidelines.”[99] The Legislature directed the newly formed Virginia Criminal Sentencing Commission (VCSC) to develop an empirically based risk assessment instrument.[100] The goal lawmakers set out for the VCSC was to divert 25 percent of the “lowest-risk, incarceration-bound, drug and property offenders” from prison to alternative sanctions such as jail, probation, community service, outpatient substance-abuse or mental health treatment, or electronic monitoring.[101] In the words of Richard Kern, the first Director of the VCSC, among the “main goals of the 1994 sentencing reforms” was to “expand alternative punishment/treatment options for some non-violent felons” by adopting statistical instruments “to divert low risk offenders” from prison.[102]

The VCSC developed the instrument by 1996; it was implemented at pilot sites in 1997.[103] The risk assessment instrument is administered only to offenders for whom the state’s sentencing guidelines recommend incarceration in prison or jail. In addition, offenders must meet certain eligibility criteria (e.g., a criminal history of only nonviolent offenses). If the offender’s total score on the instrument is below a given cut-off, he or she is recommended for an alternative, community-based sanction. If the offender’s score on the instrument is above that cut-off, the prison or jail term recommended by the sentencing guidelines remains in effect. The National Center for State Courts (NCSC) evaluated the use of the instrument at the pilot sites from 1998 to 2001.[104] The VCSC then conducted a validation study of the instrument to prepare it for state-wide use; the goal was to designate as lower-risk the group of individuals with an approximately 85 percent likelihood of not having a felony conviction within three years of release from confinement.[105]

The instrument was adopted statewide in 2002, for all felony larceny, fraud, and drug cases.[106] Thus, in 2002, the Nonviolent Risk Assessment Instrument (NVRA) was included as one of the sentencing worksheets to be completed for all eligible offenders convicted of one of four drug and property crimes—Larceny, Fraud, Drug Schedule I/II (e.g., possession of cocaine), and Drug/Other (i.e., marijuana distribution). In 2012, the Commission re-validated its instruments on large samples of eligible drug and larceny/fraud offenders.[107] Recidivism was defined, for those purposes, as a reconviction for any felony offense within three years of release. The Commission found sizable differences between those designated as lower and those designated as higher risk offenders; 12 percent of lower risk drug offenders recidivated, as compared with 44 percent of higher-risk drug offenders, for example.[108]

The NVRA instruments for each offense examine only the following static factors: (1) offender age at the time of the offense; (2) gender; (3) prior adult felony convictions; (4) prior adult incarcerations; (5) whether the person was legally restrained at the time of the offense[109]; and additionally, for drug offenses, (1) prior juvenile adjudication, and (2) prior arrest or confinement within the past twelve months (as opposed to legal restraint) at the time of the offense.[110] If the offender’s total score on the instrument is below the cut-off, the offender is recommended for an alternative sanction. If the offender’s score on the instrument is above the cut-off, the prison or jail term recommended by the sentencing guidelines remains in effect.

The NVRA instrument was adopted to be used alongside the sentencing guidelines in 1994, and therefore, the use of the NVRA is not considered a “departure” from the sentencing guidelines. Rather, an alternative sentence, when imposed after using the NVRA, is considered to be in compliance with the guidelines. After the NVRA is completed, judges have discretion whether to follow the recommendation for an alternative sentence. Judges also have discretion regarding which alternative sentence, if any, to impose.

We aimed to study whether the nonviolent risk assessment instrument is being used to divert from prison 25 percent of the “lowest-risk, incarceration-bound, drug and property offenders,” and if so, what types of alternative sentences they receive.[111]

B. Judicial Reliance on the Nonviolent Risk Assessment Instrument

First, we reviewed fiscal year (FY) 2016 sentencing data the VCSC shared with us concerning the use of the NVRA instrument.[112] The use of this instrument is important in a large number of cases; over 8,000 people were convicted of eligible offenses in Virginia in FY 2016. In that year, the Commission received a total of 23,713 sentencing guidelines worksheets.[113] Thus, one quarter of felony convicts were convicted of crimes that were eligible for the use of the NVRA. Of those, 6,787 people were eligible offenders for whom a risk assessment form was received. Over a thousand additional offenders were eligible offenders for whom a risk assessment form was either not filled out or not shared by the judge with the VCSC. The table below displays our analysis of receipt of alternative sentences in fiscal year 2016.

Table 1: NVRA-Eligible Offenders Who Received an Alternative Sanction