Policy ForumResearch Ethics and Law

Overcoming obstacles to experiments in legal practice

See allHide authors and affiliations

Science  06 Mar 2020:
Vol. 367, Issue 6482, pp. 1078-1080
DOI: 10.1126/science.aay3005

The importance of evidence-based policy rooted in experimental methods is increasingly recognized, from the Oregon Medicaid experiment to the efforts to address global poverty that were awarded a 2019 Nobel Prize. Over the past several decades, there have been attempts to extend this scientific approach to legal systems and practice. Yet, despite progress in empirical legal studies and experimental social policy research, judges, lawyers, and legal services providers often fail to subject their own practices to empirical study or to be guided by empirical data, with a particular aversion to randomized controlled trials (RCTs) (1). This is troubling, as many questions fundamental to legal practice and those it affects, such as allocation of attorney services, bail decisions, and use of mandatory mediation, could and should be informed by a rigorous evidentiary foundation. Although there are practical obstacles to undertaking legal practice RCTs, they have also been stymied by cultural barriers within the legal profession. Whereas medical practitioners are expected to subject their practices to randomized study and rely on the data produced, the default in legal practice is to rely on experience, common wisdom, and professional judgment, often in settings in which clients face constraints on their freedom. To address ethical concerns stemming from this cultural difference, we draw on lessons from biomedical and policy research, as well as experiences of the Access to Justice Lab (A2JL), a leading institution promoting the use of RCTs to inform legal practice.


Embedded Image
PHOTO: ICEDMOCHA/SHUTTERSTOCK

RCTs in law face funding challenges, a dearth of lawyers trained in experimental methods, and blanket roll-out of new laws and legal interventions across entire populations in a manner that often precludes randomization. But the most fundamental challenges are ethical. When field RCTs of legal practice interventions have been proposed, practitioners have sometimes raised concerns about violating both perceived professional obligations and the individual rights of research subjects. Institutional review boards (IRBs), responsible for assessing the permissibility of research, also sometimes exhibit difficulty applying familiar research ethics concepts to nonmedical settings, a phenomenon exhibited toward social science research more broadly. Ethical objections have posed real barriers to A2JL's work (see the table), which have occasionally been resolved through education and negotiation, but have more often led to abandonment of important, ethical studies.

There have been growing calls to systematize and expand efforts to subject legislative and regulatory interventions to randomized experimentation (2), as well as increasing attention to the ethics of policy experiments more broadly (36). We build on this work, but because our goal is to highlight objections rooted in concerns over randomization in lieu of the individual judgments of legal professionals, the principles of biomedical research ethics are the best starting point for potential solutions. The medical profession has been able to manage many ethical concerns raised by RCTs and to create an evidence-based culture, a portent of what the legal profession could achieve. The ethics of clinical research have also developed in a way that addresses the traditional fiduciary obligations of physicians, whereas perspectives regarding the obligations of legal practitioners seem to be generating erroneous resistance to legal practice RCTs.

Although there are differences between medical and legal settings, including the role of democratic authority, protected legal rights, and the unavoidably coercive nature of the criminal justice system, many ethical challenges facing field RCTs of legal practice interventions, including those related to randomization and consent, are familiar in biomedical research. Therefore, we seek to apply lessons from that context to law—and to identify novel ethical considerations where further attention is warranted. Research ethics regulations will not always apply to legal practice RCTs, depending on institutions and funding. Nonetheless, public trust, researcher obligations, and pragmatic considerations all demand that ethical principles be both clarified and upheld.

Evidence-Based Legal Practice

The core idea of evidence-based medicine is that because few medical interventions are so obviously safe and effective that they can be adopted without testing, patient care should be guided by scientific evidence. Although observational studies and historical controls can be superior to anecdote, randomization and contemporaneous controls help minimize confounders and support causal conclusions. As a result, RCTs are rightly regarded by the medical profession as essential to quality care, albeit with their own methodological constraints.

These same concepts apply to legal practice but have not been widely endorsed by the legal profession, even when stakes are high, with implications for freedom (e.g., incarceration), safety (e.g., protection from domestic violence), and access to resources essential to fundamental human needs (e.g., legal assistance in securing government benefits). Although legal practitioners may rely on certain guidelines when available, these are also often not evidence-based, being instead informed by politics or the subjective collective wisdom of professional authorities. Even when evidence is available from one of the few dozen legal practice RCTs conducted in the United States to date, and even when that evidence fails to support the use of interventions that are costlier than their alternatives, it has not been used to change practice (1). When aspects of legal practice are rooted in philosophical and normative claims about rights and interests, empirical study may have little to add. But when interventions aim to achieve certain instrumental outcomes, as is often the case, evidence is critical (1, 2).

Examples of legal practice research questions, RCT designs, objections, and replies

Randomization

One commonly articulated objection to legal practice RCTs is that it is unethical to randomize legal interventions in a way that temporarily precludes individualized professional judgment. For example, in objecting to a proposed A2JL study in which tenants facing eviction litigation would be randomized to participate in a mandatory pretrial settlement conference or proceed without one, local legal services providers argued on the basis of their anecdotal experience that settlement conferences assist in keeping low-income tenants in their homes, despite reason to question that assumption. Similarly, organizations running charitable bail funds have refused to discuss RCTs, even when they lack sufficient resources to assist all eligible arrestees, arguing that because they “already know” the difference it makes to have bail posted, it is wrong to randomize “lifesaving” legal “treatments.”

These examples are indicative, in part, of the legal profession's lagging understanding of foundational principles of research design and ethics. Similar objections have long been considered resolved in the realm of clinical research, where one might also be concerned that randomized participants could be harmed by the denial of some perceived benefit in a way that violates the physician's professional obligation to provide individualized care (7). However, an RCT can be ethical—and perhaps ethically mandatory—when there is sufficient professional uncertainty, or “equipoise,” regarding the safety or efficacy of the experimental intervention compared to the standard of care. In these cases, randomization violates no professional obligation because it will not result in practitioners knowingly exposing participants to harm.

In addition to the culture shift needed in law to adopt this view, it is also essential to challenge faulty moral intuitions exhibited in the legal profession and elsewhere that it is permissible to roll out an untested intervention across an entire population, but not to randomize population groups to rigorously examine the intervention's effect (8). Moreover, because resolving extant uncertainty is essential to the professional's fiduciary obligation to provide the best care or services, it would be logically incoherent to refuse to subject unproven approaches to study unless or until evidence is produced to challenge their effectiveness.

Even when there is relative certainty as to which intervention to prefer in ideal circumstances, practical constraints may render that intervention neither attainable nor sustainable for some or all of the relevant population (5, 9). In such cases, researchers should work to resolve “policy equipoise” (5), developing evidence needed to inform open questions, such as how to optimally allocate the intervention and whether a less resource-intensive approach might suffice. Doing this work through RCTs is methodologically ideal and in many cases ethically permissible. However, one further consideration particularly relevant to legal practice interventions is that the ethical permissibility of an RCT that fails to provide all subjects with the preferred intervention depends on the absence of legal or ethical entitlement to that intervention. Although patients may have ethical entitlements, they have few legal entitlements to medical services, at least in the United States. Those affected by legal practice interventions may have stronger claims.

Consider an offer of representation by a lawyer. This is a federal constitutional entitlement in the United States only for serious criminal charges and a few other narrow matters; in these contexts, it would be legally impermissible to withhold the offer from some individuals, even for research purposes. However, there are other circumstances in which an individual may be expected to benefit from an offer of legal representation without being legally entitled to it. In the face of scarcity, a legal clinic could reasonably randomize similarly situated individuals to receive either an offer of traditional representation, an offer of self-help resources only, or no offer at all to research how to maximize benefits across the population. Individuals randomized without entitlement have either not been harmed, because they might not have received the benefit outside the RCT, or have not been unreasonably harmed, because even if they would have received the benefit, they lack a greater claim to it than others facing similar need.

Voluntary Consent

Other objections to legal practice RCTs are based on concerns that RCTs may be inherently coercive. Perceived or actual coerciveness might stem from the involvement of lawyers, who like physicians are often in a position of power compared with prospective subjects; the provision of desired legal services only to study participants (or select participants); or the constraints on freedom present in criminal settings.

Concerns about professional power dynamics can be addressed in part by emphasizing, where true, that participation in research is voluntary and refusal will involve “no penalty or loss of benefits to which the subject is otherwise entitled” (10). Nonetheless, potential subjects may feel pressured when research is the only way to access a desired intervention, a concern that has been a persistent point of confusion in research ethics. This pressure is not coercive absent entitlement to the desired intervention (11), and it is not unduly influential if the risks of participation are reasonable and not so great as to outweigh the benefits of the desired intervention (12). In fact, where research risks are reasonable and there is the prospect of direct benefit to subjects, one might worry less about constraints on voluntary consent because research participation will be objectively reasonable.

This is important because one distinctive feature of legal practice RCTs is the inherent involuntariness of some aspects of the legal system, particularly in the context of criminal justice. Unlike medicine, where there is an expectation of informed consent to justify interventions on a patient's body, individuals who have been arrested, are facing prosecution, or have been convicted have little control over what happens to them. At least partly in recognition of these circumstances, U.S. research regulations demand special protections for individuals involuntarily confined or detained in penal institutions (10), which can also guide research in other legal settings.

To the extent voluntariness concerns persist, further analysis is warranted to evaluate whether securing traditional research consent is a reasonable expectation for all legal practice research interventions. Current norms permit consent waiver only for minimal risk research that would be impractical to conduct if consent were required (10). This would likely cover RCTs assessing mandatory mediation or online dispute resolution, for example, but “minimal risk” is challenging to define and the subject of intense debate in research ethics, especially when evaluating practices that may be part of the existing standard of care, despite remaining unproven (13). And at least some legal practice interventions will pose greater than minimal risk, such as a heightened possibility of incarceration (e.g., for arrest, bail, and sentencing interventions) or inability to access government benefits (e.g., for comparisons of self-help models versus assistance of counsel). The response to concerns about voluntary consent must not be to inhibit legal practice RCTs, however, because without them, potentially harmful legal interventions may be adopted and potentially beneficial interventions may be dismissed without either evidence or consent (8).

Similar questions arise in circumstances in which it may be infeasible to secure individual consent, a challenge faced across medical, policy, and legal practice intervention research. For example, it may not be possible to study an intervention without mandatory application to some group, such as use of a sentencing algorithm to substitute for judicial discretion. Similarly, use of randomized cluster design can sometimes preclude individual consent, such as when different legal services clinics are each randomized to offer only a single approach, removing choice from each clinic's clients unless they seek services elsewhere. It is important to recognize that these interventions could be implemented without consent outside a research context (and sometimes already have been), which may render the inability to secure research consent less ethically concerning (2, 8). Nevertheless, alternative protections may be appropriate. For example, the data collection required for research purposes should not unduly infringe participants' autonomy and privacy rights (6), and research should avoid disproportionately testing interventions designed to justify reducing resources allocated to vulnerable populations (14).

Bystanders

A final concern that may arise for legal practice RCTs, as well as other types of studies, centers on their potential impact on both incidental bystanders and intended thirdparty beneficiaries, an understudied area in research ethics generally (15). For example, research interventions regarding bail decisions might be intended to achieve public safety goals, to advance the interests of arrestees, or both. Where third parties are affected by research but not directly intervened upon, there are further questions of consent. They may not be individually identifiable in advance (e.g., the future victim of an arrestee released on bail due to a study intervention), making their consent impossible. Even if bystanders are identifiable, however, it is unclear that they should be given the authority to withhold consent and preclude research that offers the prospect of direct benefit to research subjects and future individuals like them (e.g., arrestees that may benefit from a bail decision). Yet, to the extent bystanders face research risks, those risks seem worthy of consideration when assessing a study's ethical permissibility. Conversely, although we traditionally consider risks to research subjects, where research interventions are intended to produce social rather than individual benefit, and the intervention does not entail any breach of a subject's rights or entitlements, it is not clear that subjects should be allowed to withhold their consent. Although IRBs are directed to consider benefits that extend beyond study participants when assessing a study's risk-benefit profile, they have not typically considered risks to bystanders. This area deserves further attention in both law and medicine.

Conclusion

Although objections to legal practice RCTs often have been framed in ethical terms related to randomization and consent, the biggest ethical challenge lies in failing to seek reliable answers to guide the legal profession. The profession—and IRBs—should embrace RCTs, while being thoughtful about their methodological limits and devoting further attention to unresolved issues, including which legal interventions should be viewed as entitlements; when research consent should be required; how to address bystander interests; which RCTs should be prioritized; when it may be appropriate to proceed with a promising intervention in the absence of confirmatory evidence; and how to best engage with communities that would be affected by this research.

References and Notes

  1. World Health Organization, Ethical considerations for health policy and systems research (2019); www.who.int/alliance-hpsr/resources/publications/ethical-considerations-hpsr/en/.
  2. 45 C.F.R. 46.116 (2018).
Acknowledgments: Some of Access to Justice Lab's work is supported by Arnold Ventures, The Chan Zuckerberg Initiative, The Charles Koch Foundation, The NOMIS Foundation, and The Robert Wood Johnson Foundation. Support for elements of this article was provided in part by The Pew Charitable Trusts. The opinions, findings, and conclusions or recommendations expressed here do not necessarily reflect the views of any of the entities listed. H.F.L. is funded by the Greenwall Foundation as a Faculty Scholar.
View Abstract

Stay Connected to Science

Subjects

Navigate This Article