New Work Recommends Standards for Law Scholarly Impact Analyses

An excellent new article by Karen L. Wallace & Rebecca Lutkenhaus, Professors of Law Librarianship at Drake University Law School takes a deep dive into the practice and implications of ranking law faculty scholarly impact.  In their article, “Measuring Scholarly Impact in Law,” forthcoming in the Widener Law Review, the authors argue that “although the U.S. News [scholarly impact ranking] proposal died, legal bibliometric studies will persist and the academy should develop standards for the responsible creation and use of scholarly impact analyses.”

They examine the dangers associated with rankings based on citation metrics where there is a lack of clarity on what is being measured and why.  Rather, such measures should be based on principled, evidence-based criteria adapted to fit the unique aspects of legal scholarship.

The authors then consider the applicability of a UK-based, multidisciplinary framework for responsible metric use based on the following five dimensions:

  • Robustness: basing metrics on the best possible data in terms of accuracy and scope;
  • Humility: recognising that quantitative evaluation should support—but not supplant—qualitative, expert assessment;
  • Transparency: keeping data collection and analytical processes open and transparent, so that those being evaluated can test and verify the results;
  • Diversity: accounting for variation by field, and using a range of indicators to reflect and support a plurality of research and researcher career paths across the system;
  • Reflexivity: recognising and anticipating the systemic and potential effects of indicators, and updating them in response.

The authors conclude with a set of recommendations for the legal academy:

  1. To thoughtfully consider what kinds of legal scholarship it wants to encourage and to discuss and establish bibliometric norms and create guidelines that explain how to use scholarly impact metrics in a responsible, productive manner. “Ignoring this work leaves a vacuum for corporate interests or a few vocal scholars to fill.”

  2. To test the principles and assumptions associated with bibliometric frameworks from other disciplines to see how, or whether, they apply to legal scholarship. 

  3. To respond quickly to potential changes to U.S. News methodology to prevent the redirection of resources in potentially wasteful ways that do not serve the best interests of students. 
  4. To encourage the widespread adoption of ORCID iDs and Digital Object Identifiers (DOIs) to maximize both precision and recall when matching citing and cited publications.

This article is an important contribution to the literature on law faculty scholarly impact.  Wallace and Lutkenhaus provide valuable insights on the shortcomings of existing rankings and offer important recommendations for the legal academy.  Their contribution is another excellent example of how law librarians can play a leading role on the issue of scholarly impact and visibility at our institutions and in the academy.