On the latest episode of the WI Law in Action podcast from the UW Law Library, host Kris Turner interviewed me, Bonnie Shucha, Associate Dean and Director of the Law Library. I discuss my recent paper, “Representing Law Faculty Scholarly Impact: Strategies for Improving Citation Metrics Accuracy and Promoting Scholarly Visibility,” which I presented at the Yale Citation and the Law Symposium last month. Below are a few excerpts from our discussion.
On the upcoming U.S. News scholarly impact ranking and the representation problem:
I think the most commonly stated concern about the U.S. News ranking is that it relies on law journal citations from HeinOnline, and it’s not expected to include books and interdisciplinary scholarship. So critics argue that the exclusion of scholarship in other formats and disciplines creates an incomplete and unfair representation of law faculty, total output and impact. However, there are others that say that exclusion of this content doesn’t really matter, that it’s immaterial because the relatively few law faculty that focus in these areas are spread across many different law schools. So, in essence, the failure to capture their scholarship essentially washes out over the broad comparison.
On the significant effect that the inclusion of interdisciplinary scholarship would have on the ranking:
What I wanted to do was to find out if the inclusion of interdisciplinary scholarship in a law faculty citation metrics ranking would affect the conclusions that one could draw about law faculty total scholarly impact, and if so, how it would affect it. And it turns out,… it can have a very profound effect for some schools…
University of Wisconsin wasn’t included in the Sisk law journal study, but on the Heald Sichelman study of law journal citations in HeinOnline, it ranked number 79th, which was substantially below its U.S. News ranking. It also wasn’t included in the Ruhl, Vandenberg and Dunaway interdisciplinary study. So I ran those numbers myself using the same methodology that they did and the results were very surprising. I found that UW would have been ranked number one for interdisciplinary scholarship – and not by a small margin. As I stated earlier, the highest-ranked school that was included in their [Ruhl, Vandenberg and Dunaway] study was the University of Minnesota with a weighted citation score of 190 for interdisciplinary scholarship. Wisconsin’s weighted score was 802, more than four times higher…
For better or worse, Sisk, U.S. News and others are going to continue to rank the scholarly impact of law faculty using citation metrics. But what I hope is that my findings encourage those who do produce those rankings to recognize and be upfront about acknowledging that the choice of data source and methodology that they use can dramatically alter the conclusions that can be drawn from those comparisons and that whatever decisions they make can carry heavy consequences, either positive or negative, for some schools.
On strategies for improving citation metrics accuracy and promoting scholarly visibility:
I cover a lot of different strategies in the paper. I sort of took an “everything in the kitchen sink” approach and presented all of the different options available. Some of these strategies require a relatively small amount of time and effort while others are much more time intensive. And there are some that are simply just more effective than others. I think it’s important to note when looking at these strategies is that one best practice does not fit every author and every law school. So I encourage people to sort of look at the full slate of options and then decide what might be right for them.
At the UW, many of our strategies center around mandatory one-on-one librarian meetings with each tenured and tenured track faculty member to employ a lot of the strategies that are described in the paper. We call these meetings “scholarly wellness checks.”
On the most important things that you can do to increase scholarly visibility:
The most important thing for faculty visibility is just to get it out there, make sure that the faculty scholarship is available online. So if copyright allows, get that faculty scholarship posted to SSRN or in your institutional repository where readers can find it and then hopefully eventually cite it. And then second, what we were just talking about, make sure that it’s listed correctly. This is especially important as far as the U.S. News ranking is concerned because it’s going to be drawing on the HeinOnline content. So you absolutely want to make sure that the author profile in HeinOnline is correct.
On how curating HeinOnline profiles almost tripled our faculty citations at UW Law:
In early 2019, when U.S. News announced its new scholarly impact ranking, our librarians began working with tenured and tenure track faculty to curate their HeinOnline author profiles. Since that time, the number of UW Law School faculty citations HeinOnline has increased by 196%. Now, certainly, some of that is due to the number of additional citations that have accumulated over that time, as well as a lot of the good work that Hein has been doing to improve their data and indexing processes. But I do think a large portion of that, maybe even the majority of that, is due to our librarians working with faculty members to curate those HeinOnline profiles. As I mentioned, we found a fair amount of errors, things that were missing and we worked very hard to correct all those things. And I think that it has paid off.
If you’re interested in reading more, I invite you to check out the full paper on SSRN. I just posted some substantial revisions to the paper yesterday. For a concise listing of the strategies presented, see Appendix 1, Checklist for Improving Citation Metrics Accuracy and Appendix 2, Checklist for Promoting Scholarly Visibility at the end of the paper. A list of my other scholarship is available on SSRN and in the UW Law School Digital Repository. And you can find me on Twitter @Shucha.