- In today’s academic circles, the peer review process is largely conducted behind closed doors by institutional gatekeepers, resulting in an opaque decision-making process that consistently disadvantages researchers from historically marginalized populations and under-resourced institutions.
- We’ve been working with PREreview since 2019 to invert this power imbalance by helping to build a new open source platform that supports a more inclusive, community-oriented form of peer review based on equity, openness, and collaboration.
- Our design work on the ongoing project addressed key systemic problems embedded in the peer review process, including an overreliance on antiquated concepts of prestige, a general inaccessibility for new reviewers, and an unnecessarily high degree of risk for certain scholars who may find themselves the target of harassment in the form of aggressive and harmful feedback.
Note: This post is adapted from a blog post published on eLife Labs by Georgia Bullen (Simply Secure), Daniela Saderi (PREreview), and Katie Wilson (Simply Secure). This is the second post detailing our collaboration with PREreview. For a more general overview on this project, and the power imbalances embedded in academic publishing, click here.
Change – particularly in the digital realm – must be actioned through design. In working with PREreview to overcome structural inequities in peer review, our team sought to support PREreview’s mission to invert inequitable power imbalances by helping to build a new open source platform that supports a more inclusive, community-oriented form of peer review. Our goal in doing so was to shape a platform that could serve as both a tool on which preprint evaluation work could be performed as well as a space through which community convening could be fostered. Along the way, we identified three primary challenges embedded in the peer review process: (1) an overreliance on antiquated concepts of prestige, (2) a general inaccessibility for new reviewers, and (3) an unnecessarily high degree of risk for certain scholars who may find themselves the target of aggressive and harmful feedback. Below, we detail the design solutions developed with the PREreview team and employed to mitigate these issues and help shape a more equitable way of evaluating scholarly work.
Overcoming prestige as a proxy for reviewer worthiness
In academia today, traditional publication gatekeepers are disproportionately male and from the United States or Europe. Their affiliated institutions are also almost always well-connected and well-funded. As a result, reviewer “worthiness” is often deemed to be dependent on antiquated concepts of prestige like name recognition, personal connections, and institutional affiliation. But these proxies do not always correlate with true reviewer contribution, ability, or trustworthiness – which means that many reviewers are excluded from the peer review process despite the fact that they are in fact qualified to participate (and their participation would greatly enhance diversity and inclusion in academia). It also means that certain reviewers may be viewed as integral to the review and publishing process – despite the fact that their actual contributions may amount to nothing more than a de facto stamp of prestige approval.
To help address this problem, we used design to actively highlight individuals on the PREreview platform based on their contributions to the preprint review process, rather than focusing on their status or stage in career. User profiles emphasize participation elements such as badges a PREreviewer received in recognition of their feedback on a project as well as the communities the PREreviewer belongs to on the platform. [See user profile persona in Figma, below.] Notably, career level and institutional affiliation are not removed, but instead de-emphasized in the profile itself and cannot be used as a way to filter reviews. Though these changes might seem subtle, they enable PREreview to underscore the humanity of each reviewer and ensure that they are recognized and valued for their actual contributions to the advancement of knowledge and community on the platform.
Increasing accessibility in the peer review process
As a result of the systemic inequities in academia, the majority of research submitted for peer review never makes it before a diverse pool of reviewers who would be able to better contextualize it and implement a more comprehensive evaluation of the submitted work’s merits. More researchers from underrepresented groups and regions must be included in the peer review and publication process – yet such change is unlikely to materialize within the existing system which remains inaccessible to many individuals and communities, and particularly early-career researchers.
In response to these challenges, we designed to help increase accessibility in several different ways:
- First, login and signup for PREreview was set up to be completed through the ORCID Public API, which helps to contribute to a future in which all researchers and scholars are uniquely identified and connected to their contributions and work. Looking forward, a potential goal is to also allow PREreviewers to have their preprint reviews listed on their public ORCID profile as a means to further enhance recognition for their contributions.
- Second, in an effort to mitigate the impacts of the general lack of formal peer review training, text-based template guidelines were created as a way to simplify how to write a review and empower early-career researchers who may just be getting started in preprint review. These templates can also be customized by community managers to address their own specific needs.
- Third, a system for rapid PREreviews was created to enable swift, structured feedback on preprints. The idea developed out of a partnership between PREreview and Outbreak Science, focused around community needs during outbreaks, but turned out to be of interest and useful to the broader community. Rapid PREreviews allow easy access to provide standardized feedback and contribute to the review process (while also serving as a helpful barometer of a preprint’s credibility and value within a given community).
- Fourth, the PREreview platform was structured so that users and people on other sites can request PREreview feedback directly on the platform or through the platform’s API. Together, these and other related design elements serve to lower the barriers to entry in the peer review process which have historically worked to exclude researchers from underrepresented populations and under-resourced institutions.
Protecting reviewers from unnecessarily harmful feedback
Throughout our research, we heard that for some researchers – and particularly those from underrepresented groups – the peer review process can involve unnecessarily harmful and antagonistic feedback. This type of hostile treatment can be silencing and perpetuate cycles of exclusion and inequality in academia.
In response, we sought to ensure we were designing a platform that would protect all of our users while fostering a safe space for feedback and community convening. To achieve this, we worked with the PREreview team on ways to integrate the Code of Conduct into the user experience to support the values through the UX, rather than be something that users would agree to once and then forget. Additionally, a reporting feature was also built into the platform, enabling users to flag inappropriate PREreviews for follow up by community moderators and PREreview administrators. Finally, understanding that members of the communities we are seeking to serve can also be the targets of unethical retribution in response to a critical review, we worked with the team to design a pseudonymous profile option that supports users to protect themselves while still adhering to all Code of Conduct requirements.
Credits
Project Contributors: Georgia Bullen, Katie Wilson, Carissa Yao, Lorraine Chen, Eileen Wagner
With support from PREreview, this work was funded by The Alfred P. Sloan Foundation, The Wellcome Trust, and the Mozilla Foundation.
