APS Comments on Proposed Simplified Framework for NIH Peer Review

February 23, 2023

Noni Byrnes, PhD
Director, Center for Scientific Review
National Institutes of Health
6701 Rockledge Drive MSC 7768
Bethesda MD 20892-7768

Dear Dr. Byrnes,

The American Physiological Society (APS) strongly supports the goals of the proposed revised simplified review framework: to simplify the peer review process and reduce burden on reviewers, and to reduce implicit bias in the review process. APS commends the evidence-driven approach that NIH has taken to develop a framework that addresses long-standing concerns of the scientific community. As the NIH moves toward implementation of a new framework, we ask that NIH take the opportunity to expand upon its guidance and training for reviewers and to improve the community’s understanding of the review process.

The reorganization of the primary review criteria into three factors is a laudable approach that refocuses the review on the scientific merit of a proposal. The two factors that reviewers will be asked to score — the importance of the research and the rigor and feasibility of the approach — provide a comprehensive evaluation of the most critical elements of a proposal. Evaluating both “innovation” and “significance” within the context of the importance of the research provides room for proposals that do not apply any novel approaches, but which nevertheless address critical knowledge gaps in a field.

Within the current peer review framework, reviewers are asked to provide scores for the qualifications of the investigator and the environment or institution where the proposed research will take place. These scores may provide an unfair advantage to researchers and institutions that are already well-funded, or a disadvantage to those with less established reputations. Removing the individual scoring for these elements is a step in the right direction toward reducing the effect of bias in the peer review process. However, the reputation of a proposal’s investigator or their institution can still influence a reviewer’s evaluation of other elements of a proposal as well as the overall impact score. To address this potential source of bias, NIH should include specific guidance about how to consider these criteria as part of reviewer training. Additionally, while it is important that reviewers have an opportunity to flag any specific gaps identified in expertise or resources, reviewers and applicants should be able to clearly understand how any identified gaps will affect a proposal’s final evaluation. To guard against potential hidden bias, we encourage NIH to consider adopting a uniform policy for how reviewers defend their choice for voting "out of range" so that these reasons can be incorporated into the summary discussion and summary statement that is transmitted to NIH Council and the applicant.

As the updated peer review framework is implemented, APS strongly encourages NIH to use the opportunity to improve the scientific community’s understanding of how best to navigate the peer review process. This should include offering updated training and support for peer reviewers to minimize bias, promote commentary that is helpful to applicants, and encourage appropriate use of the full range of the 1 – 9 scoring system. NIH should monitor comments and scoring outcomes from study sections and take steps to promote consistency across review groups. NIH may wish to consider hosting workshops or developing resources such as rubrics to help reviewers and applicants understand how to navigate the new framework. Finally, both reviewers and applicants would benefit from greater transparency about how a proposal’s overall impact score is determined.

The revised simplified review framework that NIH has proposed is a commendable approach to restructuring review criteria that will hopefully strengthen the current peer review process. The new framework appropriately emphasizes the evaluation of the scientific merit of proposals and reduces the burden and inconvenience for reviewers by reducing the number of elements for reviewers to consider and simplifies responses with drop-down ratings. Once the updated framework is implemented, APS urges NIH to carefully evaluate outcomes to ensure the new framework is achieving its intended goals and to assess whether any further action is necessary to counter implicit bias.

APS appreciates NIH’s efforts to develop a proposed framework that is responsive to feedback and common concerns from the scientific community. We look forward to supporting NIH and the scientific community in the implementation of the finalized framework.

Sincerely,

Dennis Brown, PhD, FAPS

Chief Science Officer