Home / Advocacy Article

August 20, 2020

American Physiological Society Responses to NOT-OD-20-130 Enhancing Rigor, Transparency, and Translatability to Improve Biomedical Research Involving Animal Models

Rigor and transparency
The challenges of rigor and transparency in animal research and actions NIH can take to improve the quality of animal research including rigor and transparency.

APS Response: While this working group was asked to address rigor and reproducibility in animal studies, comparable challenges affect all research. NIH asks its grantees to provide other researchers access to their data, research methods, and specialized materials. Data sharing is critical to enabling further analysis of findings. However, data are costly to archive and make accessible. NIH should establish a grantee data repository within NLM so data can be linked to grants in RePORTER and to the PMIDs of published articles. This will make data-sharing sustainable for investigators. Even so, difficult issues remain: What standards (raw/cleaned, formatted, annotated, etc.) should be applied to the data? Who will ensure data meet those standards? Who pays for preparing the data? What happens if data access requires proprietary software?

Access to detailed experimental methods are needed to enable other researchers to replicate findings. NIH should also create an NLM methods repository and link it to grants in RePORTER and the PMIDs of articles.

Consensus is growing around the minimum information needed to promote rigor and reproducibility in animal studies. In 2010, the NC3Rs published the Animals in Research: Reporting In Vivo Experiments (ARRIVE) Guidelines. These encouraged researchers to provide extensive detail on research design and procedures. Although widely heralded and nominally adopted by many journals, there is little evidence that these Guidelines are being used to screen articles.

In 2017, Norway’s National Consortium Consensus Platform for the advancement of the 3 Rs (Norecopa) published the Planning Research and Experimental Procedures on Animals: Recommendations for Excellence (PREPARE) Guidelines. This planning tool was intended to complement ARRIVE. In addition, the NC3Rs developed an Experimental Design Assistant (EDA) that was also intended to help researchers “design robust experiments more likely to yield reliable and reproducible results.” In addition, an updated set of ARRIVE 2.0 guidelines were published on July 14, 2020 with a focus on information most needed to evaluate the rigor of studies. In a PLOS Biology Perspective, the ARRIVE authors described this “Essential 10” as the subset of their guidelines addressing the most critical details.

While NIH cannot control how journals evaluate scientific manuscripts, it can set expectations for how research should be conducted and decide what research to fund. NIH should promote resources such as the EDA, PREPARE, and the ARRIVE Essential 10 and encourage grantees to plan their research using these tools. The additional study design information that grantees provide should go in the Vertebrate Animal Section so it does not truncate the discussion of the science. Reviewers should then be asked to consider pertinent aspects of study design.

Since grants are typically reviewed a year or more before the work begins, it would not be feasible for applicants to provide the level of detail that would go into an animal use protocol submitted to an IACUC. This is particularly true with exploratory studies that may have multiple aims that build upon one another.

How preregistration, the process of specifying the research plan in advance of the study and submitting it to a registry, would impact animal research including improving the quality of scientific research. (462)

APS Response: PHS-funded research protocols that involve vertebrate animals must be pre-approved by an IACUC. The PHS Policy for the Care and Use of Laboratory Animals requires IACUCs to use the Guide for the Care and Use of Laboratory Animals in doing so. The Guide’s purpose is to “assist institutions in caring for and using animals in ways judged to be scientifically, technically, and humanely appropriate” (Preface, p. xiii). The IACUC conducts a detailed review of planned experiments and can require the investigator to make changes before the research can proceed. In addition, before investigators can make significant changes in the research, the IACUC must approve a protocol amendment. In addition, IACUCs conduct post-approval monitoring of protocols. The American Physiological Society would therefore like to know what preregistration of animal studies would entail and whether it will duplicate the oversight already provided locally by the IACUC. If there would be duplication, what further benefit will it add?

According to the Clinicaltrials.gov website, preregistration of human clinical trials fulfills ethical obligations to participants and the research community; provides information to potential participants and referring clinicians; reduces publication bias; helps editors and readers understand the context of study results; promotes more efficient allocation of research funds; and helps institutional review boards determine the appropriateness of a research study.

APS would like to know which of these stated goals are intended to apply to animal studies and how the preregistration process would work. Specifically, what would investigators have to do—in addition to seeking approval from the IACUC—if changes to the protocol were needed for humane or scientific reasons? This might be less of an issue for experiments intended to test the safety and effectiveness of an intervention prior to human clinical studies. But what about discovery research to identify a biological mechanism or to generate or confirm a hypothesis? It is difficult to define discrete categories of research, which makes it challenging to assess what research might benefit from preregistration. It may be helpful for some kinds of late-stage, preclinical studies, but it could be detrimental to early-stage, discovery research. It should be noted that not all clinical studies conducted in the U.S. must be registered on clinicaltrials.gov, which has been a source of confusion.

Perhaps preregistration is meant to provide a repository for both positive and negative results. This could reduce duplication of unfruitful efforts and promote more efficient allocation of research funds. However, given the widespread frustration that has been expressed about the ClinicalTrials.gov website, a better way to do this should be found.

APS urges NIH to provide more specifics on its goals and intentions for preregistration and how it defines preclinical research. It should then give the research community the opportunity to comment in greater depth than the current RFI permits.

While preregistration is often considered in the context of hypothesis-testing and confirmatory experiments, would it be useful at other stages of the research process, such as the exploratory and hypothesis generating.

APS Response: Hoped-for benefits of preregistration must be weighed against potential costs. For clinical trials, preregistration requires providing details of the planned study with the expectation that the design will be followed to the letter. This is intended to reduce so-called “massaging of the model.” It makes sense to do this for clinical trials, but this would have significant downsides for exploratory and hypothesis-generating studies.

In some discovery research, so little is known about outcomes that pilot studies are needed to determine how large a sample is required to obtain statistically valid results. In addition, unexpected findings at one stage or in one arm of a study may require changes in other arms or later stages. Protocols are routinely revised as new information and technologies become available. With new procedures, adjustments may also be needed to ensure the health and comfort of animals. These kinds of changes are routinely handled at the institutional level—as specified in the PHS policy—by amending the IACUC-approved protocol. It is unclear how preregistration could accomplish the positive goal of ensuring the integrity of the model without creating significant barriers to cutting-edge research.

With human clinical trials, some journals agree in advance to publish the results of preregistered studies because there is public interest in the outcome. This practice provides access to both positive and negative findings, which is valuable. However, journals have shown much less interest in publishing null findings from discovery research. We recommend that NIH find a less cumbersome mechanism than preregistration through ClinicalTrials.gov (or a similar platform) for providing access to negative results. One such option is a PMC repository for negative findings.

Some areas of research are controversial, such as those involving human embryonic stem cells, dangerous pathogens, and certain animals. Preregistration poses a risk to such research because it could facilitate the efforts of opponents to exploit or block it.

Preregistration is not the only approach NIH could take to improve rigor: It could encourage researchers and their institutions to monitoring or even controlling the conditions in which animals are bred and housed (i.e., temperature, crowding, light sources, air flow, vibration, etc.), as well as conducting periodic recalibrations of instruments. Other steps might include periodic genotyping to determine whether there has been genetic drift in a colony; running control samples; and re-checking baselines for research procedures. Such quality-control measures can reduce the impact of under-appreciated yet highly influential variables, thus increasing reproducibility across research sites. However, NIH must be prepared to bear a significant portion of the additional costs required to implement these measures.

The Federal Demonstration Project’s Compliance Unit Standard Procedure (CUSP) is an online repository (currently in alpha testing) where institutions can share standardized protocols. Although primarily intended to reduce regulatory burden associated with developing research protocols, it could contribute to improved rigor if it were used more widely. Though CUSP may not accomplish all of the intended goals of preregistration, it offers a partial alternative that would not increase burden to investigators or costs to NIH.

How to address the complexity and expense related to use of large animals, including nonhuman primates, that may provide biologically more relevant models.

APS Response: Large animals that provide biologically relevant models of human biology are expensive to maintain and pose complex challenges in terms of scientific interventions, veterinary care, and administrative management. It is expensive to maintain colonies of large animals during a funding hiatus; difficult to restart such a colony; and almost impossible to start one de novo in the absence of a national emergency and a significant infusion of funds.

The National Primate Research Centers provide an example of how NIH has facilitated the optimal use of large animal models. NIH should establish comparable centers to provide researchers with shared access to other large animals, along with the facilities’ resident expertise caring for the animals and equipment for research procedures, tissue sampling, etc. NIH should provide baseline funding to ensure that such facilities can maintain their colonies and retain staff with the relevant expertise. Such centers should also function as clearinghouses to facilitate collaboration in whole animal studies across institutions as well as provide access to tissues or organs. NIH will also need to issue RFAs so that individuals who have identified a phenomenon in a rodent can collaborate effectively on confirmatory studies with researchers at a large animal center.

How NIH can partner with the academic community, professional societies, and the private sector to enhance animal research quality though scientific rigor and transparency.

APS Response: Many professional societies, including the American Physiological Society (APS) are working to enhance the quality of animal research by applying standards of scientific rigor and transparency. In its capacity as a publisher of 14 peer-reviewed journals, the APS policy for reporting on animal experiments is this:

Authors are strongly encouraged to refer to the ARRIVE Guidelines Checklist in preparation for reporting the methods and results of animal studies. Particular attention should be given to providing detailed information regarding the animals and the controls used in the study, the precise details of all experimental procedures and the steps taken to minimize subjective bias in the study design. [Emphasis in original.]
In addition, the APS Instructions for Authors specifically encourage the reporting of these experimental details:
  • Detailed information regarding the methods and results including:
    • Precise details of all experimental procedures (drug formulation and dose, anesthesia and analgesia used, method of euthanasia, etc.)
    • Steps taken to minimize subjective bias in the study design (randomization, blinding, etc.)
    • The animals used in the study (species, strain, sex (for vertebrates), age, source of animals, genetic modification status, housing, diet, and associated RRIDs.)
    • The controls used in the study (littermate, purchased, identical conditions, contemporaneous, historical, etc.) [

NIH should partner with professional societies to identify current and emerging animal models that should be evaluated and validated. Last year NIGMS published the NAGMSC Working Group on Sepsis report, which recommended “the standardization of animal models and the development of models that more closely mimic (1) non-immunological aspects of sepsis and (2) important co-morbidities in human sepsis.” The FY 2021 House Appropriations Committee report calls upon NIH to evaluate preclinical animal models for drug development in Duchenne Muscular Dystrophy. Professional societies can help NIH identify experts to help with such evaluations. The NIH should evaluate the success of such efforts and determine the usefulness of similar measures to evaluate preclinical animal models for other areas of research interest.

NIH should also work with professional societies on a clearinghouse of resources of research techniques. The APS journals regularly publish peer-reviewed articles on research methodology in a collection called Cores of Reproducibility in Physiology. The American Journal of Physiology: Heart and Circulatory Physiology also publishes a series on Guidelines in Cardiovascular Research. APS provides free access 12 months after publication so many of these articles are already freely accessible. APS would be happy to arrange immediate free access to newer articles for NIH to include among its Rigor and Reproducibility resources.

Optimizing the Relevance to Human Biology and Disease
Actions NIH can take to facilitate the translatability of animal research to human biology and disease.

APS Response: First and foremost, NIH should ensure that animal studies are funded adequately. This may require supplements when pilot studies unexpectedly identify different effects in male and female animals, necessitating more animals than anticipated. When NIH unilaterally reduces grant funding, it is difficult for investigators to obtain sufficient animal subjects to test their hypotheses. This puts them in the impossible position of trying to accomplish the aims of their grants without the funds to do so. NIH can also improve the translatability of animal studies by making a practice of funding studies to confirm initial results. These confirmatory studies should involve alternate species that will improve the likelihood of determining whether the intervention will have efficacy in humans.

NIH needs to take steps to ensure that Scientific Review Officers are vigilant about preventing study section members from expressing biases against non-rodent models. In addition, as noted above, NIH should fund studies to validate animal models currently in use as well as promising new ones to avoid perpetuating reliance on models with insufficient validity. This will require:

  • Funding grants seeking to make explicit comparisons between biological mechanisms in humans and animals. The goal should be to identify and characterize new translational models. (Currently such proposals tend to be considered “descriptive” and are therefore given low funding priority by study sections.)
  • Funding studies to validate translational models in non-rodent species.
  • Using mechanisms such as P60 grants to maintain large animal centers where resident investigators can work with highly translatable models.
  • Providing grants for researchers to collaborate with resident experts at P60 large animal centers. The goal should be to create collaboration mechanisms and funding streams so that promising findings in rodent species can be confirmed in other species.
How NIH can partner with the academic community, professional societies, and the private sector to enhance animal research translatability.

APS Response: NIH has previously sought to partner with the academic community and professional societies in its efforts to increase rigor and reproducibility. As part of this earlier effort, the American Physiological Society (APS) was awarded an NIGMS grant to develop a training module on Controls in Animal Studies. This course is available in the NIGMS Clearinghouse for Training Modules to Enhance Data Reproducibility. NIH should renew this effort with a greater focus on topics relevant to enhancing rigor and translatability of animal research.

As noted above, APS journals have published numerous peer-reviewed articles on research methodology under the rubrics of Cores of Reproducibility in Physiology and Guidelines in Cardiovascular Research.

The challenge, of course, is not simply to have these kinds of resources but also to make a concerted effort to make researchers aware of them and provide ready access to them. NIH should partner with professional societies to accomplish this through newsletters and in sessions at scientific meetings.

Research Culture
How research culture drives the choice of animal models.

APS Response: Researchers typically learn how to work with a particular model during their training. If they continue in the same field, they will continue adding to the existing store of data acquired using that model. However, when researchers learn about another model that appears better suited to answering their research questions, they are eager to begin using it. Nevertheless, there are hurdles to adopting a new model, e.g., training to work with the model; repeating experiments to establish new baseline data; purchasing new equipment; and ensuring that their institutions have the facilities and expertise to care for the animals. If NIH invests in identifying and validating new animal models, researchers will want to use them, but NIH will also need to use a mechanism such as the P30 grant to enable researchers to become proficient in working with the new model. Supplemental funding may also be needed for grantees to purchase new equipment.

We have heard anecdotal reports that some grant proposals were scored poorly by study sections because the work involved a non-rodent species. This is particularly discouraging when researchers know that the model they selected is a better representation of the aspect of human biology under study. This bias undermines efforts to produce more translatable research, and NIH must take steps to address it.

Some researchers are reluctant to work with large animals because animal rights activists or extremists have harassed or targeted other researchers for work with these animals. Of particular relevance to the question of research culture is the fact that there are well-known cases of researchers who were harassed in this manner and did not receive adequate protection from their institution or support from their funding agency.

A further additional consideration is the fact that early-career researchers have limited time and funding to demonstrate excellence as a scientist, which is a prerequisite to achieve promotion and tenure. This produces an incentive to work with established models, and particularly with less expensive small animals such as rodents.

How all researchers, including trainees, are educated in rigorous research design, statistical considerations, transparent research practices, and the role of NIH in this training.

APS Response: NIH requires that there is a plan in place to educate trainees supported by NIH training grants. However, as new technologies and techniques are developed, and cultural changes improve research design and data analysis approaches, NIH should consider requiring established investigators to demonstrate they are staying current with changing accepted practices. The NIH might introduce a new section in grant applications and progress reports for a professional development plan emphasizing the goals of the NIH with regards to rigorous research design, statistical consideration, and research practices. It could also include other relevant issues such as mentorship and responsible research conduct.

NIH’s role in this effort should be to encourage investigators to take advantage of opportunities for continuing education. To facilitate this, NIH should make available and fund self-training materials on study design, statistical analysis and power, model selection, etc. This is not a new concept to the NIH, as noted above in our discussion of the NIGMS grant awarded to the American Physiological Society to develop a training module on Controls in Animal Studies, which is now available in the NIGMS Clearinghouse for Training Modules to Enhance Data Reproducibility. To the extent that such materials have already been developed, NIH should collect them in a central clearinghouse on its website. The resources should be appropriately tagged and circulated in relevant institute newsletters to facilitate continued traffic to such resources. As gaps are identified, NIH should commission the development of materials that can fill them.