Who Creates the Future of AMS Peer Review? Maybe You Do!

Banners of 12 AMS journals laid out in a grid

By Gwendolyn Whittaker, AMS Publications Director

For Peer Review Week 2023, AMS and other scholarly publishers have been asked to reflect on both the essential role that peer review plays in scholarly communication, and also “the future of peer review.” In this second of our two Peer Review Week posts, we’ll take a look at how all stakeholders in AMS publications can contribute to discussions about evolving AMS peer review–and where those discussions might take place.

An evolving practice

In support of its Mission to advance science for the benefit of society, AMS publishes 12 peer-reviewed, highly regarded scientific journals. That high regard is the result of deep commitment over many decades from AMS’s volunteer leadership and from thousands of volunteer Editors and reviewers across the disciplines AMS represents. 

Researchers will take part in peer review throughout their career—sometimes as an author, sometimes playing the role of reviewer. Some will take on a journal editor role as well, with the responsibility of facilitating the review process and determining the ultimate fate of manuscripts. 

Peer review is a human endeavor, and is thus subject to human failings. Individual and systemic biases, along with global economic and social inequities, impact who has access to both the process and the results of peer review. But as with all human endeavors, its users can re-shape peer review to better serve its purposes.

As firsthand users of the tool that is peer review, researchers are the first to point out that peer review is not perfect, and the first to note where change is needed to better serve the scientific community.

At another level, disciplinary communities—such as those convened by AMS through its scientific meetings and journals—set ethical standards and best practices that reflect the communities’ values and expectations. The peer review process can and does change as those needs and values evolve. 

Peer review at AMS

For AMS, this ongoing “review of peer review” is centered in the work of the Publications Commission. Every AMS Chief Editor and the Chair of the BAMS Editorial Board is on the Commission, bringing constructive and insightful feedback from their editors, authors, reviewers, and readers to the Commission’s deliberations. The Commission sets best practices for editors, authors, and reviewers to follow, makes recommendations to AMS staff on improving processes and platforms, and provides policy and strategic recommendations to the AMS Council. 

In recent years, a particular focus for the Commission has been how to integrate AMS’s overall commitment to equity, inclusion, and justice into the publications endeavor. The Commission summarized its thinking so far in a recent editorial published in all the journals: “Equity, Inclusion, and Justice: An Opportunity for Action for AMS Publications Stakeholders.” As noted in the editorial, the Commission will be looking closely at results from AMS’s organization-wide Equity Assessment (currently underway), which will likely inform how AMS peer review evolves. 

As always, peer review at AMS will be shaped by the commitment and needs of researchers themselves, and also by scrutiny and constructive critiques from those who rely on the results—and who need the scientific endeavor to continue serving society into the future.

Have thoughts of your own on the future of peer review? Want to know more about peer review at AMS? Want to know how to volunteer to be considered as a reviewer or editor? Find out more or email us at [email protected]. We’ll be happy to hear from you!

What Do Non-Scientists Need to Understand about Peer Review?

Thoughts from AMS 2024 Editor’s Award Recipients

Peer Review Week 2023 logo

Understanding the role of peer review in science is vital not only for scientists themselves, but also for all of us who live in a society that relies on scientific research. Each September during Peer Review Week, AMS and other scholarly publishers highlight the essential role that peer review plays in scholarly communication.

In this first of two Peer Review Week posts, we’re hearing from some of AMS’s outstanding peer reviewers, recipients of the 2024 Editor’s Award, about what they think non-researchers need to understand about peer review.

At a basic level it is a check on, “do I believe the results presented here and the implications that are claimed?The check is made by other researchers working independently in the field. The checking of a single paper isn’t exhaustive, but there is an ongoing process—results and ideas established in one paper will, if they are of any significance, be re-examined and developed further in subsequent papers, which will themselves be peer reviewed.

Dr. Peter Haynes, Cambridge University
Dr. David Bodine

For non-researchers, I think it’s important to understand that peer review requires substantial effort … by volunteer reviewers, editors, and [the] scientists submitting manuscripts. A well-coordinated review process by all involved improves the quality and ensures the integrity of scientific research.

Dr. David Bodine, University of Oklahoma

Dr. Elizabeth Yankovsky

The peer review process is the only barrier standing between the writing of a scientific study and its publication. It is very easy for an unsubstantiated or erroneous paper to set an entire field back by years. In my opinion, the peer review process is as important as the research that goes into a given paper. … A given scientist may have one perspective and associated biases. Through peer review, the results are assessed by other scientists and are judged against the state of knowledge of the field. To push our boundary with the unknown forward, scientists must rely on both the historical backbone of their field as well as thorough review by their modern-day peers.

Dr. Elizabeth Yankovsky, New York University

Just because a paper was published after undergoing “peer review” does not make it absolutely correct or perfect, nor is it the final message on that idea. Unfortunately I feel the phrase “peer reviewed” is often used to imply some absolute consensus on a subject has been reached, when in reality it’s an ongoing, necessary criticism of the science that we do. As scientists we are constantly peer-reviewing each other’s work, and this may spark new, contrary ideas to be published that refute earlier findings.

Dr. Luke Madaus, Jupiter Intelligence
Dr. Sarah Buckland

Popular culture oftentimes misconstrues science in either the extreme of being purely political/agenda-driven or being the ultimate source of truth without question. The truth is, science is not ultimate, and understandings of processes and concepts are dynamic, and, as is especially evident in interdisciplinary research, scientists do have perspectives shaped by experiences. While I also cannot deny that bias exists in scientific fields and that contrasting perspectives may filter out at times, authentically anonymous and double-anonymous peer review processes (i.e., the reviewer not knowing the authors’ name(s)), act as guardrails to significantly reduce bias. [If] these processes remain clean and the selected reviewers are indeed experts in the field of the papers that they review, this significantly aids in ensuring that the end product is of the highest quality. The existence of these processes is why academic journals are deemed among the most credible sources of scientific information.

Dr. Sarah Buckland, University of the West Indies

Dr. Mimi Hughes

I think what I wish non-researchers understood about the scientific review process is how many eyes are on these papers before they’re published, and how that regularly improves the science and writing of the end-product. Most reviewers take the responsibility very seriously, and indeed are usually hesitant when they haven’t “found enough to fix” in a paper they review. It is typically a truly rigorous process.

Dr. Mimi Hughes, NOAA Physical Sciences Laboratory

Dr. Aaron Hill

I think non-researchers should know that peer review is only good and valuable when it is conducted from an unbiased position. It is vitally important that authors receive unbiased, external perspectives on their work in order to ensure that any gaps or misunderstandings can be addressed, and that the science is technically sound. Peer review is just ONE step in the scientific process as well, and sometimes bad work slips through the cracks of review. But peer review is a critical component to upholding and advancing science.

Dr. Aaron Hill, Colorado State University

Dr. Qiaohong Sun

Peer review serves as a crucial method for the scientific community to uphold the quality and credibility of scientific information accessible to the public. A paper passing peer review doesn’t guarantee absolute perfection, it indicates a level of examination and approval by experts in the field to some extent at the current time.

Dr. Qiaohong Sun, Nanjing University of Information Science and Technology

Dr. Sebastian Lerch

Peer review is a critical control mechanism in the scientific process. Mistakes can happen and may still get through the process. However, the collective nature of peer review and subsequent scrutiny by the scientific community help correct errors over time. This in particular highlights the importance of making research reproducible by publishing data and code.

Dr. Sebastian Lerch, Karlsruhe Institute of Technology

Dr. Andrew Feldman

It works! It is the main mechanism that keeps science reliable and transparent. Scientists respect and cite published work. In order to get science published, it needs to be read by 2-4 anonymous colleagues and editors and then revised. Even when it goes wrong and a paper is published with an error or not-well-supported argument, researchers are good at detecting this after the fact. It is a robust process that keeps the advancement of knowledge at a high-quality and transparent level.

Dr. Andrew Feldman, NASA Goddard Space Flight Center

Letting Scientists Benefit Us All

Lately you may have noticed that AMS has garnered media attention by standing up for NOAA scientists who are the focus of Congressional scrutiny. This scrutiny was initiated after the scientists re-analyzed global surface temperatures with newly corrected data and found that the warming trend of the second half of the 20th century has been continuing unabated since 1998 instead of experiencing what sometimes has been portrayed as a warming “hiatus.”
AMS doesn’t step casually into political arenas. As a non-profit scientific and professional society, we remain solidly grounded in the world of science. We help expand knowledge and understanding through research and, as our mission states, we work to ensure that scientific advances benefit society. We engage the policy process to help inform decision making and to help ensure that policy choices take full advantage of scientific understanding.
This case is slightly different, however, because the scientific process itself is at risk. When the scientific process is disregarded or, worse yet, possibly derailed, a political issue can become an AMS issue.
The scientific process that AMS and other like-minded institutions have championed over the centuries is about taking careful observations, conducting controlled experiments, separating personal opinions and beliefs from evidence, and, perhaps most critically, exposing scientific conclusions to rigorous and repeated testing over time by independent experts. These repeated cycles of distribution and “trial by fire” happen most notably at meetings, in peer-review, and in publication.
Crucially, the process systematically removes as much as possible of our human tendency to see what we want to see and puts the burden of proof on reproducible steps. It is a disciplined, particular way of finding truths, no matter how elusive, while rendering biases, opinions, and motivations as irrelevant as possible.
This systematic approach to separating fact from opinion occasionally goes astray, of course, but its iterative nature means that science is continually self-correcting and improving; better data and understanding ultimately replace older thinking. Science encourages people to question and challenge thinking, certainty, and accuracy—but it requires they focus exclusively on what they can detect and measure and reason.
Even though all the data, logic, and methodologies are publicly available, the paper rejecting the global warming hiatus inspired Congressional requests for additional email and discussions. Asking for these correspondences—especially from scientists themselves—can easily weigh down the ingenious process by which science has continually advanced. And so AMS made public statements in favor of letting science freely work its wonders. It’s not the first time AMS has done so, and it probably won’t be the last.
We owe much of modern prosperity to an unencumbered scientific process, and it continues to provide some of the most profound and dramatic advancements in the world. This includes medicine, biology, chemistry, computing, agriculture, engineering, physics, astronomy, and, of course, meteorology, hydrology, oceanography, and climatology. Every one of us benefits every single day from what scientists have learned, shared, and provided.
And that’s yet another reason why occasionally AMS must speak out—because of our mission “for the benefit of society.” The point is not just to protect science but also to protect the benefits that knowledge can provide to all of us, no matter what we think of the results. In this, our scientific society actually has much in common with the politicians and policy makers in Washington, D.C.
AMS stands behind the scientific process and will defend that process when necessary, but our goal is to work with policy makers to promote having the best knowledge and understanding used in making policy choices.