I’m always impressed by how often I foul up by failing to consider possible negative consequences to what I perceive to be a positive course of action. So when this NASA decision concerning evaluation of proposals was mentioned by an old friend, I began to wonder:
NASA’s Science Mission Directorate (SMD) is strongly committed to promoting a culture that actively encourages diversity and inclusion and removes barriers to participation. One important way of achieving this objective is to ensure that the review of proposals is performed in an equitable and fair manner that reduces the impacts of any unconscious or implicit biases.
To this end, and motivated by a successful study conducted for the Hubble Space Telescope, SMD is conducting a pilot program in ROSES-2020 to evaluate proposals using dual-anonymous peer review (DAPR). Under this system, not only are proposers not told the identity their reviewers, the reviewers are not told the identity of the proposers, until after they have evaluated the scientific merit of all of the anonymized proposals.
Certainly there’s almost immeasurable merit in anonymizing proposals in order to negate bias, conscious or unconscious, in reviewers. The loss of contributions from talented minority members who are not, as a group, considered competent has been demonstrated many times over.
But there is also merit in considering the history, negative or positive, of teams who have had proposals accepted previous to this NASA program, or related programs. Experience has to count for something, even if NASA supplies project managers. And some science is difficult to evaluate, or at least so I suspect, even if I am just a simple-minded programmer who, despite his degree in Computer Science, probably hasn’t committed a single science-thingie in my lifetime. But NASA seems determined:
NASA will appoint a “Leveler” to be present in the panel room for all discussions. The Leveler is not a reviewer or a panelist, but is an individual trained to ensure that the panel deliberations focus on the strengths and weaknesses of the proposal and do not deviate into a discussion of the identity, qualifications and experience of the PI and team.
Here are some specific points:
- Levelers are present to keep the panel discussions focused on scientific merit. Unlike the chairs, they are not listening for issues pertaining to the science, rather they are focused on the discussion itself.
- If the discussion veers to comments on the proposing team, their past work, their validity, or their identities, the Leveler’s job is to refocus that discussion.
- Levelers have the authority to stop the discussion on a proposal.
I wonder if NASA has instituted any attempt at measuring what they’re gaining through better inclusion of minority contributions – as well as losing. A difficult metric, I fear.
