DARPA, the American agency Defense Advanced Research Projects Agency, progenitor of the Internet and GUIs, among other things, is still at it. It would seem they’ve taken notice of reproducibility issues, as Nature reports:
In 2016, the US Defense Advanced Research Projects Agency (DARPA) told eight research groups that their proposals had made it through the review gauntlet and would soon get a few million dollars from its Biological Technologies Office (BTO). Along with congratulations, the teams received a reminder that their award came with an unusual requirement — an independent shadow team of scientists tasked with reproducing their results.
Thus began an intense, multi-year controlled trial in reproducibility. Each shadow team consists of three to five researchers, who visit the ‘performer’ team’s laboratory and often host visits themselves. Between 3% and 8% of the programme’s total funds go to this independent validation and verification (IV&V) work. But DARPA has the flexibility and resources for such herculean efforts to assess essential techniques. In one unusual instance, an IV&V laboratory needed a sophisticated US$200,000 microscopy and microfluidic set-up to make an accurate assessment.
The key here is Biological technology. This sort of thing is already being done for hardware and software:
Engineers expect their work to be subject to an IV&V process, in which the organization conducting the research uses a separate set of engineers to test, for example, whether microprocessors or navigation software work as expected. NASA’s IV&V facility was established more than 25 years ago and has around 300 employees testing code and satellite components.
I’m not entirely clear as to whether they refer to what I think of as QA (Quality Assurance), which consists of a collection of engineers and testers who try to make sure the software does what it’s designed to do without going off the rails. The description of this process sounds more advanced:
The synthetic-biology focus of DARPA’s Biological Control programme is well suited to merging biological research with reproducibility studies. The programme aims to bring engineering principles of design and control to biology. By definition, this requires the adoption of best practices from the engineering community — such as IV&V — to improve the likelihood that technologies can advance.
Awardees were told from the outset that they would be paired with an IV&V team consisting of unbiased, third-party scientists hired by and accountable to DARPA. In this programme, we relied on US Department of Defense laboratories, with specific teams selected for their technical competence and ability to solve problems creatively. To get comfortable with the concept of IV&V, investigators needed reassurance that replicating teams would not steal ideas or derail publications. They also needed to get used to their results being challenged even before peer-review submission, and they needed reminders that cooperating with these teams was a programme requirement.
It sounds fascinating. I hope they rotate the scientists through the various teams, both to vitiate hard feelings and to properly give all the team members to opportunity to learn all the techniques. In my experience, no one knows everything, and sometimes it can be quite an eye-opener to see how someone else solved a problem.
