Research assessment reform at EMBL: an interview with Wolfgang Huber
Wolfgang Huber discusses why research assessment needs reform and how this could make science more diverse, more inclusive, and better
Research assessment across EMBL is changing. EMBL’s Research Assessment Reform Working Group is leading changes to better measure the quality and impact of our research, valuing diverse outputs and activities from across the organisation.
Why are you passionate about reforming research assessment?
I started out as a scientist in theoretical physics and computing. During my postdoc, I got excited by genomics and bioinformatics and soon realised that the really impactful work in these fields did not only consist of papers, but was accompanied by software and data. I found outstanding colleagues and people to collaborate with in the R and Bioconductor communities. This work, and perhaps even more importantly, this approach, was the reason that I got the opportunity to start a research group at EMBL-EBI.
At the same time, the so-called ‘reproducibility crisis’ was unfolding, with scientists claiming spectacular discoveries in papers in high-profile journals, based on data analyses nobody could reproduce. This convinced me that fully transparent analysis paths, from raw data to published figures, were the future of scientific publishing. These needed to be based on easily available, and wherever possible, open-source software.
Making data analysis fully transparent, or providing re-usable, useful methods to other scientists as software packages is a lot of additional work that was mostly not credited until now. So the incentives in the system were, to put it bluntly, against doing the right thing. This needs to change.
What role does EMBL play in reforming research assessment?
Deciding which researchers to recruit and which projects to pursue is a core responsibility of any research organisation. It is the key to its success or failure as an institution. Nevertheless, there has been a trend to outsource such research assessment: journal editors decide which papers get which level of visibility and prestige, and subsequently which scientists advance in their careers and which type of science gets funding.
It is time for researchers, and for their institutions, to take control of their core business. For EMBL, being an early mover and innovator is simply a competitive advantage. It is also part of EMBL’s mission to be a leader in such matters, to develop and figure out approaches to doing excellent research and training that then hopefully inspire similar change elsewhere.
What are your thoughts on how research assessment is evolving?
I’m amazed how rapidly research assessment is changing. Funding bodies are a big driver for change. The Wellcome Trust, for example, was an early adopter of valuing different types of research outputs based on their impact and contribution.
It’s not only top-down; the momentum is also building bottom-up. Especially, early-career scientists are rethinking traditional systems of research assessment that set perverse incentives or are based on extensive gatekeeping and groupthink.
Goodheart’s law states: “When a measure becomes a target, it ceases to be a good measure”. This is how I think about scientific publishing: a journal’s prestige and maybe even its impact factor may be a valid statistical correlate of the quality of the papers in it. But by making it the target to publish in a certain venue, in a very narrowly defined format, rather than focusing on content and allowing a wide range of publication or distribution formats, we have put the cart before the horse. It’s important to go back to basics and focus on what we actually want to achieve.
What changes have been made across EMBL to improve research assessment?
EMBL’s recently formed Research Assessment Reform Working Group is working to drive changes in EMBL processes. EMBL is a signatory of both DORA and CoARA – two of the main initiatives in this field – and this means we have made a commitment to change our processes and practices in line with their principles.
We are implementing three major actions. The first is a clear statement, externally and internally, of EMBL’s commitment to the principles of DORA and CoARA. Secondly, we updated the type of research outputs we ask candidates to list when they apply for EMBL positions. In their CVs, candidates are now asked to augment the traditional ‘publication list’ with a text that summarises the impact and importance of their research outputs in a more conceptual, narrative form. Thirdly, we have updated the EMBL unit review procedure and awards documentation to provide guidance to internal interviewers and reviewers.
What still needs to be done across EMBL to reach our research assessment goals?
Reforming research assessment across EMBL is a work in progress, and a long-term commitment. There are no easy fixes; it requires a culture change that will take time, effort, and continuous readjustment. Doing research assessment well requires that you actually have the expertise to assess a scientist’s work and are willing to engage with it.
We also must face the fact that research assessment is not always practical. Some degree of outsourcing and reliance on external indicators will be unavoidable. We need to be creative, flexible, and always exert good judgement. Science and technology can move very quickly, but sometimes, to fully assess the impact of research outputs, we need longer timeframes, vision, and willingness to take a risk.
Making these changes to our assessment processes and acting on them now will be of enormous benefit to EMBL in the years to come. It will enable us to recognise excellence in all of its forms and secure a diverse pool of talent.
What has shaped these changes in the way we think about research assessment?
There has been a great deal of attention by the public to the ‘reproducibility crisis’ in science. Newspaper readers were confused by big claims that turned out to be wrong, or by dubious results hyped up as breakthroughs. Big pharmaceutical companies were complaining that published results from academia could not be replicated. Such things have the potential to shake the public’s perception of science and their trust in scientists. Funders started to take notice and decided they needed to do something about it.
Those who pay for science – scientists’ salaries, but also their equipment and buildings – have a right to expect something in return. We need stringent and effective ways to separate good work from bad work. I am sympathetic to the underlying idea of ‘publish or perish’, but in the past, some of the concrete implementations by institutions have created perverse incentives, leading to dishonourable behaviour and toxic workplace environments. Many good scientists decided that this was not for them and have been leaving academia.
How might reforming research assessment change science?
Better research assessment will result in more sustainable and better science. It will also make science and academia a more attractive workplace. These changes should create more inclusive and diverse working environments, increasing the talent pool, which is a competitive advantage for any institution that is leading the charge. Eventually it will improve the quality and impact science has globally.
I welcome these changes and am excited to see how reforming research assessment will benefit the science that will come out of EMBL.
Exploring best practices in research assessment: EMBO, EMBL and DORA
On 12 May 2023, the 10th anniversary of the San Francisco Declaration on Research Assessment (DORA), EMBO and EMBL hosted a hybrid panel discussion reviewing the initiative’s impact on the way research is assessed. The six panellists discussed issues with current methods of research assessment, as well as solutions and actions for improvement in which panel members have been involved. Read more about the discussion here.