The first two speakers at March's International Symposium on Grids and Clouds 2013 (ISGC2013) were Steven Newhouse, director of European Grid Infrastructure (EGI) and Daniel Katz, program director in the Division of Advanced Cyberinfrastructure at the US National Science Foundation (NSF). Together, they provided perspectives from both sides of the Atlantic Ocean on ways to measure the success of e-infrastructures, including the people and services that contribute to their development. Steven Newhouse introduced some of the metrics currently being used to determine the impact of e-infrastructures in Europe from projects such as e-FISCAL, e.inventory and ERINA+ .
So how do developers and architects of software and e-infrastructures know when their work is successful? And, more importantly, how does the outside world (especially funders) recognise exactly who collaborated and contributed to each and every scientific output? For the developer of an open source physics simulation, possible metrics could include recording downloads and users, as well as tracking paper citations. However, Katz suggests that what could be more valuable but harder to monitor are the number of citations each paper that cited your software receives. He emphasized the need to develop a system where credit is given to those involved in building the software and data sets.
The NSF has proposed a number of solutions to this conundrum. One recommendation is that all products - including software, data sets and workflows, are registered. Next, a credit map is developed. The input would be the weighted list of contributors (both people and products) with the DOI as the final output. This could lead to transitive credit enabling developers to gather evidence for their applications' widespread usage. Product usage could also be automatically recorded by simply incorporating a general usage code into each software package. Commercial tools are generally tracked by the money they generate, but this doesn't help academics understand what tools were used for what outcomes, says Katz. Tokens could be provided as part of science grants. Users could then distribute tokens either during or after using the tools.
At the moment, it is largely up to e-infrastructures to not only demonstrate an active user base, but to monitor the quality of service, track their citations and understand usage patterns. Alexandre Bonvin described the ways in which WeNMR is keeping track of how their services have contributed to progress in structural biology. In 2012, there was a significant output from the project with 36 publications acknowledging WeNMR. Their analysis found the use of their e-infrastructure to be both application/portal and discipline dependent, with a large fraction of users seeming to only use specific tools for rather limited lifetimes (approximately 0.5 to 1 years). Bonvin suggests that e-infrastructures should only expect a fraction of potential users to actually use it at any given time.
Unfortunately, as researchers don't always acknowledge the full array of tools that helped them with their work, developing an effective reward system for those developers and architects working behind the scenes will be vitally important going forward.
Read more: Measuring the success of European e-infrastructures
This article was first published on the GridCast blog. You can read the original version, here.
