Last week, the 10th e-Infrastructure Concertation Meeting took place. The chief purpose of this meeting was to bring together European e-infrastructure projects in order to discuss issues related to the completion of the European Commission's 7th Framework Programme (FP7) and the start of an e-infrastructures activity during Horizon 2020 (2014-2020). The event, which was organized by the European Commission Services with the support of the e-ScienceTalk project, was held in Brussels, Belgium. The two day meeting brought together over 130 representatives from the e-infrastructure landscape, policy makers and funding agencies, and looked at the impact and outcomes from the FP7 funding programme for e-infrastructures.
One of the highlights of the event was a panel discussion, chaired by Sverker Holmgren of Uppsala University, Sweden, on the subject of projects' impact and the metrics by which their success is judged. Kostas Glinos, head of the European Commission's eInfrastructure unit, kicked off the discussion by talking about projects such as ERINA+, e-FISCAL and The European e-Infrastructures Observatory (e·nventory), which aim to assess the impact, success and funding structures of other European FP7 e-infrastructure projects. "How will they be sustained?" he asks. Obviously they are useful to the community, says Glinos, but how will they be sustained internationally and what are the appropriate bodies to look after and maintain them?
Another highlight of the event was a discussion session, hosted by Aniyan Varghese of the European Commission, on the development of the computational infrastructure of 21st century science.
Definitions and terminology were a major theme of the discussion, leading Richard Kenway of the University of Edinburgh to lament that the community "often ties itself up in knots with questions like what exactly we mean by HPC".
During the session, Ivan Kondov of the Karlsruhe Institute of Technology gave a short presentation on the marketing of computational infrastructure. He had this to say: "We have to understand demand in the language of end-users, because they don't necessarily know what HPC is. Researchers may not be aware of what e-infrastructures are available to them. Physicists, for example may not know about service-orientated architecture."
Robert Lovas of the International Desktop Grid Federation (IGDF) also gave a short presentation in which he argued that members of the public don't usually know what it is that "scientists in their ivory towers" are really doing. Consequently, he says, when members of the public wish to contribute their computers to volunteer desktop grids, they want to know exactly what it is they're researching. This is where marketing and communication is important, he adds.
Later on in the session, Steven Newhouse spoke about who does and doesn't use the e-infrastructures provided, with particular reference to 'the long tail' in terms of scientific users. "The regimented structure you get in high-energy physics and PRACE just doesn't exist in the long-tail... we need software to deal with this challenge," he says. "Over the last 20 years, we've been guilty of too-often adopting a one-size-fits-all approach… but this just suited the high-energy physics community."
Finally, Federico Ruggieri of CHAIN-REDS spoke about some of the difficulties relating to the project-driven funding model of e-infrastructure initiatives. "Public institutions are frequently receiving project-driven funding, which means it is difficult to fund long-term contracts with commercial cloud providers," says Ruggieri. "Projects can easily have funding gaps and the question is what happens to your resources and your data when there's such a gap?"
Bob Jones, head of CERN openlab, picked up on the subject of sustainability in this context and suggested that perhaps a long-term service model, rather than a short-term project model, may be better suited to providing the assessment services offered by the three projects mentioned by Glinos. The notion of such a move was welcomed by other participants in the discussion. However, Glinos pointed out that providing such services over a long term wasn't necessarily the role of the European Commission.
Jorge-Andres Sanchez-Papaspiliou agreed with Jones on the issue of shifting towards a service model rather than a project model for long-term assessment of the success or impact of European e-infrastructure projects. He is project inventor of The European e-Infrastrucures Observatory, which aims to create a single-entry-point, or "one-stop-shop data warehouse", capable of representing European e-infrastructure benchmarks and achievements. The project seeks to disseminate information through intuitive, interactive and user-friendly visualisation interfaces to allow for progress monitoring and impact assessment of e-infrastructures at both regional and national levels across the European Union and beyond. Sanchez-Papaspiliou also stressed the importance of assessing the impact of projects on a global scale. "We need to compare ourselves to what the US is doing, what China is doing, and what Russia is doing," he says. "This means we need more collaboration with these regions."
"There is a worrying belief by politicians that if you buy and invest in infrastructure, that creates economic growth," says Mark Parsons of the Edinburgh Parallel Computing Centre. "But, of course, it's not that act that creates anything at all - it just spends money. The thing that creates the economic growth and develops the new products and services is the scientists and researchers that use that infrastructure."
Value and timescales
"I think it's very difficult for the European Commission to measure the real impact of what we do with research infrastructures, because they're actually measuring the wrong people to some extent," says Parsons. "There's been very little real measurement of the outcomes for the people and the organizations using this infrastructure." Parsons also argues that the timescale across which impact can occur makes things extremely difficult, pointing out that work from a particular project may be picked up and have value many years after a project has finished.
"We have to consider the fact that much of the impact which projects seek to achieve can never be achieved in within the short timeframe of a project," says Andrea Manieri, echoing Parsons's sentiments. Manieri is the project director of ERINA+, which was set up to assess the socio-economic impact of e-infrastructures and develop an assessment methodology for European e-Infrastructure projects to self-evaluate their own impact. "Impact may happen two, or three, or more years after the end of a project," he says. "So, it is important to consider the analysis of any metrics over the longer period... both projects and EC need to carefully address how impact data is used."
One set of metrics to rule them all?
During the discussion, Glinos stressed the importance of having tailored metrics by which to judge individual projects, so as to ensure that metrics correspond to the objectives of the project in question. Despite this, he also went on to propose the idea of having some metrics which are common across projects, so that they can easily be compared and conclusions can be drawn. He emphasized the importance of such metrics in justifying the existence of e-infrastructures at a European level, rather than, say, simply relying on commercial providers or that infrastructure which is provided at a national level. Equally, such standardized metrics are important for showing the impact of the funding body itself, he argues, since without demonstrating its own success, the unit can't campaign effectively on behalf of the e-Infrastructure community for future funding.
However, several participants expressed some measured concern regarding this suggestion. Erwin Laure, director of the PDC Center for High Performance Computing at the Swedish Royal Institute of Technology, says: "The services that the various projects are providing are so different and have different impact that needs to be measured differently - that is a big challenge that the ERINA+ project has been finding." He argues that it is very difficult to come up with metrics which can be used widely to compare projects with one another. "I think it's a very dangerous route to go down," he says.
Fotis Karagiannis, project director of e-FISCAL, which analyses the costs and cost structures of the European high-throughput and high-performance computing e-Infrastructures, agrees with Laure. "In my experience, it's just not possible to have the same metrics for different projects, because the types of users are different," says Karagiannis. He cites the differing users of the European Middleware Initiative versus the GÉANT network, as a good example of this, arguing that this makes direct comparison using any one indicator a futile exercise.
Finally, Steven Newhouse, director of the European Grid infrastructure (EGI.eu), wrapped up the discussion by proposing the creation of a body to coordinate e-infrastructures in Horizon 2020. "If the European Commission is serious about coordinating e-Infrastructures, they need to think about how to put a body in place to do this, supported by the member states, and that can do things like collation of metrics; long-term development of policy, in a manner similar to what e-IRG has been doing; and provide the long-term coordination of the e-infrastructures."