Nowadays, cloud computing is not only a buzzword, but also an increasingly popular way to provision computational and storage resources as on-demand services. With cloud computing, research communities can benefit from flexible, rapidly scalable, easy-to-use, and often cheap computing services.
Currently, public clouds account for the majority of the marketed resources, but their infrastructures are not always a viable or desirable choice for scientists and other researchers. After all, in a research context, sensitive data may have to stay within institutional boundaries, big data may be too expensive to move, or the speed and quality of long distance data transfer may be unsatisfactory.
As a consequence, many research institutions are studying ways to provision cloud and traditional services by means of virtualized resources to satisfy the evolving requirements of their research communities. Some of these institutions are reporting success stories that describe efficient provisioning and satisfied users. Nonetheless, an inconvenient truth is shared among the upholders of research-oriented cloud computing: very few academic institutions, if any, can offer that illusion of infinite resource availability that is so critical to enjoying many of the benefits promised by cloud computing.
One way forward is to federate cloud infrastructures across multiple institutions to substantially increase the total amount of available resources. Admittedly, this is not an easy thing to do. Deploying a federation of cloud infrastructures poses multiple challenges, ranging from a generalized technological immaturity and fragmentation to the difficulty of coordinating a vast number of heterogeneous resource providers.
European Grid Infrastructure (EGI) created the Federated Clouds Task Force in 2011 within the EGI-InSPIRE project. The mandate of the task force is to define a profile for a federation of clouds and to deploy a federation test bed. Following the success of its first year of activities, EGI has recently transformed the task force into a task of EGI-InSPIRE, while maintaining its original mandate.
A number of critical choices had to be made when setting up the task force. Exemplar use cases were collected and analyzed to define the set of capabilities that had to be implemented in a federation of clouds. This reaffirmed the need for the task force to be user-led, as well as the importance of taking an incremental approach to the creation of the federation test bed. The goal of the team is to foster a progressive aggregation of resources and technologies until a federation that is functional for the targeted user communities is deployed. Overall, this process will be spread over two years, but it started with an early deployment of a minimal set of capabilities that could be utilized by the users right away.
Another important decision has been not mandating the installation of specific cloud technologies by the resource providers. This approach allows freedom for each resource provider to develop in-house solutions and several resource providers affiliated with the task force have decided to deploy multiple types of cloud management platforms, such as OpenStack and OpenNebula, as well as their own home-bred solutions, such as WNoDeS and Okeanos.
The technological agnosticism endorsed by the task force also reflects a commitment to a thin federation layer, based on standardized interfaces and a pool of centralized services. The adoption of interfaces implementing the Open Cloud Computing Interface and Cloud Data Management Interface standards is necessary to avoid interface fragmentation. This is a well-known problem with cloud computing and is similar to issues suffered by grid computing not too long ago. Centralized services are used to guarantee a unified experience to the users in terms of accessing information about the available resources, their status, the amount of resources utilized, and the images from which virtual machines can be created.
Some of the central services have been implemented by reusing or extending part of the technologies adopted within the EGI, but others have required completely new solutions. So, alongside mature services already in production use by EGI, such as BDDI, Nagios and APEL there also new entries like Marketplace and rOCCI. The reuse of technology allows for the integration of cloud resources within EGI. Clearly, cloud computing is not meant to replace grid computing or HPC but, instead, is seen as an opportunity to offer new types of resources and services to the end users, the developers of domain-specific platforms, and to resource providers.
A thin federation layer allows for each institution to retain a degree of control over the contributed resources. This is a critical requirement in any inter-institutional collaboration which may cross borders and where substantial policy and agenda differences are to be expected. The task force made no attempt to enforce a set of top-down policies beyond those already established by the EGI community around acceptable use and security policies. Instead, a bottom-up approach has been endorsed where resource providers are free to contribute the amount of resources they choose while adopting the security and management policies they prefer.
At the present time, the Federated Cloud Task Force includes twenty-six institutions and sixteen European countries. Nine resource providers are contributing to the test bed and up to five different cloud management platforms are interoperating at interface and service levels. The first working demo of the federation test bed was successfully showcased after six months from the beginning of the activities. Currently, after a year, the test bed deployed by the task force is being used by two user communities, with six more waiting for support. The integration of the test bed into the existing EGI infrastructure is underway and the next demo is planned for the upcoming EGI Community Forum to be held in Manchester from 8-12 April, 2013.