Globus Online has been making waves since it was first announced at SC10. iSGTW caught up with Steve Tuecke, co-PI for the Globus project at University of Chicago and Argonne National Laboratory's Computation Institute. Read on to find out what he had to say.
iSGTW: Where did the idea for Globus Online come from?
Steve Tuecke: About two years ago, Lisa Childers and Lee Liming from our group at the University of Chicago wrote a report on a bunch of interviews they did with users about their experiences on TeraGrid. One of the themes that kept jumping out was that as soon as users started going across multiple sites and trying to make things work across administrative and security domains, problems started cropping up. Users got stuck. So the idea for Globus Online came from asking how we could solve these users' end-to-end problems.
The reason for biting off the file transfer problem first was that it was a well-constrained problem that we could tackle. It's also a fairly valuable service we can offer quickly to existing users of TeraGrid resource providers and other HPC facilities. We are offering a Software-as-a-Service facility to provide file transfer functionality, and we also have people that can jump in when things fail. It is a service that we know our way around from previous efforts in this area, and that has a lot of interesting failure cases that we could help address. All those together made it a good bite-sized chunk of work we could pursue as a stepping stone into the bigger plan.
iSGTW: What sort of failure cases are we talking about?
Tuecke: There are a lot of transient failures in file transfers, such as networks that drop connections, or servers that go offline temporarily then come back. There are so many things like that that tend to happen pretty regularly and they tend to get fixed relatively quickly either automatically or by a systems administrator. But from the user's perspective, they are failures that the user needs to deal with, when all the user cares about is getting data to a machine. When a transfer fails, there is often no apparent reason why. There are a lot of errors like that where trying again automatically can deal with it.
And then there's a whole class of errors, like a misconfigured firewall, that may cause transfers to intermittently fail, or a site that is running an old version of GridFTP that has a bug. From the user's perspective these errors are often a complete mystery. With our experts behind the Globus Online service, we can intervene on the user's behalf to fix these sorts of infrastructure issues so that things just work for the user. So that's what we're trying to tackle in the first version of Globus Online.
iSGTW: You mentioned the bigger plan earlier, and now you're describing this as the first version of Globus Online. What's in store for Globus Online users?
Tuecke: There are all sorts of challenges that typical scientists face in dealing with their large data sets, running the analysis against that data, sharing that data with other collaborators, and so on. Our plans are essentially to step up from where we are now into providing software-as-a-service that can incrementally address more and more issues that scientists run into.
iSGTW: Where would the funding for providing those services come from?
Tuecke: Near-term we have funding from a variety of sources including University of Chicago, Department of Energy, National Science Foundation, and National Institutes of Health. Longer-term, we're still working through ideas for how to make this a more sustainable offering. Remember that part of our strategy is not necessarily to be a resource provider but to be a facilitator of use of resource providers, which is a much less expensive proposition. For example, with Globus Online, the data never moves through us, so we never have to pay for storage or for forwarding all of the data that's being transferred around. It's the requests and the management of those movements that are channeling through us, which is a much less resource-intensive activity.
iSGTW: Some of the ideas you're describing, such as collaboration tools and serving as an access point to existing resource providers such as Open Science Grid, have been done by existing platforms such as HUBzero. Do you have plans to leverage or partner with any particular platforms?
Tuecke: We're not going to rewrite everything. That's silly. The challenge is to figure out what is an interesting set of existing software systems out there that one could integrate together and augment in interesting ways to create an ecosystem of software-as-a-service applications and capabilities.
At the GlobusWORLD conference, one of the things I talked about is our work in evolving Globus Online to offer more of a platform, with capabilities that other sites can latch into very easily. By treating Globus Online as an underlying platform or service that they can leverage, they can enhance their ability to deliver their value to their users. So I think you'll see a lot more of that kind of thing happening.
HUBzero, for example, provides a lot of value to some communities, such as nanoHUB for nanoscience, by giving them easy access to simulations that can run on TeraGrid and OSG resources. Reinventing all that doesn't make sense. But making it so that there's value that we can provide to HUBzero and vice versa, so that they become part of an ecosystem of capabilities that work much more seamlessly together… that does make sense, and that's part of our direction for Globus Online.
iSGTW: Back to the present. When Globus Online was first announced, my knee-jerk reaction was, "All it does is transfer data? What's so great about that?" Obviously, as we've discussed, data transfer is just your first conquest. But I also gather from what you've said so far that the apparently simple task of transferring data is actually quite complex.
Tuecke: Right. It should be a simple task. But moving a lot of data across high-speed research networks is still a real pain for a lot of users. We've spent a lot of years working with various big science communities such as high energy physics to make it easy within their specialized networks. Now we are trying to make it easy for everybody else.
For example, one thing we're doing with Globus Online is intelligently using the features of GridFTP. How you tune your GridFTP transfer can make a huge difference in your end-to-end performance. The challenge is that in the past you had to be a real expert to effectively use those tuning features. We now do that automatically in Globus Online, so that real users can get great transfer bandwidth easily - often an order of magnitude faster than un-tuned GridFTP or other common tools such as scp.
iSGTW: I noticed that several GlobusWORLD talks suggested a huge increase in transfer speed over other tools, including globus-url-copy. How do you achieve that speed-up?
Tuecke: It's a combination of a few things. Globus Online uses GridFTP, which supports a lot of optimizations that other tools like scp and rsync do not, and which are critical for getting good transfer rates on high-speed networks. Globus Online automates the use of these special features, based on our experience of what works in the real world with GridFTP, to give that good performance out of the box. Globus Online has also implemented some optimizations that are not widely supported in other tools, including globus-url-copy. So in practice, the end user gets the benefit of a highly tuned system without having to deal with any of the details.
It's not as if we've done something in Globus Online that can't be done by others. Certainly, if you look at the high energy physics grids and other such community-specific systems that have been built in recent years, they get the sort of bandwidths we see with Globus Online because they've gone to considerable lengths to tune their custom systems for their networks. But there aren't many people that bother doing the tuning themselves because it's hard. You've got to know a lot of stuff that a typical scientist doesn't, and shouldn't have to, know. That's the point of this system: to make it so that things just work.
iSGTW: Couldn't you have done this as a client or application or plug-in to GridFTP instead of a service?
Tuecke: Globus Online is a client to GridFTP, but of course it is not a "thick client" that we distribute and that users run themselves. A thick client could be written that could do some of what Globus Online does. But we have an advantage that should allow our autotuning to continue getting better and better because we have a larger purview over the data transfers. We're not just looking at one transfer; we can look at the collective of what's going on.
iSGTW: The autotuning isn't intelligent, or adaptive. It doesn't learn. It's improving because your human developers can learn from the transfers made using the website or the RESTful interface, right?
Tuecke: Yes. At the moment the autotuning is mostly based on knowing the sizes of the files you're transferring. The tuning depends on whether you're transferring big files or little files or a mixture of file sizes. So we're smart about that, including adjusting tuning for different file sizes even within a single large transfer. Tuning also depends on the nature of the network you're going over. We do a little to factor that in currently, and that's something we're adding even more to. So it's going to get even better. We're already to the point where we do better than most people unless they really get how to tune this sort of stuff for high speed networks, and we know we can do even better as we get smarter.
iSGTW: Sounds promising. So what's in the near future for Globus Online - what's the next feature you hope to roll out?
Tuecke: I'm more concerned about getting the word out near-term on what we've got today, the value we can provide today. At GlobusWorld I presented a set of enhancements we are considering for Globus Online file transfer in the coming months, and asked for feedback to help us prioritize our next steps; you can get the slides from my talk at http://www.globusworld.org. And you'll definitely be hearing more announcements from us on new stuff as it comes out.