Ian Foster

Globus Online: A cloud-based managed file transfer service

The Globus team showcased Globus Online, our new cloud-based managed file transfer service, at the SC conference in New Orleans last week. Hundreds of people came by our booth to sign up. I don't think it was just the free T-shirts: people seemed really interested. Thus, I outline here the rationale for Globus Online's development, and provide a few notes on its design and implementation.

Rationale

Growing up in New Zealand, I heard endless repeats of the Goon Show. In one episode, hero Neddy Seagoon is offered five pounds to move a piano from one room to another. It turns out that one room is in France and the other in England, so it is a more difficult task than Neddy anticipated. (He ends up sailing the piano across the Channel.)

Moving data has that flavor: it can sound trivial, but in practice is often tedious and difficult. Datasets may have complex nested structures, containing many files of varying sizes. Source and destination may have different authentication requirements and interfaces. End-to-end performance may require careful optimization. Failures must be recovered from. Perhaps only some files differ between source and destination. And so on.

Many tools exist to manage data movement: RFT, FTS, Phedex, rsync, etc. However, all must be installed and run by the user, which can be challenging for all concerned. Globus Online uses software-as-a-service (SaaS) methods to overcome those problems. It's a cloud-hosted, managed service, meaning that you ask Globus Online to move data; Globus Online does its best to make that happen, and tells you if it fails. 

Design

The Globus Online a service can be accessed via different interfaces depending on the user and their application:

• A simple Web UI is designed to serve the needs of ad hoc and less technical users
• A command line interface exposes more advanced capabilities and enables scripting for use in automated workflows
• A REST interface facilitates integration for system builders who don't want to re-engineer file transfer solutions for their end users

All three access methods allow a client to:

• establish and update a user profile, and specify the method(s) you want to use to authenticate to the service;
• authenticate using various common methods, such as Google OpenID or MyProxy providers;
• characterize endpoints to/from which transfers may be performed;
• request transfers;
• monitor the progress of transfers; and
• cancel active transfers  

Having authenticated and requested a transfer, a client can disconnect, and return later to find out what happened. Globus Online tells you which transfer(s) succeeded and which, if any, failed. It notifies you if a deadline is not met, or if a transfer requires additional credentials.

The Globus Online Web frontend, in the best traditions of Web 2.0, uses Javascript to provide a rich user experience, forwarding requests to the Globus Online service back-end. 

Globus Online REST requests are of course simple HTTP GETs and POSTs, with the destination URL indicating the requested operation and the body of the message containing any arguments. 

A command line interface (CLI) has long been valuable for client-side scripting, but requires installation of client-side libraries. What we call (somewhat tongue in cheek) CLI-2 supports client-side scripting with no client-side software installation. We achieve this behavior via a restricted shell, into which any user with a Globus Online account can ssh to execute commands. Thus, I can write 

ssh [email protected] scp alcf#dtn:/myfile nersc#dtn:/myfile

to copy myfile from source alcf#dtn to destination nersc#dtn. Two useful features are illustrated:

• Endpoints define logical names for physical nodes. For example, alcf#dtn denotes the data transfer nodes associated with the Argonne Leadership Computing Facility. Sites can publish their endpoints, and users can define their own endpoint names.
• The Globus Online scp command echoes the syntax of the popular scp (secure copy), thus facilitating access by scp users. It supports many regular scp options, plus some additional features--and is much faster because it is built on GridFTP.

There's more, including a powerful transfer command. I encourage you to browse the documentation.

Implementation

The two keys to successful SaaS are reliability and scalability. The service must behave appropriately as usage grows to 1,000 then 1,000,000 and maybe more users. To this end, we run Globus Online on Amazon Web Services. User and transfer profile information are maintained in a database that is replicated, for reliability, across multiple geographical regions. Transfers are serviced by nodes in Amazon's Elastic Compute Cloud (EC2) which automatically scale as service demands increase.

Next steps

We will support InCommon credentials and other OpenID providers in addition to Google; support other transfer protocols, including HTTP and SRM; and continue to refine automated transfer optimization, by for example optimizing endpoint configurations based on number and size of files.

Kudos

Kudos to the Globus Online team, which includes Bryce Allen, Joshua Boverhof, John Bresnahan, Ann Chervenak, Lisa Childers, Paul Dave’, Fred Dech, Ian Foster, Dan Gunter, Gopi Kandaswany, Nick Karonis, Raj Kettimuthu, Jack Kordas, Lee Liming, Mike Link, Stu Martin, JP Navarro, Karl Pickett, Mei Hui Su, Steve Tuecke, and Vas Vasiliadis.

What's faster--a supercomputer or EC2?

Ed Walker wrote a nice article last year in which he used the well-known NAS parallel benchmarks to compare the performance of a commercial infrastructure-as-a-service offering (Amazon EC2) with that of a high-end supercomputer (the National Center for Supercomputing Applications "Abe" system). Not surprisingly, the supercomputer was faster. Indeed, it was a lot faster, due primarily to its superior interprocessor interconnect. (The NAS benchmarks, like many scientific applications, perform a lot of communication.)

However, before we conclude that EC2 is no good for science, I'd like to suggest that we consider the following question: what if I don't care how fast my programs run, I simply want to run them as soon as possible? In that case, the relevant metric is not execution time but elapsed time from submission to the completion of execution. (In other words, the time that we must wait before execution starts becomes significant.)

For example, let's say we want to run the LU benchmark, which (based on the numbers in Ed's paper) when run on 32 processors takes ~25 secs on the supercomputer and ~100 secs on EC2. Now let's add in queue and startup time:

• On EC2, I am told that it may take ~5 minutes to start 32 nodes (depending on image size), so with high probability we will finish the LU benchmark within 100 + 300 = 400 secs.
• On the supercomputer, we can use Rich Wolksi's QBETS queue time estimation service to get a bound on the queue time. When I tried this in June, QBETS told me that if I wanted 32 nodes for 20 seconds, the probability of me getting those nodes within 400 secs was only 34%--not good odds.

So, based on the QBETS predictions, if I had to put money on which system my application would finish first, I would have to go for EC2.

Here is a more detailed plot showing cumulative probability of completion (the Y-axis) as estimated by QBETS as a function of time since submission (the X-axis). We see that the likelihood of my application competing on EC2 is zero until around 400 seconds, when it rapidly rises to one. For the supercomputer, the probability rises more slowly, peaking at around 0.97. (I would think that the fact that the supercomputer estimate does not reach 1  relates to the lack of data available to QBETS for  long duration predictions.)

Note that in creating this graph, I do not account for application-dependent startup time on the supercomputer or for any variability in the startup time of the EC2 instances. (Looking at Cloudstatus, the latter factor seems to be relatively minor.)

This result reflects really just the scheduling policies (and loads) that the two systems are subject to. Supercomputers are typically scheduled to maximize utilization. Infrastructure as a service providers presumably optimize for response time.

Nevertheless, these data do provide another useful perspective on the relative capabilities of today's commercial infrastructure-as-a-service providers and supercomputer centers.

World does not come to an end!

The Large Hadron Collider turned on today, and the world did not come to an end. Phew ...

Personal genome

I'm fascinated by Geoge Church's personal genome project (the name recalls another PGP), in which one volunteers to have one's genome sequenced and made available for research. Personally, I think it's a great thing and I would be delighted to participate. I can't imagine caring if I (or the world) know that I have the gene for obsessive-compulsive disorder or whatever I may turn out to have. The one thing that gives me pause is wondering whether my children or siblings should have a say in whether I publish my genome, given that they share a fair bit of it. (I don't think my parents would mind.) The PGP web site doesn't discuss that issue, although it does point out other dangers that hadn't occurred to me, e.g.:

[someone might] make synthetic DNA corresponding to the participant and plant it at a crime scene

A quick search didn't reveal too many profound thoughts on this topic, just some recognition that if is an issue. E.g., from Baylor College of Medicine:

With [personal genomes] will come a host of legal and ethical issues, said Amy McGuire, J.D., Ph.D., Assistant Professor of Medicine in BCM's Center for Medical Ethics and Health Policy.

"Sequencing a personal genome possibly will reveal information about children, parents and siblings," she said. At present, there are no real standards as to what control family members can have over sequencing of an individual's genome or its release.

And some thoughts on the opposite dimension--a physician's "duty to warn.'

Computation and Knowledge

I was invited to speak at the University of Chicago's Physics Colloquium. I foolishly agreed and then had to work out what I could possibly say that would be of interest to physicists. I figured that a talk on distributed computing wouldn't be too interesting. Instead, I gave a more general talk with the grandiose title "Computation and Knowledge":

I speak to the question of how computation can contribute to the generation of new knowledge by accelerating the work of distributed collaborative teams and enabling the extraction of knowledge from large quantities of information produced by many workers. I illustrate my presentation with examples of work being performed within the Computation Institute at the University of Chicago and Argonne National Laboratory.

They seemed to enjoy it, and I enjoyed preparing it.

BEVO Report on Virtual Organizations Released

The report from the two workshops that Jonathan Cummings, Tom Finhiolt, Carl Kesselman, Katherine Lawrence, and I ran (with much help from Diana Rhoten) on "Building Effective Virtual Organizations" is now available on the BEVO website. The title is "Beyond Being There: A Blueprint for Advancing the Design, Development, and Evaluation of Virtual Organizations"--and that is what the report provides.

We lead off with the delightful quote from Arthur Mee, writing in the Strand Magazine in 1898:

If as it is said to be not unlikely in the near future, the principle of sight is applied to the telephone as well as that of sound, earth will be in truth a paradise, and distance will lose its enchantment by being abolished altogether.

Of course things are not quite so simple (fortunately!)--but it is certainly true that the Internet has transformed the practice of science. This report explains why that it is so, reviews exemplar projects that illustrate the nature of this transformation, and explains what we need to do next to take virtual organizations to the next level.

Nice article on caBIG cancer biomedical informatics grid

I keep mentioning caBIG. But here is a nice profile in ComputerWorld that describes the project's goals and status.

Big enhancements to caGrid federation infrastructure

I've mentioned the cancer Biomedical Informatics Grid (caBIG) project before. It's goal is "to develop applications and the underlying systems architecture that connects together data, tools, scientists and organizations in an open federated environment." Underpinning caBIG is a service oriented infrastructure, caGrid, which is in turn built on Globus software.

The caBIG project announced today the release of version 1.2 of caGrid core infrastructure, with  enhancements such as

• A simplified Grid Transfer Service to handle large scale data transfer for grid-based queries
• Integration with the latest releases of Software Development Kit (SDK) and Common Security Module (CSM)
• Availability of a new Web based Single Sign On Framework, called WebSSO, for providing single sign on capability for web based applications integrated with caGrid GAARDs security infrastructure, based on collaboration with the caBIG Clinical Trials Suite (CCTS) team from the Clinical Trials Management Systems Workspace
• Substantial improvements to usability of tools such as Introduce Toolkit based on user feedbackI
• ntegration with Apache Ivy build system for improved dependency management of sub components within the caGrid core infrastructure
• Early preview to integration with the Taverna Workbench for developing workflows, based on collaboration with the Integrative Cancer Research Workspace
• Backward compatible with the caGrid 1.0 and caGrid 1.1

My colleagues Ravi Madduri and Wei Tan at the University of Chicago and Argonne National Laboratory have been very involved in this work, in particular the integration with Taverna.

On Interdisciplinary Work

I recently came across this quote from Théophile Gautier (1811–1872)'s poem L'Art:

Oui, l'oeuvre sort plus belle
D'une forme au travail
Rebelle,
Vers, marbre, onyx, émail.

[Yes, the work comes out more beautiful from a material that resists the process, verse, marble, onyx, or enamel (not email!)]

I think this summarizes well the merits of interdisciplinary work and the challenges and joys of system-level science.

Michael Turner on the National Laboratories

Our esteemed Chief Scientist has some interesting and inspiring things to say about the US Department of Energy's National Laboratories. Michael Turner led development of the report Connecting Quarks with the Cosmos, which is still a good read, at least for this computer scientist.