Ian Foster

What's faster--a supercomputer or EC2?

Ed Walker wrote a nice article last year in which he used the well-known NAS parallel benchmarks to compare the performance of a commercial infrastructure-as-a-service offering (Amazon EC2) with that of a high-end supercomputer (the National Center for Supercomputing Applications "Abe" system). Not surprisingly, the supercomputer was faster. Indeed, it was a lot faster, due primarily to its superior interprocessor interconnect. (The NAS benchmarks, like many scientific applications, perform a lot of communication.)

However, before we conclude that EC2 is no good for science, I'd like to suggest that we consider the following question: what if I don't care how fast my programs run, I simply want to run them as soon as possible? In that case, the relevant metric is not execution time but elapsed time from submission to the completion of execution. (In other words, the time that we must wait before execution starts becomes significant.)

For example, let's say we want to run the LU benchmark, which (based on the numbers in Ed's paper) when run on 32 processors takes ~25 secs on the supercomputer and ~100 secs on EC2. Now let's add in queue and startup time:

• On EC2, I am told that it may take ~5 minutes to start 32 nodes (depending on image size), so with high probability we will finish the LU benchmark within 100 + 300 = 400 secs.
• On the supercomputer, we can use Rich Wolksi's QBETS queue time estimation service to get a bound on the queue time. When I tried this in June, QBETS told me that if I wanted 32 nodes for 20 seconds, the probability of me getting those nodes within 400 secs was only 34%--not good odds.

So, based on the QBETS predictions, if I had to put money on which system my application would finish first, I would have to go for EC2.

Here is a more detailed plot showing cumulative probability of completion (the Y-axis) as estimated by QBETS as a function of time since submission (the X-axis). We see that the likelihood of my application competing on EC2 is zero until around 400 seconds, when it rapidly rises to one. For the supercomputer, the probability rises more slowly, peaking at around 0.97. (I would think that the fact that the supercomputer estimate does not reach 1  relates to the lack of data available to QBETS for  long duration predictions.)

Note that in creating this graph, I do not account for application-dependent startup time on the supercomputer or for any variability in the startup time of the EC2 instances. (Looking at Cloudstatus, the latter factor seems to be relatively minor.)

This result reflects really just the scheduling policies (and loads) that the two systems are subject to. Supercomputers are typically scheduled to maximize utilization. Infrastructure as a service providers presumably optimize for response time.

Nevertheless, these data do provide another useful perspective on the relative capabilities of today's commercial infrastructure-as-a-service providers and supercomputer centers.

A critique of "Using Clouds to Provide Grids..."

The authors of a recent OGF document, "Using Clouds to Provide Grids Higher Levels of Abstractions and Explicit Usage Modes" make several assertions with which I take exception:

1) "There is a level of agreement that computational Grids have not been able to deliver on the promise of better applications and usage scenarios."

It is fascinating to watch the Gartner hype cycle in action, if sad to see people stuck in the trough of disillusionment. But the fact is, fortunately, that there are substantial grid projects and applications that are having substantial success. Ones that come immediately to mind are the Earth System Grid, cancer Biomedical Informatics Grid, and the LIGO Scientific Collaboration, but as it was yesterday that the LHC was switched on, we should also recall the remarkable successes of the LHC Computing Grid and its partner projects such as Open Science Grid. At a different level, Globus people will be happy to talk about the millions of files moved via GridFTP every day, and Miron Livny will be happy to talk at length about how many millions of CPU hours are delivered every day via Condor.

2) To address this purported lack of success, "there is a need to expose less detail and provide functionality in a simplified way. If there is a lesson to be learned from Grids it is that the abstractions that Grids expose – to the end-user, to the deployers and to application developers – are inappropriate and they need to be higher level."

No evidence is provided for this assertion that complex interfaces are the reason for the difficulties people have with grids. I argue that the issues are more complex.

First, the interfaces themselves are not, in my view, a significant issue. We can argue whether we prefer REST or Web Services, or say Nimbus (a grid virtualization interface) or EC2 (a cloud virtualization interface), but the differences among these alternatives are not great.

On the other hand, the economic systems that apply in the two cases are extremely different:

• Amazon services are designed to support the masses, they have no political constraints on who they can provide service to, and their charging model provides strong return to scale; thus, Amazon can focus on, and succeed in providing, modest-scale, reliable, on-demand service to many.
• TeraGrid (to use a US example) is designed to support a small number of extreme computing users, with a negative return to scale (the more users, the more work for fixed budget); thus, they are not motivated to provide virtualization solutions or to operate highly reliable remote access interfaces.

The implications of these different foci for users are tremendous. On EC2, I give my credit card and start a VM--a few seconds. On TeraGrid, I request an allocation (which may not be granted!), get an account, submit a request to run a job (they won't allow me to start a VM), wait in the queue--a many week process. Furthermore, I sometimes find that the remote access interfaces fail because keeping them running is not high priority.

This alternative perspective is I think more revealing about the sources of the differences and the ways we might address them. If we want on-demand, high-quality, compute and storage services, then we need either to create an economic system in which academic providers are motivated to provide such services, or decide to outsource to industry.

The importance of higher-level interfaces is a separate issue. Yes, tools like Hadoop and Swift for data analysis, Introduce for service authoring, Taverna for service composition are important and necessary. Yes, we should be hoping to leverage and influence work done in the far larger corporate market to our advantage. (A focus of the upcoming CCA workshop.)

3) "Grids as currently designed and implemented are difficult to interoperate." The authors make a big deal of this point, but it is not clear to what purpose.

It is true that interoperation is not automatic. [If only everyone used Globus software, then all would be well :) --although of course the policy issues would remain!]. But I am not sure that this is a significant problem for users, or hard to achieve when it is needed. E.g., the caBIG team recently demonstrated a gateway to TeraGrid. The LHC Computing Grid integrates resources worldwlde. Etc. Most users never ask about interoperability, in my experience.

Argonne is Number One ...

It is worth noting that Argonne's IBM Blue Gene system is officially the fastest supercomputer worldwide for open science, according to the June 2008 Top 500 rankings. It is also third fastest overall, after systems at LANL and LLNL used for nuclear weapons work. Many kudos to Ray Bair, Pete Beckman, and the rest of the team at the Argonne Leadership Computing Facility. (And to IBM too ...)

Some of my colleagues, notably Ioan Raicu, Zhao Zhang, and Mike Wilde, have been running large "many task" applications in biology and economics on this system, using Falkon and Swift. So far we're up to 32K of the total 160K cores, and it's a delight to work with.

reCAPTCHA: stop spam read books

I'm back from a longish vacation on Lake Huron, North Manitou Island, and other pleasant places. I think I'll try starting up my blog again.

Here's a fascinating follow up to my post a while back on open source problem solving in science: Luis von Ahn's latest project that uses fragments of old text as CAPTCHAs. Thus, when typing those irritating words you have to type to prove you are human you are helping transcribe old books. Brilliant.

Iphone

In a rare moment of technological enthusiasm, I bought an iPhone. I am learning about fun things to do with it. (No, it seems to be a great device, although it's not yet clear to me that it is better than  my Blackberry.)

Article describes U.Chicago TeraPort

An article today in the University of Chicago Chronicle talks about our TeraPort cluster, which while a humble system, is of great importance to us in the Computation Institute. As well as the usual applications from the physical and biological sciences, we have less conventional users who can comment that, for example:

“Initial work on lexical choices in this collection has revealed striking differences in the word use and sense between male and female authors and characters, as well as American and non-American authors.” [Mark Olsen and his co-authors in a paper on data mining of black drama.]

University of Canterbury buys Blue Gene

My alma mater, the University of Canterbury in New Zealand, just announced that they will acquire a 4,096 CPU IBM Blue Gene (BG/L) supercomputer. This will give New Zealand a second entry in the Top 500 list, well ahead of Weta Digital. (Funnily, the local paper in Christchurch reported that they were acquiring a 130,000 CPU, 270 teraop/sec Blue Gene. That would have been something.)
I'm off to New Zealand the first week of July for the KAREN Forum in Auckland--basically an eScience conference, organized by the New Zealand advanced network organization.

And then there were five ...

Following my post about the four European middleware platforms, I am told about a fifth, ExtreemOS. "The XtreemOS system will offer an alternative to the Globus toolkit, which is currently the most widespread middleware system." A noble goal!

ExtreemOS will extend Linux with native (kernel-level) support for virtual organizations. I like the concept of building support for grid execution in at the lowest levels, so that every computer, whether desktop or palmtop, is ready to participate in collaborative activities. On the other hand, I am not at all sure what form that support should take--or even if we need more than is already provided in today's kernel. I would think that a working hypothesis should be that we don't need any.

Interestingly, Red Hat announced plans to integrate Condor scheduling capabilities into their Linux distribution. However, their goals are fairly narrow, focused on facilitating the use of remote computers for computation.

Zettabytes!

A recent IDC report claims that:

• The world created 161 exabytes (1.6 x 10^20 bytes) of digital data in 2006
• By 2010, annual data output will reach one zettabyte (1 x 10^21 bytes)
• In 2006, there were one billion devices capable of capturing digital images

The 2003 Berkeley study estimated 5 exabytes were produced in 2002, and a 30% annual growth rate. Thus the IDC estimates are considerably higher. The reason is that IDC takes a much broader definition of "created", including digital cameras, local copies of data, etc.

MidWest Grid Workshop

The MidWest Grid Workshop will be held at the University of Illinois in Chicago on March 24 and 25. From the Web page

The aim of the workshop is to give the students a basic foundation in distributed computing, and valuable hands-on training in computing techniques. The workshop introduces essential skills that will be needed by students in the natural and applied sciences, engineering, and computer science to conduct and support scientific analysis in the emerging grid computing environment.

Participants will work with some of the world's leading experts in grid computing, through a blend of lectures, discussions and hands-on computing exercises on large-scale grid hardware and software resources.