Ian Foster

What's faster--a supercomputer or EC2?

Ed Walker wrote a nice article last year in which he used the well-known NAS parallel benchmarks to compare the performance of a commercial infrastructure-as-a-service offering (Amazon EC2) with that of a high-end supercomputer (the National Center for Supercomputing Applications "Abe" system). Not surprisingly, the supercomputer was faster. Indeed, it was a lot faster, due primarily to its superior interprocessor interconnect. (The NAS benchmarks, like many scientific applications, perform a lot of communication.)

However, before we conclude that EC2 is no good for science, I'd like to suggest that we consider the following question: what if I don't care how fast my programs run, I simply want to run them as soon as possible? In that case, the relevant metric is not execution time but elapsed time from submission to the completion of execution. (In other words, the time that we must wait before execution starts becomes significant.)

For example, let's say we want to run the LU benchmark, which (based on the numbers in Ed's paper) when run on 32 processors takes ~25 secs on the supercomputer and ~100 secs on EC2. Now let's add in queue and startup time:

• On EC2, I am told that it may take ~5 minutes to start 32 nodes (depending on image size), so with high probability we will finish the LU benchmark within 100 + 300 = 400 secs.
• On the supercomputer, we can use Rich Wolksi's QBETS queue time estimation service to get a bound on the queue time. When I tried this in June, QBETS told me that if I wanted 32 nodes for 20 seconds, the probability of me getting those nodes within 400 secs was only 34%--not good odds.

So, based on the QBETS predictions, if I had to put money on which system my application would finish first, I would have to go for EC2.

Here is a more detailed plot showing cumulative probability of completion (the Y-axis) as estimated by QBETS as a function of time since submission (the X-axis). We see that the likelihood of my application competing on EC2 is zero until around 400 seconds, when it rapidly rises to one. For the supercomputer, the probability rises more slowly, peaking at around 0.97. (I would think that the fact that the supercomputer estimate does not reach 1  relates to the lack of data available to QBETS for  long duration predictions.)

Note that in creating this graph, I do not account for application-dependent startup time on the supercomputer or for any variability in the startup time of the EC2 instances. (Looking at Cloudstatus, the latter factor seems to be relatively minor.)

This result reflects really just the scheduling policies (and loads) that the two systems are subject to. Supercomputers are typically scheduled to maximize utilization. Infrastructure as a service providers presumably optimize for response time.

Nevertheless, these data do provide another useful perspective on the relative capabilities of today's commercial infrastructure-as-a-service providers and supercomputer centers.

There's Grid in them thar Clouds*

You’ve probably seen the recent flurry of news concerning “Cloud computing.” Business Week had a long article on it (with an amusing and pointed critique here). Nick Carr has even written a book about it. So what is it about, what is new, and what does it mean for information technology?

The basic idea seems to be that in the future, we won’t compute on local computers, we will compute in centralized facilities operated by third-party compute and storage utilities. To which I say, Hallelujah, assuming that it means no more shrink-wrapped software to unwrap and install.

Needless to say, this is not a new idea. In fact, back in 1960, computing pioneer John McCarthy predicted that “computation may someday be organized as a public utility”—and went on to speculate how this might occur.

In the mid 1990s, the term grid was coined to describe technologies that would allow consumers to obtain computing power on demand. I and others posited that by standardizing the protocols used to request computing power, we could spur the creation of a computing grid, analogous in form and utility to the electric power grid. Researchers subsequently developed these ideas in many exciting ways, producing for example large-scale federated systems (TeraGrid, Open Science Grid, caBIG, EGEE, Earth System Grid, …) that provide not just computing power, but also data and software, on demand. Standards organizations (e.g., OGF, OASIS) defined relevant standards. More prosaically, the term was also co-opted by industry as a marketing term for clusters. But no viable commercial grid computing providers emerged, at least not until recently.

So is “cloud computing” just a new name for grid? In information technology, where technology scales by an order of magnitude, and in the process reinvents itself, every five years, there is no straightforward answer to such questions.

Yes: the vision is the same—to reduce the cost of computing, increase reliability, and increase flexibility by transforming computers from something that we buy and operate ourselves to something that is operated by a third party.

But no: things are different now than they were 10 years ago. We have a new need to analyze massive data, thus motivating greatly increased demand for computing. Having realized the benefits of moving from mainframes to commodity clusters, we find that those clusters are darn expensive to operate. We have low-cost virtualization. And, above all, we have multiple billions of dollars being spent by the likes of Amazon, Google, and Microsoft to create real commercial grids containing hundreds of thousands of computers. The prospect of needing only a credit card to get on-demand access to 100,000+ computers in tens of data centers distributed throughout the world—resources that be applied to problems with massive, potentially distributed data, is exciting! So we’re operating at a different scale, and operating at these new, more massive scales can demand fundamentally different approaches to tackling problems. It also enables—indeed is often only applicable to—entirely new problems.

Nevertheless, yes: the problems are mostly the same in cloud and grid. There is a common need to be able to manage large facilities; to define methods by which consumers discover, request, and use resources provided by the central facilities; and to implement the often highly parallel computations that execute on those resources. Details differ, but the two communities are struggling with many of the same issues.

Unfortunately, at least to date, the methods used to achieve these goals in today’s commercial clouds have not been open and general purpose, but instead been mostly proprietary and specialized for the specific internal uses (e.g., large-scale data analysis) of the companies that developed them. The idea that we might want to enable interoperability between providers (as in the electric power grid) has not yet surfaced. Grid technologies and protocols speak precisely to these issues, and should be considered.

A final point of commonality: we seem to be seeing the same marketing. The first “cloud computing clusters”—remarkably similar to the “grid clusters” of a few years ago—are appearing. Perhaps Oracle 11c is on the horizon?

What does the future hold? I will hazard a few predictions, based on my belief that the economics of computing will look more and more like those of energy. Neither the energy nor the computing grids of tomorrow will look like yesterday’s electric power grid. Both will move towards a mix of microproduction and large utilities, with increasing numbers of small-scale producers (wind, solar, biomass, etc., for energy; for computing, local clusters and embedded processors—in shoes and walls?) co-existing with large-scale regional producers, and load being distributed among them dynamically. Yes, I know that computing isn’t really like electricity, but I do believe that we will nevertheless see parallel evolution, driven by similar forces.

In building this distributed “cloud” or “grid” (“groud”?), we will need to support on-demand provisioning and configuration of integrated “virtual systems” providing the precise capabilities needed by an end-user. We will need to define protocols that allow users and service providers to discover and hand off demands to other providers, to monitor and manage their reservations, and arrange payment. We will need tools for managing both the underlying resources and the resulting distributed computations. We will need the centralized scale of today’s cloud utilities, and the distribution and interoperability of today’s grid facilities.

Some of the required protocols and tools will come from the smart people at Amazon and Google. Others will come from the smart people working on grid. Others will come from those creating whatever we call this stuff after grid and cloud. It will be interesting to see to what extent these different communities manage to find common cause, or instead proceed along parallel paths.

*An obscure cultural reference: the phrase “There’s gold in them thar hills” was first uttered, according to some, by an old prospector in the 1948 movie “Treasure of the Sierra Madre”, starring Humphrey Bogart.

Celebrating Licklider: human-computer symbiosis 50 years on

I recently completed an article for the HPC conference held in Cetraro, Italy, in 2006. (Ok, I was a little late.) I took the opportunity to talk about how a topic that I find fascinating, namely the vision and legacy of JCR Licklider. The abstract:

Licklider advocated in 1960 the construction of computers capable of working symbiotically with humans to address problems not easily addressed by humans working alone. Since that time, many of the advances that he envisioned have been achieved, yet the time spent by human problem solvers in mundane activities remains large. I propose here four areas in which improved tools can further advance the goal of enhancing human intellect: services, provenance, knowledge communities, and automation of problem-solving protocols.

Needless to say, I could hardly do justice to such a grand topic, but perhaps my comments will spur some interesting thoughts and responses.

Red Shift Meets Event Horizon

Sun's CTO Greg Papadopoulos coined the term "red shift" to denote the massive IT buildout that is occurring in the likes of Google, Amazon, eBay, and the like as they provide ever-more-sophisticated services to ever-larger numbers of customers. He posits that this trend will continue, ultimately resulting in a "neutron star collapse of datacenters" to a small number of massive, centralized, highly efficient providers.It's a bizarre choice of term--doesn't a bigger red shift mean that the star we are seeing is further away and thus older (and probably already extinct)?--but it's certainly a compelling analysis.

I liked an article by Phil Wainewright (riffing on an article by Dan Farber about red shift theory) about the pros and cons of ultra-large data centers. As others have commented to me in the past, having all of your data and computing in a single location is not necessarily a good idea.

Condor will be open source?

An interesting article on Red Hat's plans for its "Enterprise MRG" distribution. MRG stands for Messaging (an implementation of the Advanced Message Queuing Protocol), Real-time (real-time Linux kernel), and Grid (which for Red Hat, means virtualization support and Condor support for application deployment).

And for those who have always wanted to see the supposedly open source, but never really accessible Condor software, some good news:

As part of the agreement between Red Hat and the University of Wisconsin, the Condor software will seek an OSI-compliant software license that will allow the code to be distributed as part of a RHEL stack.

MapReduce info

Rick Stevens pointed me at this post describing Yahoo!'s work with Hadoop, the open source implementation of MapReduce. Of course, that is old news, but some navigation led me to this nice MapReduce cookbook for machine learning.

Many many years ago I wrote a book on parallel programming. My initial plan was to show how just about everything could be expressed using map reduce, but I couldn't quite see how to package things in a way  way that made sense. (Probably I wanted to do it all via language constructs.) So instead I ended up covering a lot of other material instead. It's neat to see how with the right abstractions things turn out so nicely.

Earth to be paradise; distance to lose enchantment

I recently had the occasion to present a talk on "virtual environments and knowledge production." This got me thinking about just how virtual environments can help with the important tasks of enhancing both individual and collective human creativity. My talk tries to highlight some of what has been accomplished and some of the challenges that lie ahead. (However, as I continue to strive to remove words from slides [recalling Beckett's admonition that "every word is an unnecessary stain on silence and nothingness"], the slides themselves may not make much sense!) I mentioned the following as some of issues that must be addressed for virtual worlds (a subset of virtual environments) to be useful for "knowledge production":

• Integration with the physical world, e.g., sensors and instrumentation
• Integration with the rest of the cyberworld
• Integration with simulation
• Security and trust, in their many forms
• Abstractions, metaphors, interfaces
• Scale (data volumes, simulation fidelity)

During my reading, I came across a nice article by Gary Olson and Judith Olson, Distance Matters, which included a delightful quote from Arthur Mee:

If, as it is said to be not unlikely in the near future, the principle of sight is applied to the telephone as well as that of sound, earth will be in truth a paradise, and  distance will lose its enchantment by being abolished altogether.

(Arthur Mee was best known for his Children's Encylopedia, but here he is writing in 1898 about the impact of the telephone.)

Relative debugging with Guard

When modifying software, or ported it from one system to another, one often ends up checking for correctness (and attempting to diagnose errors) by comparing the execution of a new program with a reference version that is known to work. My Australian friend and colleague David Abramson developed a tool called Guard to automate this process. With Guard, you specify the variables you want to monitor, and when, and then fire up the new and reference versions of your software. Guard monitors the specified variables, and notifies you when their values differ.

John Michalakes and I had the occasion to work with  Guard back in 1996 when we applied it in a project developing a parallel mesoscale weather model, MM5. It was spookily wonderful to see differences between the parallel and sequential implementations become visible in real-time in a 3-D visualization. (The figure shows a 2-D plot of differences, which is also useful but less beautiful.) It also then became extremely easy to fix those problems. We wrote a paper together on this work, which won a best paper award at SC'96, due I think to David's presentation skills--and the videos.)

This technology has now been licensed by Cray for use in their new Cascade program. It's exciting to see the technology making its way into mainstream use.

The end of Grid computing?

Guy Tel-Zur announces the end of grid computing, based on Google trends data for "grid computing" vs. "virtualization."

The data are fun, but I'm not convinced. Certainly the Google Trends data captures the fact that "virtualization" has replaced "grid computing" as the most popular industry buzzword. But given that industry has used "grid computing" mostly to mean "cluster computing" (e.g., Oracle 10-G, SGE), that doesn't say too much about grid per se.

Measuring adoption and impact is nevertheless an important goal. Thus we have integrated usage reporting mechanisms into our Globus software. We see continued growth in use, as captured by metrics such as service deployments. We're now trying to understand the underlying usage modalities. We believe that many are concerned with "eResearch" functions other than "federating computers"--e.g., on-demand access to computing [on HPC systems and/or EC2], data distribution, service publication and composition, etc. Do these functions count as "grid"? They do according to our article "The Anatomy of the Grid"--and if you look at the goals of projects such as D-Grid.

It would be interesting to see Google Trends data for just "grid." However, that word alone has too many different uses.

Digital Rome

I like this story (and see images and video)--June 11, 2007 -- ROME -- Rome's Mayor Walter Veltroni will officiate at the first public viewing of "Rome Reborn 1.0," a 10-year project based at the University of Virginia and begun at the University of California, Los Angeles (UCLA) to use advanced technology to digitally rebuild ancient Rome. The event will take place at 2 p.m. in the Palazzo Senatorio on the Campidoglio.  An international team of archaeologists, architects and computer specialists from Italy, the United States, Britain and Germany employed the same high-tech tools used for simulating contemporary cities such as laser scanners and virtual reality to build the biggest, most complete simulation of an historic city ever created. “Rome Reborn 1.0" shows almost the entire city within the 13-mile-long Aurelian Walls as it appeared in A.D. 320. At that time Rome was the multicultural capital of the western world and had reached the peak of its development with an estimated population of one million.