My Photo

« Won't you blog about this song? | Main | Call for Participation: Open Source Grid & Cluster Conference »

January 08, 2008

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d8341c6ed053ef00e5504e25748833

Listed below are links to weblogs that reference There's Grid in them thar Clouds*:

» There's a Grid in them thar Clouds* from Grid Gurus
You’ve probably seen the recent flurry of news concerning “Cloud computing.” Business Week had a long article on it (with an amusing and pointed critique here). Nick Carr has even written a book about it. So what is it about, [Read More]

Comments

Lisa Childers

I agree. While a scientist running a simulation may rely on the computing power provided by a cloud (or petascale machine, or whatever) for runs of a massive scale, results will often be propagated within the interested scientific community for further analysis. There is life outside the cloud!

Bert Armijo

>>> the methods used to achieve these goals in today’s commercial clouds have not been open and general purpose, but instead been mostly proprietary and specialized for the specific internal uses (e.g., large-scale data analysis) of the companies that developed them

I agree, but this is a common first step in technological development. Those who encounter the need first build solutions of necessity.

In the case of cloud/utility computing, general purpose software and computer hardware vendors didn't foresee or respond to the needs of internet operators who needed to run massive server farms because they were focused on corporate IT. As a direct result Google, Amazon, Yahoo, and others were forced to build their own infrastructure systems which naturally are tailored to their unique needs.

However, while well publicized, these are not the only systems. We at 3tera have built a grid OS specifically for enabling utility computing services. Plus, just to further your point about mixing microproduction and large utilities, rather than build our own data centers we're partnering with commodity hosting providers who already operate numerous data centers and hundreds of thousands of servers.

Michael Behrens

Thanks for the blog entry Ian.
I would like to see less focus on clouds and more focus on grid 2.0 capabilities which might include autonomic application/service virtualiation and transparency. I'm Looking forward to future advances in distributed computing made by acadamia, and industry. And I hope that all of us, together with the Standards Developing Organizations (SDOs), can solidify them into adopted interoperable standards.

Marlon Pierce

I particularly agree with your point that the next generation of computing will need to range seamlessly from "micro-production" computers to mega-clusters. The thought that I would want to rely solely on online cloud computing and data services when I can have a supercomputer under my desk seems counter-intuitive. Most people will always be consumers of services, but there will be an increasing number of "long-tail" (sorry) service providers, including home enthusiasts. There should therefor be an important opportunity to provide "home edition" cloud and Web service software to these folks.

Paul Wallis

Ian,

During 2003, the late Jim Gray made an analysis of Distributed Computing Economics:

“’On Demand’ computing is only economical for very cpu-intensive (100,000 instructions per byte or a cpu-day-per gigabyte of network traffic) applications. Pre-provisioned computing is likely to be more economical for most applications - especially data-intensive ones.”

And

“If telecom prices drop faster than Moore’s law, the analysis fails. If telecom prices drop slower than Moore’s law, the analysis becomes stronger.”

Since then, telecom prices have fallen and bandwidth has increased, but more slowly than processing power, leaving the economics worse than in 2003.

By 2012, the proposed Blue Gene/Q will operate at about 10,000 TFLOPS outstripping Moore’s law by a factor of about 10.

I’ve tried to put The Cloud in historical context and discussed some of its forerunners here

http://www.keystonesandrivets.com/kar/2008/02/cloud-computing.html

My take is that:

“I’m sure that advances will appear over the coming years to bring us closer, but at the moment there are too many issues and costs with network traffic and data movements to allow it to happen for all but select processor intensive applications, such as image rendering and finite modelling.”

I don’t know when enterprises will switch to “The Cloud” but given current technological circumstances, and recent events like The Gulf cables being cut and Amazon S3 failing, today the business is being asked to take a leap of faith to put mission critical applications in The Cloud.

Suraj Pandey

With the advent of new requirements for large scale computing like dynamic provisioning, massive storage space, on the fly computations etc., researchers coined the new word "Cloud Computing" citing the inadequacies of grid to provide these facilities.
"Grid Computing" as it has emerged is considered as loosely coupled and fragile unlike the new vision of Clouds.
The problems are the same, scale is different. With multi-core technologies getting firm grip on computers, centralized Grid or Cloud is already becoming a reality.

Sambath Narayanan

It appears to me that cloud is a new name given to Grid. One of the key difference between Grid computing and Cloud computing lies in the nature of applications. The applications those run on cloud seems to be general purpose ranging from the traditional scientific applications to modern social networking applications. As Ian explains in his book, 'managed, shared, virtual system' as the next step in the evolution of Grid. May be this is nothing but Cloud.

igre

Interesting reading and I agree that methods used to achieve goals in today’s commercial clouds have not been opened to general purpose, but instead been mostly proprietary and specialized for the specific internal uses for companies.

Snehal Antani

Clouds and Grids are complements, not supplements. More over, they solve very different problems and should really be treated as such. Unfortunately though, Cloud/Grid/Utility are terms that have become very overloaded :).

Cloud Computing is about the dynamic provisioning of logical partitions (LPAR's) and leveraging utility-computing technologies like metering and workload management to provide chargeback and goals-oriented execution for the physical resources consumed by those LPAR's. So Amazon EC2 for example can quickly and cheaply provision new OS images (LPAR's) on which applications can run. The LPAR's don't care about the applications, what they do, what data they need to access, etc.

Grid Computing is about the coordinated execution of some complex task across a collection of resources. For example, protein folding is some complex task which could be broken into discrete units of work, and each unit of work could be executed concurrently across a cluster of servers. The grid application infrastructure has the burden of creating the partitions of work and providing operational management (start, stop, dispatching, results aggregration, etc) of those discrete chunks.

Within grid computing there are sub-categories: compute grids and data grids. Compute Grids are responsible for breaking large tasks into discrete chunks, executing some computations on them, and aggregating the results. Data Grids are about partitioning data across a collection of resources for scalability and higher performance. You should see Compute Grids and Data Grids working together to provide a high-performance, scalable processing infrastructure. You can read more about building high performance grids at: http://www-128.ibm.com/developerworks/websphere/techjournal/0804_antani/0804_antani.html. For an example architecture where compute grids and data grids are working together, see this specific section: http://www-128.ibm.com/developerworks/websphere/techjournal/0804_antani/0804_antani.html#xdegc

So to summarize: Clouds are about the dynamic provisioning of LPAR's and leveraging metering and WLM technologies to manage and charge for those logical partitions. Grid computing is an application infrastructure and programming style that allows complex tasks to be broken into smaller pieces and executed across a collection of resources. Grid Computing is about executing business logic quickly, Cloud Computing is about provisioning infrastructure. The grid computing infrastructure would run on top of a cloud computing infrastructure.

This isn't all that new. Big Iron hardware, their operating systems, and the middleware stack have been doing this type of work for decades. The difference today is that Amazon EC2 can quickly and cheaply provision new LPAR's whereas LPARs in big iron machines are statically defined. Both leverage hardware virtualization (executing on shared hardware resources), OS virtualization via a VM/hypervisor, have some application container for executing the business logic per some QoS requirements(application servers, etc), and leverage some type of workload management mechanism for metering and goals-oriented execution.

There is nothing wrong with statically defined LPAR's as long as we have very smart workload management, good hypervisors, and hardware virtualization technologies (See System Z). Dynamic provisioning of resources within the datacenter already exists, see Tivoli Provisioning Manager and other such technologies. The future will probably be a more integrated and cohesive hardware and software stack for "private clouds" and "enterprise grids" (the 'enterprise' in "enterprise grids" implies a high level of QoS like security, resource management, transactions, etc are expected).

I also discuss this in another blog post: http://www-128.ibm.com/developerworks/forums/thread.jspa?threadID=214794&tstart=0

Sam Johnston

Hi Ian,

Having spent the last few days working on finding a consensus definition for Cloud Computing it seems there is a lot (too much IMO) of focus on grid computing, and indeed confusion between the more general concept of cloud computing and the very specific grid computing solution to horizontal scalability. A grid, however intelligent, is one component of cloud computing rather than cloud computing itself... I don't care if my cloud based services are powered by a mainframe, an army of cheap PCs, my neighbour's screensaver or a cluster of commodore 64's, so long as it's fast, cheap and secure - it just so happens that the best way to scale today is by strapping a bunch of PCs together... will this still be true when chips with thousands of cores arrive? I doubt it.

I agree fully with your vision about creation of a computing grid (like the electricity grid) but I think the same opportunistic vendors and overnight experts who hijacked the term 'grid computing' for local installations (the electrical equivalent of in-house diesel gensets) are trying to do the same with cloud computing.

Perhaps you will find my recent article (The Cloud and Cloud Computing consensus definition?) on the topic interesting.

Sam

taoqian

will cloud coputing make grid obsolete?
grid workflow can be important research fields?

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Comments are moderated, and will not appear until the author has approved them.