Ian Foster

Nobel Peace Prize and the Grid

The Intergovernmental Panel on Climate Change (IPCC) and Al Gore have been awarded the Nobel Peace Prize.

A perhaps underappreciated aspect of this award is that (as I have noted previously) the climate model simulation data that underpinned the IPCC analysis was made available to the international community via the Earth System Grid, which itself uses (among other things) Globus software. Thus the Grid and Globus communities could (if they were less modest) claim a tiny little bit of credit for that prize.

The next phase of IPCC will require the analysis of far more data than in the current round, as models become yet more sophisticated and more scenarios are run. The next phase of Earth System Grid will feature a more decentralized, federated structure to enable this analysis.

Globus in Seattle Next Week

Next week is OGF21, where grid gurus from around the world assemble to discuss technologies, applications, standards, and how gray the weather is in Seattle.

We have organized a full day of Globus material on Wednesday October 17. We'll have overviews of old favorites such as GridFTP, RLS, OGSA-DAI and the GT4 distribution, as well as introductions to some of our many new Incubator projects: Shannon Hastings, OSU, discussing the service authoring tool Introduce, Steve Tuecke of UnivaUD discussing Data Catalyst, their open source higher level data solution, and Stephan Erberich who will overview the Internet2 IDEA Award-winning MEDICUS medical data tool, among others. Come hear about the latest updates and where Globus is going to next, and/or to talk to Globus architects and developers about things like:

• Your applications and how you can apply Globus technologies
• Problems or questions with Globus technologies
• Your wish list for future Globus features
• How to contribute your software to the dev.globus community

If you'd like to meet with someone from the Globus team in Seattle, please email us: we'll see you there!

New release of Globus Eclipse code

Kay Dörnemann writes about a new release of the Grid Development Tools for Eclipse (GDTE), a bundle of Eclipse Plugins for Service and Application Development in the Eclipse Integrated Development Environment for the Globus Toolkit 4:

I have the pleasure to announce the release of GDTE 1.2.0. The release can be easily installed using the Eclipse update mechanism: see our Update site.

Installation instructions, tutorials, and documentation can be found at the GDTE dev.globus incubator site.

Tutorials are available online and if you install the GDT feature, there's also Help available in your Eclipse workbench, describing more details about the use of the GDT. (We also support using Eclipse headless, i.e. running the GDT tools as commandline tools, offering the possibility to include them in automated build scripts.)

Whats new: We added service-side security descriptor creation through a wizard. Added Singleton Resource support and load on Startup support. Reworked the wizard pages and fixed several bugs. For more information please read the changelogs.

GT 4.0.5 Released

I enclose below the text that Charles Bacon sent today to announce@globus.org.

There are a LOT of improvements in GT4.0.5. A few highlights: GRAM staging performance is doubled, thanks to optimized local invocations; GridWay metascheduler is included; lots of new trigger service functionality.

Weka4WS 1.0 Release -- Data Mining

Domenico Talia tells me about his groups release of Weka4WS version 1.0. Weka4WS is a Web Services-based framework for  distributed data mining. It's based on the popular Weka Explorer data mining system from the University of Waikato in New Zealand, and is based on Globus Toolkit 4.

I'd love to hear from people who are using this software.

IBERGRID Video Online

The organizers of IBERGRID kindly posted this video of my talk on "Scaling eScience Impact" (the slides are here). The abstract follows:

Computational approaches to problem solving have proven their worth in many fields of science, allowing the collection and analysis of unprecedented quantities of data, and the exploration via simulation of previously obscure phenomena. We now face the challenge of scaling the impact of these approaches from the specialist to entire communities.  I speak here about work that seeks to address this goal by rethinking science's information technology foundations in terms of service-oriented architecture.  In principle, service-oriented approaches can have a transformative effect on scientific communities, allowing tools formerly accessible only to the specialist to be made available to all, and permitting previously manual data-processing and analysis tasks to be automated. However, while the potential of such "service-oriented science" has been demonstrated, its routine application across many disciplines raises challenging technical problems. One important requirement is to achieve a separation of concerns between discipline-specific content and domain-independent infrastructure; another is to streamline the formation and evolution of the "virtual organizations" that create and access content. I describe the architectural principles, software, and deployments that I am and my colleagues have produced as we tackle the first of these problems, and point to future technical challenges and scientific opportunities.

In addition to covering issues discussed in my 2005 Science paper, I touched upon recent work by the OSU team and Ravi Madduri on Introduce and RAVE, and by the U.Chicago team on Swift and Falkon.

Release of Globus Toolkit 4.1.2

The wonderful GT development team just announced the latest development release. The 4.1.2 release contains early looks at new features destined for GT 4.2, and as such is not recommended for production environments--but is highly recommended for people who want to try new features, of which there are many. Highlights include:

• Persistent HTTP/S connection support in Java WS Core
• Dynamic deployment support in Java WS Core
• JBOSS 4.0.x support in Java WS Core
• An implementation of WS-ServiceGroup added to C WS Core
• C command-line tools for WSRF operations
• Support for GetResourceProperties and QueryResourceProperties in the Delegation Service
• Added support for the OGSA-AuthZ Authorization Service to CAS
• Server-side attribute-based authorization framework enhancements
• Support for a pluggable Policy Decision Point (PDP) designed to minimize common authorization errors
• Enhanced security descriptor framework
• A Web service interface for the Replica Location Service (RLS)
• Improved data service resource support in OGSA-DAI
• Improved support for multiple TriggerRules in the Trigger Service
• Improved configuration interface for the Trigger Service
• Java API to assist in creating resource properties from external information sources
• A new resource manager (RM) adapter API in GRAM4

Globus Training at Karlsruhe

Ruediger Berlich from the Forschungszentrum Karlsruhe comments on my article on European middleware:

[it is] worth noting that the uptake of Globus by European summer schools is quite big. These are at least partially supported by the EU.
In particular, GridKa School [September 12, Karlsruhe] will again run a comprehensive course on Globus this year, led by Jennifer Schopf and Ben Clifford.
Last year, almost 120 participants attended the school, and the Globus course was very well regarded (indeed it had the second highest rating of all courses in the school!).

The GridKA school is one of several upcoming training events in Europe and the US to feature Globus software. By all accounts, the Karlsruhe event is particularly well run 

Grid middleware in Europe

Trips to Germany and Spain in the past month have helped me catch up the latest state of grid middleware in Europe. It's an exciting but also curious scene:

• Tens of millions of Euros are being spent on the Enabling Grid for eScience (EGEE) grid infrastructure and its gLite middleware, and on applying that middleware in various domains and  promoting its adoption internationally. EGEE and gLite are heavily focused on the computing requirements of the Large Hadron Collider.
• In the Nordic countries, we have the NorduGrid infrastructure and its Advanced Resource Connector (ARC) middleware. NorduGrid and ARC are also focused on physics.
• Unicore has been developed with German and European funding over the past eight years. Unicore is focused on enabling remote access to supercomputer centers, and sees heavy use in Germany, where much of its development occurs.
• Finally, the UK has put much money into OMII-UK, which in addition to supporting the popular myGrid and OGSA-DAI products, has created its own distinct middleware platform.

So there are at least four distinct European middleware solutions [actually five: see below]. Is this a good  thing? Officially, yes. Each is a big success, diversity is positive, and  interoperability is assured.

Unofficially, users talk with frustration about being pressured to use their funding agency's middleware, software developers bemoan the need to target different middlewares, and sites complain about having to support multiple software stacks. Meanwhile, interoperability is stymied by differing versions of standards and different configurations and policies.

The only software that no-one in Europe is pressured to use or deploy is Globus, and I am personally satisfied to see how many projects use it nonetheless. Indeed, while I cannot back up the following assertion with data (for one thing, privacy-conscious Europeans tend to turn off Globus usage reporting), I'd bet good money that Globus is the most widely used grid middleware in Europe. (That's not taking into account the Globus components included in ARC and gLite.) The people that talk to me must be somewhat self selecting, but they speak with tremendous enthusiasm about what it lets them do.

It's hard to see where this will all lead. While "made in Europe" (or "made in the Nordic countries" or "made in the UK") is a powerful rallying cry, surely four different systems can't be sustained indefinitely. Nevertheless, I don't see anything changing soon. Perhaps we can just hope for incremental steps: e.g., integrating the latest Globus components into gLite (they're using code that is several years old) and achieving interoperability between Globus and Unicore (we're on the hook for that).

It also seems strange to have no EU support for European Globus users--surely that would make good scientific sense? The reason seems to be that Globus is viewed by the EU as "US software"--even though it is all open source, and its developer and user community includes many Europeans.

ADDED LATER ON May 24: I discover that there are in fact not four but five grid middleware projects in Europe--ExtreemOS is the latest. They say: "The XtreemOS system will offer an alternative to the Globus toolkit, which is currently the most widespread middleware system." A noble goal!

Ignacio Llorente's Talk

I wrote about Ignacio Llorente's nice talk on Grid Scheduling Architectures. The talk is now available online. As well as introducing the Gridway metascheduler, he presents a taxonomy of metaschedulers, describing and giving concrete examples of a range of different architectures, such as:

• Single metascheduler grids: one metascheduler instance with access to resources within one or several administrative domains. Examples: European Space Astronomy Center (several clusters); AstroGrid-D (22 resources, 5 sites, 800 CPUs); UABGrid (3 resources: PBS, SGE, Condor).
• Multiple metascheduler grids: each metascheduler serving the needs of a distinct community. Example: Fusion and Biomedical users on EGEE.
• Multiple grid infrastructures: metascheduler maps user requests to different infrastructures. Example: dispatch to OSG, EGEE, TeraGrid.
• Multiple metascheduler layers: Hierarchical organization of multiple metaschedulers, with requests handed off to lower layers to meet peak demands. Example: offload from GRIDIMadrid to EGEE.
• From the cluster to the Grid: Transfer queues used to offload tasks from a cluster to remote grid resources.