Ian Foster

SAML-Based Attributes for Globus Grids

The GridShib Project announced last week the release of GridShib for Globus Toolkit v0.6.0.  This is an exciting development, as GridShib software allows for powerful new authorization architectures in which access control decisions are made based on attributes obtained from many different sources. From their announcement:

This  release culminates a 20-month effort to bring SAML-based attribute push to X.509-based Grids.

GridShib for Globus Toolkit (GT) is an implementation of a Grid Service Provider, an entity much like a SAML Service Provider but for Grids.  A Grid Service Provider consumes X.509-bound SAML tokens, a new type of security token that enables attributed-based authorization in X.509-based Grids.

Most everything you need to know about GridShib for GT is on this web page:

http://gridshib.globus.org/docs/gridshib-gt-0.6.0/readme.html

On this readme page, you will find more detailed information about the GridShib for GT software as well as links to downloads and documentation.

A major advance in this version of GridShib for GT is support for the TeraGrid Science Gateway use case where an intermediary makes a grid request on behalf of a browser user.  The Gateway binds a SAML token to an X.509 proxy certificate and makes a request to a
gridshib-enabled web service.  On the service side, GridShib for GT consumes the SAML token and makes an access control decision based on the security information in the token.

As a SAML-consuming software component, GridShib for GT complements the previously released GridShib SAML Tools and GridShib Certification Authority (CA), which are SAML-producing software components.  These three components together enable attribute-based authorization in X.509-based Grids.  See the Quick Start for step-by-step instructions that show how to use GridShib for GT v0.6, GridShib SAML Tools v0.3, and GridShib CA v0.5.1 together on Windows and UNIX systems:

http://gridshib.globus.org/docs/gridshib/quick-start.html

For links to all GridShib software downloads and additional documentation, visit the GridShib Downloads page:

http://gridshib.globus.org/download.html

Funding for GridShib software has been provided by the NSF NMI program and the NSF TeraGrid program.

10 Reasons to Attend Open Source Grid and Cluster Conference

Ok, I admit it is corny--but I assembled a list of 10 reasons why you should attend the Open Source Grid and Cluster Conference, to be held in Oakland May 12-16 (www.opensourcegridcluster.org).

1) Globus program is fantastic, including tutorials, advanced technical presentations, contributed talks, and community events on every aspect of Globus.

2) Gobs of other material on Sun Grid Engine and Rocks, and other open source grid and cluster software.

3) Gathering: A great opportunity to meet colleagues, peers, collaborators from the grid and cluster community. The only grid meeting in the US the rest of this year--the next two OGFs are in Spain (June) and Singapore (September).

4) GT4.2: You'll get to learn about the exciting new features in Globus Toolkit 4.2. New execution, data, security, information, virtualization, and core services.

5) Gratfication (immediate) as you get to provide your input on future directions for Globus, Sun Grid Engine, Rocks, and other open source systems--and maybe sign up to contribute to those developments.

6) Grid solutions: You'll get to meet the people using Globus to build enterprise grid solutions in projects like caBIG, TeraGrid, Earth System Grid, MEDICUS, and LIGO, and learn about solution tools like Introduce, MPI-G, Swift, Taverna, and UniCluster.

7) Gurus: You get to grill the Globus gurus--or, if you prefer, show off your own Globus guru status.

8) Great price: $490 registration is substantially cheaper than OGF or HPDC, for example, and the hotel rate is reasonable ($149).

9) Gorgeous location: Oakland is easy to get to -- SFO (with easy BART  train ride), Oakland, and San Jose airports also nearby. Just a 10 minute train ride to download San Francisco. A lovely time to be in the Bay Area.

10) Gorilla and guerilla free: None of the corporate marketing talks that diluted the last GridWorld conference--apart from two sponsor talks, this is pure tech, and highly useful tech at that.

We look forward to seeing you in Oakland!

Regards -- Ian.

Clouds over Chicago

Way before clouds were popular (remember then?) my colleagues Kate Keahey and Tim Freeman started work on their workspace service, a system for on-demand creation and management of virtual machines on remote computing systems. They now have an implementation that interfaces both to clusters running conventional schedulers and to Amazon EC2. It's distributed as part of the Globus software, or you can download it separately.

Kate and Tim have recently established a deployment of the workspace service in the Computation Institute at U.Chicago and Argonne. With a nod to the new cloud meme, they've named it Nimbus. They say:

The University of Chicago Science Cloud, codenamed "Nimbus", is a web service that delivers compute capacity in the cloud for scientific communities. The Nimbus' simple client allows you to obtain customized compute nodes (that we call "workspaces") that you have full control over quickly, easily, and in ways that can be fully automated. Using the Nimbus cloud you can request the exact compute capability you currently need for your application and scale it up or down as your needs dictate.

Nimbus provides compute capability in the form of Xen virtual machines (VMs) that are deployed on physical nodes of the University of Chicago TeraPort cluster using the workspace service. We currently make 16 nodes of the TeraPort cluster available for cloud computing. Nimbus is available for members of scientific community wanting to run in the cloud. To obtain access you will need to provide a justification (a few sentences explaining your science project) and a valid grid credential (If you don't have a credential, email us. We can help). Based on the project, you will be given an allocation on the cloud. Send your requests, demands and cries of anguish to workspace-user@globus.org (for cries of anguish mp3 format is acceptable).

In a typical session you will make a request to deploy a workspace based on a specified VM image. You can either use one of the VM images already available on the cloud (we provide a command that allows you to see what's already there) or upload your own VM image. On deployment, the image will be configured with an ssh public key you provide -- in this way once the workspace is deployed, you will be able to ssh into it and configure it further, upload data, or run your applications. Have fun!

Services for Science

I gave a talk on Services for Science (PDF, PPT) at the INGRID 2008 conference in Ischia, Italy, on Wednesday. I decided to do something different and include demonstrations. I think this worked well. I created and deployed a GT4 service using Introduce and gRAVI, and then created and ran a workflow invoking GT4 services using Taverna.  Well, to be honest, there was some steps I skipped along the way (in the style of Julia Childs), but nevertheless I found it impressive how much could be done interactively, in a short time. In summary, I was able to show how we can:

1. Create a new service using the Introduce integrated development environment, defining operations and resource properties, folding in required functionality (e.g., security, notification), and selecting types from both base types and predefined libraries. (Using gRAVI we can also encapsulate executables.)
2. Publish this service into registries (GT4 index services).
3. Discover available services.
4. Compose services into workflows, e.g., via the use of Taverna, which thanks to recent work by Wei Tan and Ravi Madduri (and much help from the Taverna team) can now invoke GT4 services.
5. Deploy and publish the workflow in turn ...

Grid Summer School

The sixth in the highly successful series of International Summer  Schools on Grid Computing will be held at the Hotel Füred Conference and Congress Centre of Balatonfüred, Hungary, from 6th to 18th July 2008.

The School will include lectures, discussions, laboratory sessions, tutorials and group work delivered by leading authorities in the fields of advanced grid technology, applications of e-Science and distributed systems research. Reports from world leaders in deploying and exploiting Grids will complement lectures from research leaders shaping future e-Infrastructure.

Hands-on laboratory exercises will give students experience with widely used Grid middleware. The school will conclude with an integrating practical that will enable students, working in teams, to bring together all they have learnt on an extended exercise that simulates collaborative research using e-Infrastructures. Indeed during the school, participants will meet like-minded students from many parts of the world, working in many disciplines, and form valuable long-term working relationships.

We invite applications from enthusiastic and ambitious researchers who have recently started or are about to start working on Grid projects. Students may come from any country. We expect participants from computer science, computational science and any application discipline. The School will assume that students have diverse backgrounds and build on that diversity. However, in order to fully participate in the practical exercises you should be a confident programmer who will have fulfilled certain prerequisites.

To find further details visit the web site at: http://www.issgc.org

Nice article on caBIG cancer biomedical informatics grid

I keep mentioning caBIG. But here is a nice profile in ComputerWorld that describes the project's goals and status.

New Globus release -- big GRAM improvements

Charles Bacon, Globus release manager, writes: "On behalf of the Globus Toolkit development team I am pleased to announce that a new incremental release of GT4 is now available for download. GT4.0.7 is recommended for all users.It was released because of bug 5910, a potential RFT data corruption bug.  The bug affected only GT4.0.6, and users of GT4.0.6 can apply the update package from http://www.globus.org/toolkit/advisories.html.  New users are encouraged to start with the 4.0.7 release, as other bugs were also fixed as listed in the release notes.

Relevant 4.0.7 links:
- Release notes: http://www.globus.org/toolkit/releasenotes/4.0.7/
- Software: http://www.globus.org/toolkit/downloads/4.0.7/
- Documentation: http://www.globus.org/toolkit/docs/4.0/

Thanks for your support of Globus software!"

Among other things, GT4.0.7 includes further significant improvements to GRAM performance and scalability.

Big enhancements to caGrid federation infrastructure

I've mentioned the cancer Biomedical Informatics Grid (caBIG) project before. It's goal is "to develop applications and the underlying systems architecture that connects together data, tools, scientists and organizations in an open federated environment." Underpinning caBIG is a service oriented infrastructure, caGrid, which is in turn built on Globus software.

The caBIG project announced today the release of version 1.2 of caGrid core infrastructure, with  enhancements such as

• A simplified Grid Transfer Service to handle large scale data transfer for grid-based queries
• Integration with the latest releases of Software Development Kit (SDK) and Common Security Module (CSM)
• Availability of a new Web based Single Sign On Framework, called WebSSO, for providing single sign on capability for web based applications integrated with caGrid GAARDs security infrastructure, based on collaboration with the caBIG Clinical Trials Suite (CCTS) team from the Clinical Trials Management Systems Workspace
• Substantial improvements to usability of tools such as Introduce Toolkit based on user feedbackI
• ntegration with Apache Ivy build system for improved dependency management of sub components within the caGrid core infrastructure
• Early preview to integration with the Taverna Workbench for developing workflows, based on collaboration with the Integrative Cancer Research Workspace
• Backward compatible with the caGrid 1.0 and caGrid 1.1

My colleagues Ravi Madduri and Wei Tan at the University of Chicago and Argonne National Laboratory have been very involved in this work, in particular the integration with Taverna.

Globus @ Google Summer of Code

Globus has been selected as a mentor organization for the 2008 Google Summer of Code. This means that if you are a talented student programmer, you can sign up to work on a project. You get a modest amount of fortune (US$4500) and some degree of fame (TBD), and Globus gets a  nice open source code contribution. We have a set of project ideas online, but if you think of something else that should be done, let us know!

Call for Participation: Open Source Grid & Cluster Conference

OPEN SOURCE GRID & CLUSTER CONFERENCE 2008

Featuring: GlobusWorld, Grid Engine Workshop, Rocks Cluster Workshop

Join Users, Administrators, and Developers of Open Source Grid and
Cluster Software from across the Globe at this unique event

Open Source Grid & Cluster Conference 2008
May 12-16, 2008
Oakland Marriott
Oakland, California

DEADLINE FOR SUBMISSIONS:  March 21, 2008

Whether you are a Grid or Cluster expert with technical advice to
share, or a leader with visions for the future of open source Grid and
Cluster computing in research or industry, the Open Source Grid &
Cluster Conference is the premier event for delivering your message to
the Grid and Cluster community.  In past years, hundreds of Grid and
Cluster professionals from research and industry have attended
individual events such as GlobusWorld, the Grid Engine Workshop, and
Rocks-a-Palooza to discuss Grid and Cluster adoption issues, to
receive training and exchange information related to these widely used
Grid and Cluster software systems. This year the Globus, Grid Engine,
and Rocks communities are joining forces to create the most
comprehensive event on open source Grid and Cluster computing to date.

The Open Source Grid & Cluster Conference program will offer a wide
variety of conference sessions, mini-symposiums, panel discussions,
workshops, and tutorials. Speaking opportunities range from highly
technical research, development, and deployment presentations to
targeted panels on commercial and research adoption
considerations. The Open Source Grid & Cluster Conference will run
parallel tracks, some focused on Globus, Grid Engine, and Rocks
community-specific topics, and others focused on cross-cutting and
other open source Grid and Cluster software technologies and uses.

KEY DATES AND DEADLINES
Abstract submission deadline - March 21, 2008
Acceptance notification - April 15, 2008
Presentation Slides Due - April 30, 2008

SPEAKING TOPICS
Submissions should be centered on the theme of uses and implementation
of Open Source Software for Grid and Cluster Computing.

All proposals should be submitted online at
   http://www.OpenSourceGridCluster.org/CFP.html

Questions should be sent to program@OpenSourceGridCluster.org

SUBMISSION GUIDELINES
---------------------

ABSTRACT GUIDELINES
All submissions must include an abstract of no more than 500 words,
and a brief bio for each presenter. Abstracts should be written so as
to be self-contained and to provide the technical substance required
for the program committee to evaluate the session's contribution to
the Open Source Grid and Cluster community. Please indicate whether
the proposed session is specific to just one of Globus, Grid Engine,
or Rocks. If the presentation was given at another conference, then
the name, date, and location of the event must be noted in the
submission. Abstracts should be submitted in plain text format either
as an attachment or in the main body of the e-mail. Abstracts and bios
for accepted submissions will be published on the Open Source Grid &
Cluster Conference website and in other conference material as the
description of the session. Presentation slides may be published on
the Conference website and distributed with conference material.

PRESENTATIONS
Presentation proposals may be submitted for individual time slots of
thirty minutes. Please be sure to allow ten minutes for Q&A within
this allotted time. Individual presentations will be grouped with
similar topic presentations to fill an entire session.

BUILD YOUR OWN SESSION
Participants are invited to organize their own, complete,
ninety-minute session, including but not limited to the following
categories. The submission must include an agenda, and the names and
associations of all participants.

Panel Session / Mini-Symposium: These sessions will enable conference
attendees to learn from a group of experts on a particular topic. The
session organizer may deliver an opening talk to set the context for
the remainder of the session. Panelists will then give presentations
designed to stimulate audience participation, on their preferably
diverse opinions, experiences or expertise regarding the theme of the
session. At least ten minutes should be reserved at the end for
questions from the audience.

Birds-of-a-Feather (BOF) Sessions: These sessions will allow
conference attendees to discuss focused subject areas. The session may
include presentations and open discussion. Session organizers will be
responsible for moderating these sessions and reporting on their
outcomes.

WORKSHOPS AND TUTORIALS
Ample room is available for half-day and full-day pre-conference
(Monday) and post-conference (Friday) workshops and
tutorials. Workshops may include topical meetings with open
registration or community/group meetings with resricted attendance.
Tutorials may be on any topic related to the Open Source Grid and
Cluster theme of the conference. Submissions must include preferred
and minimum acceptable room size, and preferred and acceptable
times. An extra nominal fee may be required of attendees or the
organizer to cover additional costs such as A/V and food.

All proposals should be submitted online at
   http://www.OpenSourceGridCluster.org/CFP.html

Questions should be sent to program@OpenSourceGridCluster.org

Grid.org

Univa UD has re-launched "grid.org"--previously dedicated to philanthropic computing projects--as:

a single location where open-source cluster and grid information can be aggregated and where community members can exchange information, experiences, and ideas related to the complete open source cluster software stack. In particular, but not exclusively, Grid.org provides a community where users of Cluster Express, and the various open source components it comprises, can interact with each other and with the source code.

I recall finding it frustrating that the URL "grid.org" (which I would have loved to have back when our Grid book first came out) was devoted to "desktop computing." But things sometimes come to those who wait!

For those interested in applying their home computers to philanthropic computing projects, I recommend the World Community Grid, which harnesses idle computers for useful purposes such as modeling potential AIDS and cancer drugs.

Globus in Reno next week

As usual, many in the grid and HPC community are migrating to the "SC" conference, this year in delightful Reno. There will be a huge number of talks and demonstrations on Globus applications, infrastructures, and technologies. Here is a partial list of talks and demonstrations. Come to the Argonne booth to see our new advance reservation service, learn about the latest in Globus technologies, hear what Earth System Grid has been up, see parallel programs running via Swift!

It is not too late to sign up for the Globus and GridFTP tutorials, to be held on Sunday and Monday respectively.

My colleagues from Univa UD will be there also, showing off their new Cluster Express product. Stop by their booth to say hello to Steve Tuecke and others.

Please let us know if you want to meet to discuss your use of Globus, your ideas for future Globus development, and/or technologies that you think complement Globus software.

I myself will not be attending SC this year due to other commitments--the first time in many years that I am missing the conference. And no, it is not because it is being held in Reno.

caGrid article

An article on iSGTW describes the National Institutes of Health's cancer biomedical information grids (caBIG) project and its caGrid infrastructure. Not too much new information relative to previous posts, but a good reminder of the nice work that this group is doing. The event that spurred this article is the recent release of caGrid 1.1.

Interestingly, recent articles note that cancer rates continue to decline in the US, but increase worldwide--to and estimated 15 million new cases by 2020. That's a lot of people, emphasizing the importance of this work.

From a technology perspective, caBIG and caGrid are exciting because of the extensive and powerful use they make of the Web Services infrastructure developed over the past several years. In particular, I can't resist pointing out that the entire infrastructure is based on Globus software, and in particular its implementation of the WSRF, WS-Notification, WS-Addressing, security, and related specifications.

RAVI project launched; Charles Bacon new IMP chair

The dev.globus open source grid software community continues to expand with the formation of another Incubator, RAVI: Remote Application Virtualization Infrastructure.RAVI leverages the Introduce system to provide GUI-based tools to guide a user through the process of identifying an application, mapping from strongly typed Web Services operations to application arguments, defining authentication and authorization requirements, and deploying a service onto an execution site. (RAVI, by the way, was the name I suggested humorously for this project originally proposed by Ravi Madduri. He tried to change it to RAVE, but that was already taken.)

This brings the total of active dev.globus projects up to twenty-five. Information on all twenty-five can be found at http://dev.globus.org/wiki/Welcome. (If you're interested in starting a project, please contact incubator-committers [at] globus.org.)

Meanwhile, Jennifer Schopf has completed her term as chair of the dev.globus Incubator Management Project (IMP). During her tenure, the dev.globus community has expanded considerably: we owe her a big thanks for her outstanding and tireless work.

Charles Bacon has kindly agreed to step into the IMP Chair role. We wish him the very best for his tenure.

GridWay wins EGEE prize

A belated note of congratulations to our GridWay colleagues at the Universidad Complutense de Madrid. (A dev.globus incubator.) Their demo entitled "GridWay Interfaces for on-Demand Access to EGEE" won the "Best Demo Prize" at the EGEE'07 Conference. The demo showed the several interfaces provided by GridWay to port applications:

• OGF DRMAA standard support (C, JAVA, Ruby, Perl and Python bindings)
• LRM-like CLI
• From a cluster to the Grid (SGE Transfer Queues)
• GridGateWay (WS-GRAM interface to a whole grid infrastructure), which allows end user to access a meta-scheduling instance using Globus commands

I also note that they just announced a new stable release (5.2.3) of the GridWay metascheduler.

Nobel Peace Prize and the Grid

The Intergovernmental Panel on Climate Change (IPCC) and Al Gore have been awarded the Nobel Peace Prize.

A perhaps underappreciated aspect of this award is that (as I have noted previously) the climate model simulation data that underpinned the IPCC analysis was made available to the international community via the Earth System Grid, which itself uses (among other things) Globus software. Thus the Grid and Globus communities could (if they were less modest) claim a tiny little bit of credit for that prize.

The next phase of IPCC will require the analysis of far more data than in the current round, as models become yet more sophisticated and more scenarios are run. The next phase of Earth System Grid will feature a more decentralized, federated structure to enable this analysis.

Globus in Seattle Next Week

Next week is OGF21, where grid gurus from around the world assemble to discuss technologies, applications, standards, and how gray the weather is in Seattle.

We have organized a full day of Globus material on Wednesday October 17. We'll have overviews of old favorites such as GridFTP, RLS, OGSA-DAI and the GT4 distribution, as well as introductions to some of our many new Incubator projects: Shannon Hastings, OSU, discussing the service authoring tool Introduce, Steve Tuecke of UnivaUD discussing Data Catalyst, their open source higher level data solution, and Stephan Erberich who will overview the Internet2 IDEA Award-winning MEDICUS medical data tool, among others. Come hear about the latest updates and where Globus is going to next, and/or to talk to Globus architects and developers about things like:

• Your applications and how you can apply Globus technologies
• Problems or questions with Globus technologies
• Your wish list for future Globus features
• How to contribute your software to the dev.globus community

If you'd like to meet with someone from the Globus team in Seattle, please email us: we'll see you there!

New release of Globus Eclipse code

Kay Dörnemann writes about a new release of the Grid Development Tools for Eclipse (GDTE), a bundle of Eclipse Plugins for Service and Application Development in the Eclipse Integrated Development Environment for the Globus Toolkit 4:

I have the pleasure to announce the release of GDTE 1.2.0. The release can be easily installed using the Eclipse update mechanism: see our Update site.

Installation instructions, tutorials, and documentation can be found at the GDTE dev.globus incubator site.

Tutorials are available online and if you install the GDT feature, there's also Help available in your Eclipse workbench, describing more details about the use of the GDT. (We also support using Eclipse headless, i.e. running the GDT tools as commandline tools, offering the possibility to include them in automated build scripts.)

Whats new: We added service-side security descriptor creation through a wizard. Added Singleton Resource support and load on Startup support. Reworked the wizard pages and fixed several bugs. For more information please read the changelogs.

GT 4.0.5 Released

I enclose below the text that Charles Bacon sent today to announce@globus.org.

There are a LOT of improvements in GT4.0.5. A few highlights: GRAM staging performance is doubled, thanks to optimized local invocations; GridWay metascheduler is included; lots of new trigger service functionality.

Weka4WS 1.0 Release -- Data Mining

Domenico Talia tells me about his groups release of Weka4WS version 1.0. Weka4WS is a Web Services-based framework for  distributed data mining. It's based on the popular Weka Explorer data mining system from the University of Waikato in New Zealand, and is based on Globus Toolkit 4.

I'd love to hear from people who are using this software.

IBERGRID Video Online

The organizers of IBERGRID kindly posted this video of my talk on "Scaling eScience Impact" (the slides are here). The abstract follows:

Computational approaches to problem solving have proven their worth in many fields of science, allowing the collection and analysis of unprecedented quantities of data, and the exploration via simulation of previously obscure phenomena. We now face the challenge of scaling the impact of these approaches from the specialist to entire communities.  I speak here about work that seeks to address this goal by rethinking science's information technology foundations in terms of service-oriented architecture.  In principle, service-oriented approaches can have a transformative effect on scientific communities, allowing tools formerly accessible only to the specialist to be made available to all, and permitting previously manual data-processing and analysis tasks to be automated. However, while the potential of such "service-oriented science" has been demonstrated, its routine application across many disciplines raises challenging technical problems. One important requirement is to achieve a separation of concerns between discipline-specific content and domain-independent infrastructure; another is to streamline the formation and evolution of the "virtual organizations" that create and access content. I describe the architectural principles, software, and deployments that I am and my colleagues have produced as we tackle the first of these problems, and point to future technical challenges and scientific opportunities.

In addition to covering issues discussed in my 2005 Science paper, I touched upon recent work by the OSU team and Ravi Madduri on Introduce and RAVE, and by the U.Chicago team on Swift and Falkon.

Release of Globus Toolkit 4.1.2

The wonderful GT development team just announced the latest development release. The 4.1.2 release contains early looks at new features destined for GT 4.2, and as such is not recommended for production environments--but is highly recommended for people who want to try new features, of which there are many. Highlights include:

• Persistent HTTP/S connection support in Java WS Core
• Dynamic deployment support in Java WS Core
• JBOSS 4.0.x support in Java WS Core
• An implementation of WS-ServiceGroup added to C WS Core
• C command-line tools for WSRF operations
• Support for GetResourceProperties and QueryResourceProperties in the Delegation Service
• Added support for the OGSA-AuthZ Authorization Service to CAS
• Server-side attribute-based authorization framework enhancements
• Support for a pluggable Policy Decision Point (PDP) designed to minimize common authorization errors
• Enhanced security descriptor framework
• A Web service interface for the Replica Location Service (RLS)
• Improved data service resource support in OGSA-DAI
• Improved support for multiple TriggerRules in the Trigger Service
• Improved configuration interface for the Trigger Service
• Java API to assist in creating resource properties from external information sources
• A new resource manager (RM) adapter API in GRAM4

Globus Training at Karlsruhe

Ruediger Berlich from the Forschungszentrum Karlsruhe comments on my article on European middleware:

[it is] worth noting that the uptake of Globus by European summer schools is quite big. These are at least partially supported by the EU.
In particular, GridKa School [September 12, Karlsruhe] will again run a comprehensive course on Globus this year, led by Jennifer Schopf and Ben Clifford.
Last year, almost 120 participants attended the school, and the Globus course was very well regarded (indeed it had the second highest rating of all courses in the school!).

The GridKA school is one of several upcoming training events in Europe and the US to feature Globus software. By all accounts, the Karlsruhe event is particularly well run 

Grid middleware in Europe

Trips to Germany and Spain in the past month have helped me catch up the latest state of grid middleware in Europe. It's an exciting but also curious scene:

• Tens of millions of Euros are being spent on the Enabling Grid for eScience (EGEE) grid infrastructure and its gLite middleware, and on applying that middleware in various domains and  promoting its adoption internationally. EGEE and gLite are heavily focused on the computing requirements of the Large Hadron Collider.
• In the Nordic countries, we have the NorduGrid infrastructure and its Advanced Resource Connector (ARC) middleware. NorduGrid and ARC are also focused on physics.
• Unicore has been developed with German and European funding over the past eight years. Unicore is focused on enabling remote access to supercomputer centers, and sees heavy use in Germany, where much of its development occurs.
• Finally, the UK has put much money into OMII-UK, which in addition to supporting the popular myGrid and OGSA-DAI products, has created its own distinct middleware platform.

So there are at least four distinct European middleware solutions [actually five: see below]. Is this a good  thing? Officially, yes. Each is a big success, diversity is positive, and  interoperability is assured.

Unofficially, users talk with frustration about being pressured to use their funding agency's middleware, software developers bemoan the need to target different middlewares, and sites complain about having to support multiple software stacks. Meanwhile, interoperability is stymied by differing versions of standards and different configurations and policies.

The only software that no-one in Europe is pressured to use or deploy is Globus, and I am personally satisfied to see how many projects use it nonetheless. Indeed, while I cannot back up the following assertion with data (for one thing, privacy-conscious Europeans tend to turn off Globus usage reporting), I'd bet good money that Globus is the most widely used grid middleware in Europe. (That's not taking into account the Globus components included in ARC and gLite.) The people that talk to me must be somewhat self selecting, but they speak with tremendous enthusiasm about what it lets them do.

It's hard to see where this will all lead. While "made in Europe" (or "made in the Nordic countries" or "made in the UK") is a powerful rallying cry, surely four different systems can't be sustained indefinitely. Nevertheless, I don't see anything changing soon. Perhaps we can just hope for incremental steps: e.g., integrating the latest Globus components into gLite (they're using code that is several years old) and achieving interoperability between Globus and Unicore (we're on the hook for that).

It also seems strange to have no EU support for European Globus users--surely that would make good scientific sense? The reason seems to be that Globus is viewed by the EU as "US software"--even though it is all open source, and its developer and user community includes many Europeans.

ADDED LATER ON May 24: I discover that there are in fact not four but five grid middleware projects in Europe--ExtreemOS is the latest. They say: "The XtreemOS system will offer an alternative to the Globus toolkit, which is currently the most widespread middleware system." A noble goal!

Ignacio Llorente's Talk

I wrote about Ignacio Llorente's nice talk on Grid Scheduling Architectures. The talk is now available online. As well as introducing the Gridway metascheduler, he presents a taxonomy of metaschedulers, describing and giving concrete examples of a range of different architectures, such as:

• Single metascheduler grids: one metascheduler instance with access to resources within one or several administrative domains. Examples: European Space Astronomy Center (several clusters); AstroGrid-D (22 resources, 5 sites, 800 CPUs); UABGrid (3 resources: PBS, SGE, Condor).
• Multiple metascheduler grids: each metascheduler serving the needs of a distinct community. Example: Fusion and Biomedical users on EGEE.
• Multiple grid infrastructures: metascheduler maps user requests to different infrastructures. Example: dispatch to OSG, EGEE, TeraGrid.
• Multiple metascheduler layers: Hierarchical organization of multiple metaschedulers, with requests handed off to lower layers to meet peak demands. Example: offload from GRIDIMadrid to EGEE.
• From the cluster to the Grid: Transfer queues used to offload tasks from a cluster to remote grid resources.

Opportunities to Learn about Globus

There are many good opportunities in coming months to learn about Globus technologies. Here are the events that I know about:

• (Kansas City, USA, June 2) 6-hour Globus tutorial at the Great Plains Network meeting.
• (Madison, USA, June 4-7) 6-hour Globus tutorial, as well as component-specific meetings, Gridway demo, etc., at the TeraGrid Meeting.
• (Brownsville, Texas, USA, June 8-10) Lots of Globus content in the 3-day Rio Grande Grid Summer School.
• (Denver, Colorado, USA, June 11) 3-hour tutorial on Globus solutions at the Virtual Observatories in Geosciences conference.
• (For Spanish speakers, Madrid, Spain, July 4-31) 100-hour UCM Summer Course on Grid Technology--based on GT4.
• (Mariefred, Sweden, July 8-20) A fair bit of Globus content in the International Summer School on Grid Computing.
• (Lincoln, Nebraska, USA, August 8-10) Lots of Globus content in this version of the 3-day Rio Grande Grid Summer School.
• (Vancouver, Canada, September 2-7) 3-hour BOF at CHEP.
• (Gdansk, Poland, September 9) Globus and Gridway tutorials at PPAM.
• (Karlsruhe, Germany, September 12) 6-hour Globus tutorial at the GridKA school.
• (Goa, India, December 18-21) 3-hour Globus tutorial at the HiPC conference.

For up-to-date information on these and other events, see the Globus Outreach Wiki.

Please let me know of others that I have missed.

Swift in Munich

I visited LRZ in Munich a few weeks ago. As well as meeting the excellent team there working on Globus for D-Grid and related projects, I gave a talk on our recent work with Swift. (See earlier post.) The talk includes results that my colleague Yong Zhao we will present at the IEEE Workshop on Scientific Workflows in July, and also some results on dynamic provisioning that Ioan Raicu obtained for a recent SC submission. I'm excited about how this work is going--we're running increasingly large computations for an increasing number of applications.

The abstract for the talk follows:

Gridway in Santiago de Compostela

I had the good fortune to attend the 1st Iberian Grid Infrastructure Conference this week, in beautiful Santiago de Compostela, Galicia, Spain. (Having also attended the 1st German eScience Conference two weeks before, I had to skip OGF 20 in Manchester last week.) It was an excellent meeting in every respect.

The keynote today was given by Professor Ignacio Martin Llorente from the Universidad Complutense de Madrid. He gave a nice survey of scheduling strategies for distributed grid systems and also presented the GridWay metascheduler developed in his group. This Globus-based system (and dev.globus project) enables large-scale, reliable and efficient sharing of computing resources (clusters, computing farms,servers, supercomputers...), managed by different local resource management systems, such as PBS, SGE, LSF, Condor..., within a single organization or distributed across several  administrative domains.

Writing this entry, I realize that I like GridWay for five reasons:

1. It provides powerful capabilities, including interfaces to many local resource managers, rich scheduling policies, interfaces from many schedulers (via so-called transfer queues.)
2. It is used by many people to solve real problems--including people who are not paid to use it! (Example users: UABGrid at the University of Alabama Birmingham, AstroGrid-D in Germany, projects in China and India.)
3. The GridWay team understands Globus deeply, and leverages Globus mechanisms to great advantage--just as the designers of those mechanisms intended.
4. The GridWay team has embraced the dev.globus community development process.
5. The GridWay team (like D-Grid and other brave souls) have been prepared to resist EU pressure to use only European software. Instead, they believe (correctly in my view) that we all benefit from the development and use of international software.

I also found this interesting comparison of GridWay and the EGEE workload management system, which shows GridWay in a good light. (Admittedly it was written by the GridWay team!)

Univa Globus Cluster Edition

We've been increasingly concerned within the Globus community on creating end-user "solutions" as well as the enabling technologies that make such solutions possible. Thus I've been excited by the UCLA Campus Grid Portal and the Swift efforts, for example. Another new effort, Univa Globus Cluster Edition, looks like a significant step forward in terms of level of integration. Combining Grid Engine, Ganglia, Rocks, the Globus Toolkit, and other components, it provides a complete open source cluster management and job scheduling solution. It will be interesting to see who picks up on this.

Globus at OGF 20, May 7-11

May 7-11 is the Open Grid Forum meeting in Manchester (OGF 20). I can't be there myself, as I was already in Europe this week (speaking at the very nice 1st German eScience Conference--more about that later). However, many colleagues from around the world will be there.

In particular, the Globus team will be participating in a broad set of session. There'll be a session in the Software Forum track on Tuesday, 2pm, which will include a discussion of what's new in the core software, and more in depth information about GridWay, the latest Globus project, and OGSA-DAI's upcoming 3.0 release. There will also be some "meet the developer" times in the Exhibit Hall, so you can chat with Globus developers informally. Additional information on Globus at OGF is available online. Or, you can contact Jennifer Schopf.

Forest Observation via GT4

It seems that every day I learn about a neat new GT4-based cyberinfrastructure project that I had previously not heard of. This week, Bill St Arnaud writes about the SAFORAH forest observation system:

The Canadian SAFORAH has many objectives - of which one is to measure the 
amount of carbon dioxide absorbed by Canadian forests. This cyber-infrastructure 
project also supports studies in bird habitat across Canada. It uses 
Globus Toolkit v.4 at all of the SAFORAH participating sites. Currently, 
four Canadian Forestry Centres located in Victoria British Columbia, 
Cornerbrook Newfoundland, Edmonton, Alberta and Laurentian Québec are 
operationally connected to the SAFORAH data grid.  SAFORAH offers 
Grid-enabled OGC services which are used to increase interoperability of 
EO data between SAFORAH and other geospatial information systems. The 
Grid-enabled OGC services consist of the following main components: 
Grid-enabled Web Map Service (GWMS), Grid-enabled Web Coverage Service 
(GWCS), Grid-enabled Catalog Service for Web (GCSW), Grid-enabled Catalog 
Service Federation (GCSF), Control Grid Service (CGS) and the Standard 
Grid Service Interfaces and OGC Standard User Interfaces.

I want to learn more!

Globus MEDICUS wins award

I wrote a while back about the Globus MEDICUS work being done in LA. They've now won an Internet2 IDEA Award. Stephan Erberich, project leader and Director Functional Imaging and Biomedical Informatics at the University of Southern California, writes:

Today we routinely expect information to be available on the Internet, but this is still not the case with medical information. We believe that making it available, in a secure fashion, is crucial: it has the potential to deliver better, more informed care at reduced cost. We believe that our Globus MEDICUS project takes important first steps toward this goal. Our system lets doctors and patients utilize the power of high-speed Internet to easily and securely share information. Much remains to be done, but we are gratified by the benefits that are already apparent.

Grid data mining tool based on GT4

The European Data Mining Grid project announced this week the release of their grid-enabled data mining software. This Apache-licensed, Globus-based software "is a general purpose  software facilitating user-friendly, Grid-based data mining."

It sounds pretty interesting. I hope someone will try it and let me know how it goes.

Webinar on "supercharging your cluster with Globus"

My colleagues Carl Kesselman and Steve Tuecke are presenting a Web-based seminar (aka Webinar) on March 20 on "supercharging your cluster with Univa Globus." Univa Globus is the commercially supported version of Globus--the same  open source software, but with someone to call if things go wrong.

The abstract sounds a little dull, but Carl and Steve are both excellent speakers and have interesting things to say, so this should be worth listening to:

Dev.Globus Announces First Escalation and Welcomes Six New Incubator Projects

Some good news from dev.globus--a first incubator project, the GridWay Metascheduler project, has completed incubation and is now a full Globus project. In addition, six new projects have joined the dev.globus incubation process, bringing the total number of projects to twenty-two.

A more detailed description from an announcement prepared by Jennifer Schopf, dev.globus Incubator Management Project (IMP) chair:

Globus at sea

Italian Globus enthusiast Raffaele Montella sent me this picture of the sailboat Sarima V  that he races out of Naples. Apparently he uses grid to good effect in his racing, using an online Globus-based forecasting service to obtain up-to-date weather forecasts prior to each race. Being a sailor myself, I can only applaud (and feel jealous).

He has a paper on this work in the upcoming Grid and Pervasive Computing conference, to be held in Paris May 2-4. Title: "Development of a GT4-based Resource Broker Service: an application to on-demand weather and marine forecasting." I'll post a pointer to the p