Ian Foster

Wikisauri: Thesauri from Wikipedia

David Milne, Olena Medelyan and Ian Witten have a nice paper at this conference I am attending in Hong Kong, on mining domain-specific thesauri from Wikipedia. As they say:

How can you obtain a thesaurus to support a library of documents in a particular domain? Manual construction is prohibitively expensive; automatic generation is woefully inaccurate. General thesauri do not incorporate the specialist terminology that pervades our professions, nor can they keep pace with the deluge of new topics and concepts that arrive each day. Yet a contemporary resource that incorporates expertise in all fields of human endeavour already exists: the widely known Wikipedia.

Basically, they  mine the structure of Wikipedia (its redirects, hierarchy, and hyperlinks) to infer the equivalence, hierarchical, and associative relations needed to build a thesaurus. Comparison with a professionally prepared thesaurus (from agriculture) shows that this approach can be effective. Another example of crowdsourcing, based on a rather nonobvious use of the work of its contributors.

The flattening of supercomputers

Matei Ripeanu has an interesting brief article in IEEE Distributed Systems Online in which he analyzes the shape of the by-now-(in)famous Top 500 list of supercomputers, released every six months since 1993.

He notes first that a plot of performance vs. rank gives a power law. Not in itself surprising. But then he notes that the power law coefficient is getting smaller over time: in other words, the bigger machines are, on average, getting faster more slowly than the slower machines. Thus, for example, the bottom 25 machines in the Top500, if aggregated together, would match only the #30 machine in 1993, but match the #5 machine in 2005.

Why this change? Alex Szalay attributes it to the top500 spurring people to buy bigger computers. (I.e., the act of measuring supercomputer evolution perturbs that evolution!) A provocative thesis, but hard to evaluate. Matei attributes it simply to the increasing ease with which one can aggregate systems.

Hadoop on EC2

Here's something neat (and details here).

Hadoop, an open source clone of Google FS and MapReduce, can be run on top of Amazon EC2, a hosting service that allows leasing servers on an hourly basis.

As Greg Linden goes on to say:

Developers may now be able to rapidly bring up hundreds of servers, run a massive parallel computation on them using Hadoop's MapReduce implementation, and then shut down all the instances, all with low effort and at low cost. Very cool.

My colleague Tim Freeman points out that you can run those same VMs on your own resources using the Globus Workspace service.

Grid in Spanish/La Grid en Español

A recent article on "la Computacion Grid" in a Spanish Newspaper, by Borja Sotomayor. He concludes:

La Computación Grid, a pesar de ser una tecnología con bastante madurez, sigue estando en constante evolución. Actualmente existen muchas grids computacionales (como EGEE y TeraGrid), construidas para dar salida principalmente a problemas científicos, pero todavía no existe «La Grid». De la misma manera que internet nació en el ámbito científico para luego llegar al público general, lo mismo puede esperarse, a largo plazo, de la Computación Grid. Cuando la tecnología madure lo suficiente, será posible que cualquier usuario, desde su ordenador personal de casa, pueda enviar complejos trabajos computacionales a «La Grid», como si tuviésemos un supercomputador en el salón de casa.

Borja asked me to write a few words, too, which I did. I didn't realize my Spanish was so fluent ...

Connections Between Grid and P2P

What do Grid and P2P have to do with each other? Adriana Iamnitchi and I wrote a paper on that question a few years ago. The title is "On Death, Taxes, and the Convergence of Peer-to-Peer and Grid Computing", which we explained as follows:

It has been reported that life holds but two certainties, death and taxes. And indeed, it does appear that any society-and in the context of this article, any large-scale distributed system-must address both death (failure) and the establishment and maintenance of infrastructure (which we assert is a major motivation for taxes, so as to justify our title!).

IBM has a secret island headquarters hideaway inside a computer game

Sometime soon, I hope to say something intelligent about Second Life and what it means for numerous things, including science and innovation. First, though, I need to get a computer with a graphics card modern enough to run it myself.

In the meantime, you can read some interesting comments by Charlie Stross (who coined the title for this post, and fears his science fiction magnus opus is being overtaken by events) and Irving Wladawsky-Berger, the very smart IBM executive who has a history of getting IBM into wacky things like Linux, parallel computing, and ... well, now it seems, Second Life. If you prefer traditional dead-tree media, the October 2006 issue of Wired has a nice article.