Xavier Llorà » distributed computing

Meandre: Semantic-Driven Data-Intensive Flows in the Clouds

Xavier — Sat, 15 Nov 2008 23:34:02 +0000

by Llorà, X., Ács, B., Auvil, L., Capitanu, B., Welge, M.E., Goldberg, D.E. (2008).

This paper has been accepted at the 4th IEEE International Conference on e-Science. An early draft of the paper can be found as IlliGAL technical report 2008013. You can download the pdf here. More information is also available at the Meandre website as part of the SEASR project.

Abstract:

Data-intensive flow computing allows efficient processing of large volumes of data otherwise unapproachable. This paper introduces a new semantic-driven data-intensive flow infrastructure which: (1) provides a robust and transparent scalable solution from a laptop to large-scale clusters,(2) creates an unified solution for batch and interactive tasks in high-performance computing environments, and (3) encourages reusing and sharing components. Banking on virtualization and cloud computing techniques the Meandre infrastructure is able to create and dispose Meandre clusters on demand, being transparent to the final user. This paper also presents a prototype of such clustered infrastructure and some results obtained using it.

ICEIS 2008: Blogging summary and final strings

Xavier — Wed, 18 Jun 2008 15:28:37 +0000

If you are looking for a list of the related blogging done during ICEIS 2008 just follow this link. During Sunday morning I run into Angel A. Juan, an assistant professor at Open University of Catalonia (UOC), interested on analyzing online teaching efforts and how tools to assist professors monitoring students performance on online media. I visited him yesterday at his office and we got and interesting exchange of ideas. Most of them revolved around the work we have conducted under the DISCUS project, and how similar is our efforts on marketing focus groups and their online teaching environment. His group, Distributed, Parallel and Collaborative Systems, was also interested on the work done under the SEASR project, mostly focusing on the Meadre infrastructure for data-intensive flow computing we are getting close to release.

ICEIS 2008: Final sprint and Ricardo Baeza-Yates

Xavier — Mon, 16 Jun 2008 11:10:46 +0000

This is the final sprint for ICEIS. I have been mostly focusing on posters this morning. It his hard to pick one up. I would just say that there was some interesting work on personalized recommender systems—paper 219. But as I said, there were a bunch of interesting ones and quite a few interesting by-the-poster conversations. Actually, I am having a very interesting time after the mix of attendees’ profiles.

The morning finally meandered into Ricardo Baeza-Yates‘s keynote talk. After the initial technical problems—presentation mode of OpenOffice running on Ubuntu 8.0.4 got up just 75% of the slide area—they finally succeeded on getting something up and get the talk started. This was a pretty technical talk about Yahoo! research effort on caching to improve the performance and also help scalability and contain cost on the coming years. Besides several cache techniques, he also presented a bunch of possible paralyzation models based on document/term partitions. A thing he breezed over was the machine learning model for classifying queries. That surfaced several places, from predicting common and rare content, to frequent, unfrequent, and rare queries. I was glad that the technical problems were solve and we could enjoy it.

And the conference is finally close. Next year, Milan.

ICEIS 2008: Blogging from Barcelona (Friday Morning)

Xavier — Fri, 13 Jun 2008 14:26:52 +0000

The hotel finally got the wireless system rebooted, so I am finally back up. I found quite unusual to open the ICEIS 2008 conference with a panel form by the four invited speakers: Moira Norrie (Global Information System Group, ETH), Jorge Cardoso (SAP), Jean-Marie Favre (UFR, IMA), Ricardo Baeza-Yates (Yahoo! Research Barcelona). Each of them did a short introduction of their invited talks, and just three questions were asked. Since the conference started at 11am, after the panel, lunch break

ZooKeeper and distributed applications

Xavier — Thu, 22 May 2008 13:39:36 +0000

Lately I have been exploring different alternatives for coordinating the execution of distributed applications. Yes, you guessed it right, I am working on the distribution of the execution of Meandre flows. Chopping the data-intensive flow and mapping the chunks onto a set of distributed processors requires several elements (graph analysis, resource management, etc.). However, the basic element that needs to be solved first is the need for a reliable and scalable coordination system.

During my trip to the Hadoop Summit and the Big Data Computing Study Group I ran into the ZooKeeper project, a Yahoo Research project. In their own words:

ZooKeeper is a high available and reliable coordination system. Distributed applications use ZooKeeper to store and mediate updates key configuration information. ZooKeeper can be used for leader election, group membership, configuration maintenance, etc.

There is also an interesting introductory lesson here, and some recipes for the most common data structures (queues, priority queues, distributed locks, etc.) are also available here. Looks promising, and may make the coordination of the distributed execution of a Meandre flows easier. The other reason that pushed me to explore this directions was that one of the Hadoop Summit highlights was the adoption of the ZooKeeper project by Hadoop itself.