Xavier Llorà

Meandre 2.0 Alpha Preview = Scala + MongoDB

Xavier — Thu, 15 Jul 2010 19:45:00 +0000

A lot of water under the bridge has gone by since the first release of Meandre 1.4.X series. In January I went back to the drawing board and start sketching what was going to be 1.5.X series. The slide deck embedded above is a extended list of the thoughts during the process. As usual, I started collecting feedback from people using 1.4.X in production, things that worked, things that needed improvement, things that were just plain over complicated. The hot recurrent topics that people using 1.4.X could be mainly summarized as:

Complex execution concurrency model based on traditional semaphores written in Java (mostly my maintenance nightmare when changes need to be introduced)
Server performance bounded by JENA‘s persistent model implementation
State caching on individual servers to boost performance increases complexity of single-image cluster deployments
Could-deployable infrastructure, but not cloud-friendly infrastructure

As I mentioned, these elements where the main ingredients to target for 1.5.X series. However as the redesign moved forward, the new version represented a radical disruption from 1.4.X series and eventually turned up to become the 2.0 Alpha version described here. The main changes that forced this transition are:

Cloud-friendly infrastructure required rethinking of the core functionalities
Drastic redesign of the back-end state storage
Revisited flow execution engine to support flow execution
Changes on the API that render returned JSON documents incompatible with 1.4.X

Meandre 2.0 (currently already available in the the SVN trunk) has been rewritten from scratch using Scala. That decision was motivated to benefit from the Actor model provided by Scala (modeled after Erlang‘s actors). Such model greatly simplify the mechanics of the infrastructure, but it also powered the basis of Snowfield (the effort to create a scalable distributed flow execution engine for Meandre flows). Also, the Scala language expressiveness has greatly reduced the code based size (2.0 code base is roughly 1/3 of the size of 1.4.X series) greatly simplifying the maintenance activities the infrastructure will require as we move forward.

The second big change that pushed the 2.0 Alpha trigger was the redesign of the back end state storage. 1.4.X series heavily relied on the relational storage for persistent RDF models provided by JENA. For performance reasons, JENA caches the model in memory and mostly assumes ownership of the model. Hence, if you want to provide a single-image Meandre cluster you need to inject into JENA cache coherence mechanics, greatly increasing the complexity. Also, the relational implementation relies on the mapping model into a table and triple into a row (this is a bit of a simplification). That implies that large number of SQL statements need to be generated to update models, heavily taxing the relational storage when changes on user repository data needs to be introduced.

An ideal cloud-friendly Meandre infrastructure should not maintain state (neither voluntarily, neither as result of JENA back end). Thus, a fast and scalable back end storage could allow infrastructure servers to maintain no state and be able to provide the appearance of a single image cluster. After testing different alternatives, their community support, and development roadmap, the only option left was MongoDB. Its setup simplicity for small installations and its ability to easily scale to large installations (including cloud-deployed ones) made MongoDB the candidate to maintain state for Meandre 2.0. This was quite a departure from 1.4.x series, where you had the choice to store state via JENA on an embedded Derby or an external MySQL server.

A final note on the building blocks that made possible 2.0 series. Two other side projects where started to support the development of what will become Meandre 2.0.X series:

Crochet: Crochet targets to help quickly prototype REST APIs relying on the flexibility of the Scala language. The initial ideas for Crochet were inspired after reading Gabriele Renzi post on creating a picoframework with Scala (see http://www.riffraff.info/2009/4/11/step-a-scala-web-picoframework) and the need for quickly prototyping APIs for pilot projects. Crochet also provides mechanisms to hide repetitive tasks involved with default responses and authentication/authorization piggybacking on the mechanics provided by application servers.
Snare: Snare is a coordination layer for distributed applications written in Scala and relies and MongoDB to implement its communication layer. Snare implements a basic heartbeat system and a simple notification mechanism (peer-to-peer and broadcast communication). Snare relies on MongoDB to track heartbeat and notification mailboxes.

Minor Update

Xavier — Fri, 25 Jun 2010 05:04:12 +0000

If you used to check my Twitter stream in the center column at the bottom of my blog, you would have noticed I have just replaced it with my Google Buzz profile stream instead. Yes, you can still see the my Twitter activity, but you will also be able to (1) see aggregated content/activity coming from other sources in one place, (2) subscribe to the stream, and (3) comment on each of the entries. As, I said, a minor update to improve its functionality a bit more.

IWLCS 2010 – Discussion session on LCS / XCS(F)

Xavier — Mon, 21 Jun 2010 22:04:08 +0000

I just got an email from Martin Butz about a discussion session being planned for IWLCS 2010 and his request to pass it along.

Hope all is well and you are going to attend GECCO this year.

Regardless if you attend or not:

Jaume asked me to lead a discussion session on

“LCS representations, operators, and scalability – what is next?”

… or similar during IWLCS… Basically everything besides datamining, because there will be another session on that topic.

So, I am sure you all have some issues in mind that you think should be tackled / addressed / discussed at the workshop and in the near future.

Thus, I would be very happy to receive a few suggestions from your side – anything is welcome – I will then compile the points raised in a few slides to try and get the discussion going at the workshop.

Thank you for any feedback you can provide.

Looking forward to seeing you soon!

Martin

P.S.: Please feel free to also forward this message or tell me, if you think this Email should be still sent to other people…
—-

PD Dr. Martin V. Butz <butz@psychologie.uni-wuerzburg.de>

Department of Psychology III (Cognitive Psychology)
Roentgenring 11
97070 Wuerzburg, Germany
http://www.coboslab.psychologie.uni-wuerzburg.de/people/martin_v_butz/
http://www.coboslab.psychologie.uni-wuerzburg.de
Phone: +49 (0)931 31 82808
Fax: +49 (0)931 31 82815

LCS and Software Development

Xavier — Fri, 18 Jun 2010 16:43:25 +0000

“On the Road to Competence” is a slide deck by Jurgen Appelo with interesting analogies between learning classifier systems and software development. Definitely worth taking a look at it.

GAssist and GALE Now Available in Python

Xavier — Fri, 11 Jun 2010 17:06:35 +0000

Ryan Urbanowicz has released Python versions of GAssits and GALE!!! Yup, so excited to see a new incarnation of GALE doing the rounds. I cannot wait to get my hands on it. Ryan has also done an excellent job porting UCS, XCS, and MCS to Python and making those implementations available via “LCS & GBML central” for people to use. I think Ryan’s efforts deserve recognition. His code is helping others to have an easier entry to the LCS and GBML.

More information about Ryan’s implementations can found below

Side note: my original GALE implementation can also be downloaded here.

GECCO 2010 Around the Bend, GECCO 2011 Gearing Up

Xavier — Wed, 09 Jun 2010 04:35:01 +0000

GECCO 2010 is getting ready to open July 7th at Portland (Oregon). But GECCO does not sleep and the 2011 edition is gearing up to take the torch after Portland.

There is currently an announcement “GECCO 2011: Call for New Frontiers Track Proposals”. The deadline is June 30th, and you can get more information at http://goo.gl/m4my or at the public GECCO 2011 public announcements Wave (embedded below).

You can follow GECCO 2011 on Twitter at @GECCO2011, join the group at GECCO 2011 on FaceBook or just Wave with GECCO 2011.

LCS & GBML Central Gets a New Home

Xavier — Fri, 04 Jun 2010 04:32:38 +0000

Today I finished migrating the LCS & GBML Central site from its original URL (http://lcs-gbml.ncsa.uiuc.edu) to a more permanent and stable home located at http://gbml.org. The original site is already currently redirecting the trafic to the new site, and it will be doing so for a while to help people transition and update bookmarks and feed readers.

I have introduced a few changes to the functionality of the original site. Functional changes can be mostly summarized by (1) dropping the forums section and (2) closing comments on posts and pages. Both functionalities, rarely used in their current form, have been replaced by a simpler public embedded Wave reachable at http://gbml.org/wave. The goal, provide people in the LCS & GBML community a simpler way to discuss, share, and hang out.

About the feeds being aggregated, I have revised the list and added the feeds now available of the table of contents from

I have also added a few other links to relevant research groups doing work on related areas. Please, leave a comment on this post if you know/have a related site that could be aggregated, or if there are missing links to research groups or useful resources.

Scaling eCGA Model Building via Data-Intensive Computing

Xavier — Thu, 08 Apr 2010 16:17:39 +0000

I just uploaded the technical report of the paper we put together for CEC 2010 on how we can scale up eCGA using a MapReduce approach. The paper, besides exploring the Hadoop implementation, it also presents some very compelling results obtained with MongoDB (a document based store able to perform parallel MapReduce tasks via sharding). The paper is available as PDF and PS.

Abstract:
This paper shows how the extended compact genetic algorithm can be scaled using data-intensive computing techniques such as MapReduce. Two different frameworks (Hadoop and MongoDB) are used to deploy MapReduce implementations of the compact and extended com- pact genetic algorithms. Results show that both are good choices to deal with large-scale problems as they can scale with the number of commodity machines, as opposed to previous ef- forts with other techniques that either required specialized high-performance hardware or shared memory environments.

Soaring the Clouds with Meandre

Xavier — Mon, 15 Mar 2010 22:55:11 +0000

You may find the slide deck and the abstract for the presentation we delivered today at the “Data-Intensive Research: how should we improve our ability to use data” workshop in Edinburgh.

Abstract

This talk will focus a highly scalable data intensive infrastructure being developed at the National Center for Supercomputing Application (NCSA) at the University of Illinois and will introduce current research efforts to tackle the challenges presented by big-data. Research efforts include exploring potential ways of integration between cloud computing concepts—such as Hadoop or Meandre—and traditional HPC technologies and assets. These architecture models contrast significantly, but can be leveraged by building cloud conduits that connect these resources to provide even greater flexibility and scalability on demand. Orchestrating the physical computational environment requires innovative and sophisticated software infrastructure that can transparently take advantage of the functional features and to negotiate the constraints imposed by this diversity of computational resources. Research conducted during the development of the Meandre infrastructure has lead to the production of an agile conductor able to leverage the particular advantages in the physical diversity. It can also be implemented as services and/or in the context of another application benefitting from it reusability, flexibility, and high-scalability. Some example applications and an introduction to the data intensive infrastructure architecture will be presented to provide an overview of the diverse scope of Meandre usages. Finally, a case will be presented showing how software developers and system designers can easily transition to these new paradigms to address the primary data-deluge challenges and to soar to new heights with extreme application scalability using cloud computing concepts.

Fast REST API prototyping with Crochet and Scala

Xavier — Thu, 21 Jan 2010 14:59:59 +0000

I just finished committing the last changes to Crochet and tagged version 0.1.4vcli now publicly available on GitHub (http://github.com/xllora/Crochet). Also feel free to visit the issues page in case you run into question/problems/bugs.

Motivation

Crochet is a light weight web framework oriented to rapid prototyping of REST APIs. If you are looking for a Rails like framework written in Scala, please take a look at Lift at http://liftweb.net/ instead.

Crochet targets quick prototyping of REST APIs relying on the flexibility of the Scala language. The initial ideas for Crochet were inspired while reading Gabriele Renzi post on creating the STEP picoframework with Scala and the need for quickly prototyping APIs for pilot projects. Crochet also provides mechanisms to hide repetitive tasks involved with default responses and authentication/authorization piggybacking on the mechanics provided by application servers.

Who uses Crochet?

Crochet was born from the need for quickly prototyping REST APIs which required exposing legacy code written in Java. I have been actively using Crochet to provide REST APIs for a variety of projects developed at the National Center for Supercomputing Applications. One of the primary adopters and movers of Crochet is the Meandre Infrastructure for data-intensive computing developed under the SEASR project.

Crochet in 2 minuts

Before you start please check you have Scala installed on your system. You can find more information on how to get Scala up and running here.

Get the latest Crochet jar from the Downloads section at GitHub and the third party dependencies.

Copy the following code into a file named hello-world.scala.

import crochet._
new Crochet {
     get("/message") { 
         <html>
               <head><title>Hello World</title></head>
               <body><h1>Hello World!</h1></body>
         </html>
     }
} on 8080

Get your server up and running by running (please change the version number if needed)
```
$ scala -cp crochet-0.1.4.jar:crochet-3dparty-libraries-0.1.X.jar hello-world.scala
```

You just have your first _Crochet_ API up and running. You can check the API working by opening your browser and pointing it to http://localhost:8080/message and you should get the message Hello World! back.

Where to go from here?

You will find more information on the Crochet wiki at GitHub. The wiki contains basic information as a QuickStart guide (which also includes how to deal with static content), descriptions of the basic concepts used in Crochet, and several examples that can get up and running fast.