Xavier Llorà » Presentations

Meandre 2.0 Alpha Preview = Scala + MongoDB

Xavier — Thu, 15 Jul 2010 19:45:00 +0000

A lot of water under the bridge has gone by since the first release of Meandre 1.4.X series. In January I went back to the drawing board and start sketching what was going to be 1.5.X series. The slide deck embedded above is a extended list of the thoughts during the process. As usual, I started collecting feedback from people using 1.4.X in production, things that worked, things that needed improvement, things that were just plain over complicated. The hot recurrent topics that people using 1.4.X could be mainly summarized as:

Complex execution concurrency model based on traditional semaphores written in Java (mostly my maintenance nightmare when changes need to be introduced)
Server performance bounded by JENA‘s persistent model implementation
State caching on individual servers to boost performance increases complexity of single-image cluster deployments
Could-deployable infrastructure, but not cloud-friendly infrastructure

As I mentioned, these elements where the main ingredients to target for 1.5.X series. However as the redesign moved forward, the new version represented a radical disruption from 1.4.X series and eventually turned up to become the 2.0 Alpha version described here. The main changes that forced this transition are:

Cloud-friendly infrastructure required rethinking of the core functionalities
Drastic redesign of the back-end state storage
Revisited flow execution engine to support flow execution
Changes on the API that render returned JSON documents incompatible with 1.4.X

Meandre 2.0 (currently already available in the the SVN trunk) has been rewritten from scratch using Scala. That decision was motivated to benefit from the Actor model provided by Scala (modeled after Erlang‘s actors). Such model greatly simplify the mechanics of the infrastructure, but it also powered the basis of Snowfield (the effort to create a scalable distributed flow execution engine for Meandre flows). Also, the Scala language expressiveness has greatly reduced the code based size (2.0 code base is roughly 1/3 of the size of 1.4.X series) greatly simplifying the maintenance activities the infrastructure will require as we move forward.

The second big change that pushed the 2.0 Alpha trigger was the redesign of the back end state storage. 1.4.X series heavily relied on the relational storage for persistent RDF models provided by JENA. For performance reasons, JENA caches the model in memory and mostly assumes ownership of the model. Hence, if you want to provide a single-image Meandre cluster you need to inject into JENA cache coherence mechanics, greatly increasing the complexity. Also, the relational implementation relies on the mapping model into a table and triple into a row (this is a bit of a simplification). That implies that large number of SQL statements need to be generated to update models, heavily taxing the relational storage when changes on user repository data needs to be introduced.

An ideal cloud-friendly Meandre infrastructure should not maintain state (neither voluntarily, neither as result of JENA back end). Thus, a fast and scalable back end storage could allow infrastructure servers to maintain no state and be able to provide the appearance of a single image cluster. After testing different alternatives, their community support, and development roadmap, the only option left was MongoDB. Its setup simplicity for small installations and its ability to easily scale to large installations (including cloud-deployed ones) made MongoDB the candidate to maintain state for Meandre 2.0. This was quite a departure from 1.4.x series, where you had the choice to store state via JENA on an embedded Derby or an external MySQL server.

A final note on the building blocks that made possible 2.0 series. Two other side projects where started to support the development of what will become Meandre 2.0.X series:

Crochet: Crochet targets to help quickly prototype REST APIs relying on the flexibility of the Scala language. The initial ideas for Crochet were inspired after reading Gabriele Renzi post on creating a picoframework with Scala (see http://www.riffraff.info/2009/4/11/step-a-scala-web-picoframework) and the need for quickly prototyping APIs for pilot projects. Crochet also provides mechanisms to hide repetitive tasks involved with default responses and authentication/authorization piggybacking on the mechanics provided by application servers.
Snare: Snare is a coordination layer for distributed applications written in Scala and relies and MongoDB to implement its communication layer. Snare implements a basic heartbeat system and a simple notification mechanism (peer-to-peer and broadcast communication). Snare relies on MongoDB to track heartbeat and notification mailboxes.

LCS and Software Development

Xavier — Fri, 18 Jun 2010 16:43:25 +0000

“On the Road to Competence” is a slide deck by Jurgen Appelo with interesting analogies between learning classifier systems and software development. Definitely worth taking a look at it.

From Galapagos to Twitter: Darwin, Natural Selection, and Web 2.0

Xavier — Fri, 18 Sep 2009 18:14:05 +0000

Yesterday I was visiting Monmouth College to participate on the Darwinpalooza which commemorates the 200th anniversary of Charles Darwin’s birth and the 150th anniversary of the publication of On the Origin of Species. After scratching my head about about what to present, I came out with quite a mix. You will find the abstract of the talk below, as well as the slides I used.

Abstract: One hundred and fifty years have passed since the publication of Darwin’s world-changing manuscript “The Origins of Species by Means of Natural Selection”. Darwin’s ideas have proven their power to reach beyond the biology realm, and their ability to define a conceptual framework which allows us to model and understand complex systems. In the mid 1950s and 60s the efforts of a scattered group of engineers proved the benefits of adopting an evolutionary paradigm to solve complex real-world problems. In the 70s, the emerging presence of computers brought us a new collection of artificial evolution paradigms, among which genetic algorithms rapidly gained widespread adoption. Currently, the Internet has propitiated an exponential growth of information and computational resources that are clearly disrupting our perception and forcing us to reevaluate the boundaries between technology and social interaction. Darwin’s ideas can, once again, help us understand such disruptive change. In this talk, I will review the origin of artificial evolution ideas and techniques. I will also show how these techniques are, nowadays, helping to solve a wide range of applications, from life science problems to twitter puzzles, and how high performance computing can make Darwin ideas a routinary tool to help us model and understand complex systems.

Large Scale Data Mining using Genetics-Based Machine Learning

Xavier — Wed, 15 Jul 2009 21:56:17 +0000

Below you may find the slides of the GECCO 2009 tutorial that Jaume Bacardit and I put together. Hope you enjoy it.

Slides

Abstract

We are living in the peta-byte era.We have larger and larger data to analyze, process and transform into useful answers for the domain experts. Robust data mining tools, able to cope with petascale volumes and/or high dimensionality producing human-understandable solutions are key on several domain areas. Genetics-based machine learning (GBML) techniques are perfect candidates for this task, among others, due to the recent advances in representations, learning paradigms, and theoretical modeling. If evolutionary learning techniques aspire to be a relevant player in this context, they need to have the capacity of processing these vast amounts of data and they need to process this data within reasonable time. Moreover, massive computation cycles are getting cheaper and cheaper every day, allowing researchers to have access to unprecedented parallelization degrees. Several topics are interlaced in these two requirements: (1) having the proper learning paradigms and knowledge representations, (2) understanding them and knowing when are they suitable for the problem at hand, (3) using efficiency enhancement techniques, and (4) transforming and visualizing the produced solutions to give back as much insight as possible to the domain experts are few of them.

This tutorial will try to answer this question, following a roadmap that starts with the questions of what large means, and why large is a challenge for GBML methods. Afterwards, we will discuss different facets in which we can overcome this challenge: Efficiency enhancement techniques, representations able to cope with large dimensionality spaces, scalability of learning paradigms. We will also review a topic interlaced with all of them: how can we model the scalability of the components of our GBML systems to better engineer them to get the best performance out of them for large datasets. The roadmap continues with examples of real applications of GBML systems and finishes with an analysis of further directions.

Data-Intensive Computing for Competent Genetic Algorithms: A Pilot Study using Meandre

Xavier — Tue, 14 Jul 2009 04:15:51 +0000

Below you may find the slides I used during GECCO 2009 to present the paper titled “Data-Intensive Computing for Competent Genetic Algorithms: A Pilot Study using Meandre”. An early preprint in form of technical report can be found as an IlliGAL TR No. 2009001 or the full paper at the ACM digital library

NIGEL 2006 Part VI: Bacardit

Xavier — Mon, 13 Jul 2009 23:54:27 +0000

After coming back from GECCO I just uploaded the last of the NIGEL 2006 talks at LCS & GBML Central. This last talk was by Jaume Bacardit and GBML for protein structure prediction.

NIGEL 2006 Part V: Bernardó vs. Lanzi

Xavier — Mon, 29 Jun 2009 18:30:14 +0000

After the vacation break, two more NIGEL 2006 talks are available at LCS & GBML Central. This week Ester Bernardó presents how LCS can perform in the presence of class imbalance, whereas Lanzi continues his quest on computed predictions.

NIGEL 2006 Part IV: Llorà vs. Casillas

Xavier — Mon, 08 Jun 2009 21:20:02 +0000

Two more NIGEL 2006 talks are available at LCS & GBML Central. This week Xavier Llorà presents how linkage learning can be achieve in Pittsburgh LCS, whereas Jorge Casillas reviews his work using XCS and Fuzzy LCS.

NIGEL 2006 Part III: Butz vs. Barry

Xavier — Tue, 19 May 2009 14:38:05 +0000

NIGEL 2006 talks is available at LCS & GBML Central. This week Martin Butz review reviews the state of the union of XCS, where as Alwyn Barry introduces the theoretical framework for LCS that he and Jan Drugowitsch worked on.

NIGEL 2006 Part II: Dasgupta vs. Booker

Xavier — Mon, 11 May 2009 16:08:38 +0000

The second weekly release of NIGEL 2006 talks is available at LCS & GBML Central. This week Dipankar Dasgupta reviews the negative selection algorithm, where as Lashon Booker travels in time to the past and future of learning classifier systems.