Abstract:
This paper shows how the extended compact genetic algorithm can be scaled using data-intensive computing techniques such as MapReduce. Two different frameworks (Hadoop and MongoDB) are used to deploy MapReduce implementations of the compact and extended com- pact genetic algorithms. Results show that both are good choices to deal with large-scale problems as they can scale with the number of commodity machines, as opposed to previous ef- forts with other techniques that either required specialized high-performance hardware or shared memory environments.

Related posts:

1. Scaling Genetic Algorithms using MapReduce
2. Data-Intensive Computing for Competent Genetic Algorithms: A Pilot Study using Meandre
3. [BDCSG2008] Data-Intensive Scalable Computing (Randy Bryant)

Abstract

This talk will focus a highly scalable data intensive infrastructure being developed at the National Center for Supercomputing Application (NCSA) at the University of Illinois and will introduce current research efforts to tackle the challenges presented by big-data. Research efforts include exploring potential ways of integration between cloud computing concepts—such as Hadoop or Meandre—and traditional HPC technologies and assets. These architecture models contrast significantly, but can be leveraged by building cloud conduits that connect these resources to provide even greater flexibility and scalability on demand. Orchestrating the physical computational environment requires innovative and sophisticated software infrastructure that can transparently take advantage of the functional features and to negotiate the constraints imposed by this diversity of computational resources. Research conducted during the development of the Meandre infrastructure has lead to the production of an agile conductor able to leverage the particular advantages in the physical diversity. It can also be implemented as services and/or in the context of another application benefitting from it reusability, flexibility, and high-scalability. Some example applications and an introduction to the data intensive infrastructure architecture will be presented to provide an overview of the diverse scope of Meandre usages. Finally, a case will be presented showing how software developers and system designers can easily transition to these new paradigms to address the primary data-deluge challenges and to soar to new heights with extreme application scalability using cloud computing concepts.

Related posts:

1. Meandre: Semantic-Driven Data-Intensive Flows in the Clouds
2. Meandre: Semantic-Driven Data-Intensive Flow Engine
3. Data-Intensive Computing for Competent Genetic Algorithms: A Pilot Study using Meandre

Abstract:Genetic algorithms(GAs) are increasingly being applied to large scale problems. The traditional MPI-based parallel GAs do not scale very well. MapReduce is a powerful abstraction developed by Google for making scalable and fault tolerant applications. In this paper, we mould genetic algorithms into the the MapReduce model. We describe the algorithm design and implementation of GAs on Hadoop, the open source implementation of MapReduce. Our experiments demonstrate the convergence and scalability upto 105 variable problems. Adding more resources would enable us to solve even larger problems without any changes in the algorithms and implementation.

The draft of the paper can be downloaded as IlliGAL TR. No. 2009007. For more information see the IlliGAL technical reports web site.

Related posts:

1. Scaling eCGA Model Building via Data-Intensive Computing
2. Data-Intensive Computing for Competent Genetic Algorithms: A Pilot Study using Meandre
3. Data-Intensive Computing for Competent Genetic Algorithms: A Pilot Study using Meandre

Slides

Abstract

We are living in the peta-byte era.We have larger and larger data to analyze, process and transform into useful answers for the domain experts. Robust data mining tools, able to cope with petascale volumes and/or high dimensionality producing human-understandable solutions are key on several domain areas. Genetics-based machine learning (GBML) techniques are perfect candidates for this task, among others, due to the recent advances in representations, learning paradigms, and theoretical modeling. If evolutionary learning techniques aspire to be a relevant player in this context, they need to have the capacity of processing these vast amounts of data and they need to process this data within reasonable time. Moreover, massive computation cycles are getting cheaper and cheaper every day, allowing researchers to have access to unprecedented parallelization degrees. Several topics are interlaced in these two requirements: (1) having the proper learning paradigms and knowledge representations, (2) understanding them and knowing when are they suitable for the problem at hand, (3) using efficiency enhancement techniques, and (4) transforming and visualizing the produced solutions to give back as much insight as possible to the domain experts are few of them.

This tutorial will try to answer this question, following a roadmap that starts with the questions of what large means, and why large is a challenge for GBML methods. Afterwards, we will discuss different facets in which we can overcome this challenge: Efficiency enhancement techniques, representations able to cope with large dimensionality spaces, scalability of learning paradigms. We will also review a topic interlaced with all of them: how can we model the scalability of the components of our GBML systems to better engineer them to get the best performance out of them for large datasets. The roadmap continues with examples of real applications of GBML systems and finishes with an analysis of further directions.

Related posts:

1. Observer-Invariant Histopathology using Genetics-Based Machine Learning
2. Deadline extended for special issue on Metaheuristics for Large Scale Data Mining
3. Machine learning & Statistical Learning in R

The morning finally meandered into Ricardo Baeza-Yates‘s keynote talk. After the initial technical problems—presentation mode of OpenOffice running on Ubuntu 8.0.4 got up just 75% of the slide area—they finally succeeded on getting something up and get the talk started. This was a pretty technical talk about Yahoo! research effort on caching to improve the performance and also help scalability and contain cost on the coming years. Besides several cache techniques, he also presented a bunch of possible paralyzation models based on document/term partitions. A thing he breezed over was the machine learning model for classifying queries. That surfaced several places, from predicting common and rare content, to frequent, unfrequent, and rare queries. I was glad that the technical problems were solve and we could enjoy it.

And the conference is finally close. Next year, Milan.

Related posts:

1. ICEIS 2008: Blogging summary and final strings
2. ICEIS 2008: First Friday afternoon session
3. ICEIS 2008: Blogging from Barcelona (Friday Morning)

On the second side of this which may actually be more exciting is the issue of–instead of the structured data world of the relational database but the semi–the semi-structured world. You look at what is being done today with CouchDB, you look at Amazon ScaleDB, to a lesser extent but to a similar extent you–not ScaleDB, SimpleDB–to a lesser extent or a similar extent Tokyo Cabinet, those databases are really kind of fascinating because those databases are redefining really how we access data and how we are going to be searching and using data. So there’s a whole world out there that’s just starting to open up in that direction.

For a while now, I have been using different flavors of metadata stores. Everything tends to work nice and dandy as long as you do not push the storage volume too far. For instance, together with Bernie Acs at NCSA, we have run experiments where we could deal with up to 280 million triples using Virtuoso or up to 60 million triples using Jena with a MySQL back end without much trouble and still be able to run arbitrary SPARQL queries on a reasonable time. However, these were relatively small tests. The first one represented only 120 documents in a collection, where as the second one was only a subset of the Wikipedia link graph. Yes, there are ways to move beyond via proper striping and replication of the data, but that is not by default a key concern of such engines. Another sad note was that we had to drop Mulgara because we had a pretty hard time pushing it that far (most of the bugs we run into has been fixed since last year and they have started a push toward getting SPARQL in, so it may be time for revisiting it).

However, all the above approaches were not born out of a distributed environment. Lately, I have started looking for alternative large scale storage born from the distributed environment’s soup. After the Hadoop Summit/BCDSG 2008 trip I started looking into HBase (open source take to Google’s Big Table by the Hadoop community). Not a bad alternative if you can fit your application data needs into their structure. Since runs on Hadoop FS  you get all its benefits for free. But, after reading the above-mentioned interview, however, I just found myself intrigued about CouchDB and Tokyo Cabinet. I guess I better go and take a look at them

No related posts.

During my trip to the Hadoop Summit and the Big Data Computing Study Group I ran into the ZooKeeper project, a Yahoo Research project. In their own words:

ZooKeeper is a high available and reliable coordination system. Distributed applications use ZooKeeper to store and mediate updates key configuration information. ZooKeeper can be used for leader election, group membership, configuration maintenance, etc.

There is also an interesting introductory lesson here, and some recipes for the most common data structures (queues, priority queues, distributed locks, etc.) are also available here. Looks promising, and may make the coordination of the distributed execution of a Meandre flows easier. The other reason that pushed me to explore this directions was that one of the Hadoop Summit highlights was the adoption of  the ZooKeeper project by Hadoop itself.

Related posts:

1. Temporary storage for Meandre’s distributed flow execution
2. Embedded Jetty applications
3. Parallel and Distributed Computational Intelligence book is out for pre-order