This summer I decided to go on a quest to get me out of the woods. I do not mind relying on the JVM and the large collection of libraries available, but I would also like to get my sanity back. Yes, I tested some of the usual suspects for the JVM (Jython, JRuby, Clojure, and Groovy) but not quite what I wanted. For instance, I wrote most of the Meandre infrastructure services using Jython (much more concise than Java), but still not quite happy to jump on that boat. Clojure is also interesting (functional programming) but it would be hard to justify for the group to move into it since not everybody may feel comfortable with a pure functional language. I also toyed with some not-so-usual ones like Erlang and Haskell, but again, I ended up with no real argument that could justify such a decision.

So, as I started doing back in 2007, I went back to my original idea of using Scala and its mixed object-oriented- and functional-programming- paradigm. To test it seriously, I started developing the distributed execution engine for Meandre in Scala using its Earlang-inspired actors. And, boom, suddenly I found myself spending more time thinking that writing/debugging threaded/networking code . Yes, I regret my 2007 decision instead of running with my original intuition, but better late than never. With a working seed of the distributed engine working and tested (did I mention that scalacheck and specs are really powerful tools for behavior driven development?), I finally decided to start gravitating the Meandre infrastructure development effort from Java to Scala—did I mention that Scala is Martin Odersky’s child? Yes, such a decision has some impact on my colleagues, but I envision that the benefits will eventually weight out the initial resistance and step learning curve. At least, the last two group meetings nobody jumped off the window while presenting the key elements of Scala, and demonstrating how concise and elegant it made the first working seed of the distributed execution engine . We even got in discussions about the benefits of using Scala if it delivered everything I showed. I am lucky to work with such smart guys. If you want to take a peek at the distributed execution engine (a.k.a. Snowfield) at SEASR’s Fisheye.

Oh, one last thing. Are you using Atlassian’s Fisheye? Do you want syntax highlighting for Scala? I tweaked the Java definitions to make it highlight Scala code. Remember to drop the scala.def file on $FISHEYE_HOME/syntax directory add an entry on the filename.map to make it highlight anything with extension .scala.

Related posts:

1. Meandre 2.0 Alpha Preview = Scala + MongoDB
2. Meandre: Semantic-Driven Data-Intensive Flow Engine
3. Fast REST API prototyping with Crochet and Scala

• Transaction ready
• Light weight implementation
• Efficient write and read to minimize the contention on ports

Also, it is important to keep in mind that in a distributed execution scenario, each node requires to have its one separated and standalone storage system. Thus, it is also important to minimize the overhead of installation and maintenance of such storage subsystem. There are several alternatives available ranging from traditional relational data base systems to home-brewed solutions. Relational data base systems provide a distributed, reliable, stable, and well tested environment, but they may tend to require a quite involved installation and maintenance. Also, tuning those systems to optimize performance may required quite an involved monitoring and tweaking. On the other hand, home-brewed solutions can be optimized for performance by dropping non required functionality and focussing on writing and reading performance. However, such solutions tend to be bug prone and tend to become time consuming, not to mention that proving transaction correctness can be quite involved.

Fortunately there is a middle ground where efficient and stable transaction aware solutions are available. They may not provide SQL interfaces, but they still provide transaction boundaries. Also, since they are oriented to maximize performance, they can provide better throughput and operation latency than having to traverse the SQL stack. Examples of such storage systems can be found under the areas of key-value stores and column stores. Several options were considered while writing these line, but key-value stores were the ones that better matches the three requirements described above. Several options were informally tested, including solutions like HDF and Berkely DB, however the best performing by far under similar stress test conditions as the sketched temporary storage subsystem was Tokyo Cabinet. I already introduced and tested Tokyo Cabinet more than a year ago, but this time I was going to give it a stress test to basically convince myself that that was what I wanted to use for as temporary storage of the distributed flow execution.

The experiment

Tokyo cabinet is a collection of storage utilities including, among other facilities, key-value stores implemented as hash files or B-trees and flexible column stores. To illustrate the performance and throughput you can achieve. To implement multiple queues on a single casket (Tokyo Cabinet file containing the data store) B-trees with duplicated keys can help achieving such goal. The duplicated keys are the queue names, and the values are the UUIDs of the objects being store. Objects are also stored in the same B-tree by using the UIUD as a key and the value become the payload to store (usually an array of bytes).

Previously, I have been heavily using Python bindings to test Tokyo Cabinet, but this time I went down the Java route (since the Meandre infrastructure is written on Java). The Java bindings are basically build around JNI and statically link to the C version of Tokyo Cabinet library, giving away the best of both world. To measure how fast can I write data out of a port into the local storage in a transactional mode, I used the following piece of code.

	public static void main ( String args [] ) {
		int MAX = 10000000;
		int inc = 10;
		int cnt = 0;
		float fa [] = new float[8];
		int reps = 10;
 
		for ( int i=1 ; i<=MAX ; i*=inc  ) {
			//System.out.println("Size: "+i);
			for ( int j=0 ; j<reps ; j++ ) {	
				//System.out.println("\tRepetition: "+j);
 
				// open the database
				BDB bdb = new BDB();
 
				if(!bdb.open(TEST_CASKET_TCB, BDB.OWRITER | BDB.OCREAT | BDB.OTSYNC )){
					int ecode = bdb.ecode();
					fail("open error: " + bdb.errmsg(ecode));
				}
 
				// Add a bunch of duplicates
				long start = System.currentTimeMillis();
				bdb.tranbegin();
				for ( int k=0; k<i; k++ ) {
					String uuid = UUID.randomUUID().toString();
					bdb.putdup(QUEUE_KEY, uuid);
					bdb.putdup(uuid.getBytes(), uuid.getBytes());	
				}
				bdb.trancommit();
				fa[cnt] += System.currentTimeMillis()-start;
 
				// Clean up
				bdb.close();
				new File(TEST_CASKET_TCB).delete();
			}
			fa[cnt] /= reps;
			System.out.println(""+i+"\t"+fa[cnt]+"\t"+(fa[cnt]/i));
			cnt++;
		}
	}

The idea is very simple. Just go and star storing 1, 10, 100, 1000, 10000, 1000000, and 10000000 pieces of data at once in a transaction. Measure the time. For each data number repeat the operation 10 times and average the time trying to palliate the fact that the experiment was run on a laptop running all sorts of other concurrent applications. Plot the results to illustrate:

1. time required to insert one piece of data as a function of the number of data involve in the transaction
2. number of pieces of data wrote per second as a function of the number of data involve in the transaction

The idea is to expose the behavior of Tokyo Cabinet as more data is involved in a transaction to check if degradation happens as the volume increase. This is an important issue, since data intensive flows can generate large volumes of data per firing event.

The results

Results are displayed on the figures below.

The first important element to highlight is that the time to insert one data element does not degrade as the volume increase. Actually, it is quite interesting that Tokyo Cabinet feels more comfortable as the volume per transaction grows. The throughput results are also interesting, since it shows that it is able to sustain transfers of around 40K data units per second, and that the only bottleneck is the disk cache management and bandwidth to the disk itself—which gets saturated after pushing more than 10K pieces of data.

The lessons learned

Tokyo Cabinet is a excellent candidate to support the temporary transactional storage required in a distributed execution of a Meandre flow. Other alternatives like MySQL, embedded Apache Derby, the Java edition of Berkeley DB, SQLite JDBC could not get even get close to such performance falling at least one order of magnitude behind.

Related posts:

1. Easy, reliable, and flexible storage for Python
2. ZooKeeper and distributed applications
3. Meandre: Semantic-Driven Data-Intensive Flow Engine

Slides

Abstract

We are living in the peta-byte era.We have larger and larger data to analyze, process and transform into useful answers for the domain experts. Robust data mining tools, able to cope with petascale volumes and/or high dimensionality producing human-understandable solutions are key on several domain areas. Genetics-based machine learning (GBML) techniques are perfect candidates for this task, among others, due to the recent advances in representations, learning paradigms, and theoretical modeling. If evolutionary learning techniques aspire to be a relevant player in this context, they need to have the capacity of processing these vast amounts of data and they need to process this data within reasonable time. Moreover, massive computation cycles are getting cheaper and cheaper every day, allowing researchers to have access to unprecedented parallelization degrees. Several topics are interlaced in these two requirements: (1) having the proper learning paradigms and knowledge representations, (2) understanding them and knowing when are they suitable for the problem at hand, (3) using efficiency enhancement techniques, and (4) transforming and visualizing the produced solutions to give back as much insight as possible to the domain experts are few of them.

This tutorial will try to answer this question, following a roadmap that starts with the questions of what large means, and why large is a challenge for GBML methods. Afterwards, we will discuss different facets in which we can overcome this challenge: Efficiency enhancement techniques, representations able to cope with large dimensionality spaces, scalability of learning paradigms. We will also review a topic interlaced with all of them: how can we model the scalability of the components of our GBML systems to better engineer them to get the best performance out of them for large datasets. The roadmap continues with examples of real applications of GBML systems and finishes with an analysis of further directions.

Related posts:

1. Observer-Invariant Histopathology using Genetics-Based Machine Learning
2. Deadline extended for special issue on Metaheuristics for Large Scale Data Mining
3. [BDCSG2008] Algorithmic Perspectives on Large-Scale Social Network Data (Jon Kleinberg)

Last Thursday I was talking to Abhishek Verma, and he pointed out Google’s Protol Buffer project—Google’s take data interchange formats. Not a new idea—for instance Corba’s IDL has been around for a long time—but what caught my eye was their claims about: (1) efficiency, and (2) multiple language bindings. I was contemplating using XStream for Meandre distributed flow execution needs, but the XML heavy weight made me quite reluctant to walk down that path.  The Java native serialization is not a bad choice in terms of efficiency, but does not provide friendly mechanics for modifying data formats without rendering already serialized objects useless, neither a transparent mechanism to allow bindings for other languages/platforms. So the Google’s Protol Buffer seemed an option worth trying. So there I went, and I prepare a simple comparison between the tree: (1) Java serialization, (2) Google’s Protol Buffer, and (3) XStream. Yes, you may guess the outcome, but I was more interested on getting my hands dirty, see how Google’s Protol Buffer perform, and how much overhead for the developer it required.

The experiment

Before getting into the description, this experiment does not try to be an exhaustive performance evaluation, just an afternoon diversion. Having said so, the experiment measured the serialization/deserialization time and space used for a simple data structure containing just one array of integers and one array of strings. All the integers were initialized to zero, and the strings to “Dummy text”. To allow measuring the time required to serialize this simple object, the number of integers and strings were increased incrementally. The code below illustrates the implementation of the Java native serialization measures.

package org.meandre.tools.serialization.xstream;
 
public class TargetObject {
 
       public String [] sa;
       public int [] ia;
 
       public TargetObject ( int iStringElements, int iIntegerElements ) {
             sa = new String[iStringElements];
             for ( int i=0 ; i<iStringElements ; i++ )
                  sa[i] = "Dummy text";
             ia = new int[iIntegerElements];
       }
}

The experiment consisted on generating objects like the above containing from 100 to 10,000 elements by increments of 100. Each object was serialized 50 times, measuring the average serialization time and the space required (in bytes) per object generated. Below you may have the sample code I used to measure native java serialization/deserialization times.

package org.meandre.tools.serialization.java;
 
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
 
import org.junit.Test;
 
public class JavaSerializationTest {
 
       @Test
       public void testJavaSerialization () 
       throws IOException {
             final int MAX_SIZE = 10000;
             final int REP = 50;
             final int INC = 100;
 
             System.out.println("Java serialization times");
             for ( int i=INC ; i<=MAX_SIZE ; i+=INC ) {
                  TargetObjectSerializable tos = new TargetObjectSerializable(i,i);
                  long lAccTime = 0;
                  long lSize = 0;
                  long lTmp;
                  ByteArrayOutputStream baos;
                  ObjectOutputStream out;
                  for ( int j=0 ; j<REP ; j++ ) {
                      baos = new ByteArrayOutputStream();
                      out = new ObjectOutputStream(baos);
                      lTmp = System.currentTimeMillis();
                      out.writeObject(tos);
                      lTmp -= System.currentTimeMillis();
                      out.close();
                      lAccTime -= lTmp;
                      lSize = baos.size(); 
                  }
                  System.out.println(""+i+"\t"+(((double)lAccTime)/REP)+"\t"+lSize);
             }
       }
 
 
       @Test
       public void testJavaDeserialization () 
       throws IOException, ClassNotFoundException {
             final int MAX_SIZE = 10000;
             final int REP = 50;
             final int INC = 100;
 
             System.out.println("Java deserialization times");
             for ( int i=INC ; i<=MAX_SIZE ; i+=INC ) {
                  TargetObjectSerializable tos = new TargetObjectSerializable(i,i);
                  ByteArrayOutputStream baos = new ByteArrayOutputStream();
                  ObjectOutputStream out = new ObjectOutputStream(baos);
                  out.writeObject(tos);
                  out.close();
                  ByteArrayInputStream bais;
                  ObjectInputStream ois;
                  long lAccTime = 0;
                  long lTmp;
                  for ( int j=0 ; j<REP ; j++ ) {
                      bais = new ByteArrayInputStream(baos.toByteArray());
                      ois = new ObjectInputStream(bais);
                      lTmp = System.currentTimeMillis();
                      ois.readObject();
                      lTmp -= System.currentTimeMillis();
                      lAccTime -= lTmp;
                  }
                  System.out.println(""+i+"\t"+(((double)lAccTime)/REP));
             }
       }
}

Equivalent versions of the code shown above were used to measure Google’s Protol Buffer and XStream. If you are interested on seeing the full code you can download it as it is—no guarantees provided. Also, for completion of the experiment code, you can find below the proto file use for testing the Java implementation of Google’s Protol Buffer.

package test;
 
option java_package = "org.meandre.tools.serialization.proto";
option java_outer_classname = "TargetObjectProtoOuter";
 
message TargetObjectProto { 
  repeated int32 ia = 1; 
  repeated string sa = 2;
}

In order to run the experiment, besides Google’s Protol Buffer and XStream libraries, you will also need JUnit.

The results

The experiments were run on an first generation MacBook Pro using Apple’s Java 1.5 virtual machine with 2Gb of RAM. The figure below illustrated the different memory requirements for each of the the three serialization methods compared. Figures and data processing was done using R.

Figures show the already intuited bloated size of XML-based XStream serialization, up to 6 time larger than the original data being serialized. On the other hand, the Java native serialization provides a minimal increase on the serialized equivalent. Google’s Protocol Buffer presents a slightly larger requirement than the native Java serialization, but never doubled the original size. Moreover, it does not exhibit the constant initial payload overhead displayed by both XStream and the Java native serialization. The next question was how costly was the serialization process. Figures below show the amount of time required to serialize an object.

The Java native serialization was, as expected the fastest, however Google’s Protocol Buffer took only, on average, four times the more time than the Java native version. However, that is peanuts when compared to the fifty times slower XStream version. Deserialization times of the encoded object presents the same trends as the serialization, as the figures below show.

It is also interesting to note that serialization—as the figures below show—is faster than deserialization (as common sense would have suggested). However, it is interesting to note that Google’s Protocol Buffer is the method where these difference is more pronounced.

The lessons learned

As I said, this is far from being an exhaustive or even representative example, but just one afternoon exploration. However, the results show interesting trends. Yes, XStream could also be tweaked to make the searialized XML leaner, and even would—with the proper tinkering—make possible deserialize the object on a different platform/language, but at an enormous cost—both in size and time. The Java native serialization is by far the fastest and the most size efficient, but is made from and for Java. Also, changes on the serialized classes—imagine wanting to add or remove a field—may render the serialize objects unreadable. Google Protocol Buffers on the other hand delivers the best of both scenarios: (1) the ability to serialize/deserialize objects in a compact and relatively fast manner, and (2) allows the serialization/deserialization to happen between different languages and platforms. For these reasons, it seems to be a very interesting option to keep exploring, if you need both.

Related posts:

1. SVNKit or analyzing SVN content in Java
2. Temporary storage for Meandre’s distributed flow execution
3. R and Java

• Standardization of web services and their response information. Also added new services to help manage users, repositories, and jobs.
• The infrastructure is now able to provide a single system image cluster of Meandre servers. 
• It also provides a new light weight HTML interface for easy administration.
• Location addition has been speeded up minimizing the network usage. 
• Component contexts are locally cached, drastically reducing the preparation time of a flow before executing.

Also, all the reported bugs reported so far has been fixed. Hopefully, the release will be pushed out this coming week. Stay tuned . Also, a few documentation for developers interested in the infrastructure can be find on the development tools Confluence wiki.

Related posts:

1. Meandre Infrastructure 1.4 RC1 tagged
2. Meandre 1.4.0 released, 1.4.1 coming short after
3. Meandre: Semantic-Driven Data-Intensive Flow Engine

Related posts:

1. ICEIS 2008: Final sprint and Ricardo Baeza-Yates
2. ICEIS 2008: Blogging from Barcelona (Friday Morning)
3. Summary of BDCSG2008 blogging

During my trip to the Hadoop Summit and the Big Data Computing Study Group I ran into the ZooKeeper project, a Yahoo Research project. In their own words:

ZooKeeper is a high available and reliable coordination system. Distributed applications use ZooKeeper to store and mediate updates key configuration information. ZooKeeper can be used for leader election, group membership, configuration maintenance, etc.

There is also an interesting introductory lesson here, and some recipes for the most common data structures (queues, priority queues, distributed locks, etc.) are also available here. Looks promising, and may make the coordination of the distributed execution of a Meandre flows easier. The other reason that pushed me to explore this directions was that one of the Hadoop Summit highlights was the adoption of  the ZooKeeper project by Hadoop itself.

Related posts:

1. Temporary storage for Meandre’s distributed flow execution
2. Embedded Jetty applications
3. Meandre: Semantic-Driven Data-Intensive Flow Engine

Related posts:

1. Meandre: Semantic-Driven Data-Intensive Flows in the Clouds
2. Data-Intensive Computing for Competent Genetic Algorithms: A Pilot Study using Meandre
3. Meandre 1.4.0 final release candidate tagged