Efficient storage for Python

Xavier — Tue, 01 Jul 2008 21:44:57 +0000

Did you ever run into the situation that your analysis/simulation data is too large to fit it in memory? Does the flat file format you use for storing your data sets become to big that renders it slow to a crawl? If you answered yes, you may want to give a spin to the HDF5 library. HDF5 file are not replacement for relational data bases. They are catered for storing complex data objects and a wide variety of metadata. It is also optimize for efficiency of storage and retrieval. The underlying library is written in C.

If you are a Python user, PyTables provides a very efficient wrapper for HDF5 files. It gives you access to all the HDF5 api, plus it is nicely integrated with NumPy and provides natural naming conventions. In another words, you can quickly store and retrieve your arrays/matrix to HDF5 files, giving you a very interesting persistence layer. For instance you can do a simple table scan by:

import tables;
 
h5f = tables.openFile("test.h5f","r")
for r in h5f.root.Table :
    pass
h5f.close()

The code above opens an H5DF file called “test.h5f” which contains a table at the root named “Table”–see the natural notation here. Then it just scans the table using the common Python syntax. You can also do grouped reads, read all into memory, filtered wheres, and the like.

Definitely worth to give it a spin if you have to much data for your available memory or flat file format. Also, I am wondering how fast this may fly on a SSD…

SVNKit or analyzing SVN content in Java

Xavier — Tue, 20 May 2008 14:47:52 +0000

The other day I was looking for a piece of software that could help me pull data out of SVN repository so I can do a little analysis on it. Browsing over the net, I run into several tools, but the one that finally caught my eye was SNVKit. The package does a lot of stuff, but for instance, if you are only interested on pulling information out, it makes your like quite easy. The code below is just a simple example of how you can use it to pull information out of a SVN repository.

import java.util.LinkedList;

import org.tmatesoft.svn.core.ISVNDirEntryHandler;
import org.tmatesoft.svn.core.SVNDirEntry;
import org.tmatesoft.svn.core.SVNException;
import org.tmatesoft.svn.core.SVNURL;
import org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryFactoryImpl;
import org.tmatesoft.svn.core.internal.wc.admin.SVNEntry;
import org.tmatesoft.svn.core.wc.SVNClientManager;
import org.tmatesoft.svn.core.wc.SVNLogClient;
import org.tmatesoft.svn.core.wc.SVNRevision;
import org.tmatesoft.svn.core.SVNNodeKind;

public class SVNReader {

    public static void main ( String [] sArgs ) throws SVNException {
          final LinkedList lstFiles = new LinkedList();

          SVNRepositoryFactoryImpl.setup();

          SVNClientManager clientManager = SVNClientManager.newInstance();

          SVNLogClient lc = clientManager.getLogClient();
          SVNURL svnUrl = SVNURL.parseURIDecoded(
                    "svn://some.server.com:3690/path/to/trunk"
                    );
          lc.doList(svnUrl, SVNRevision.HEAD,  SVNRevision.HEAD, false, true, new ISVNDirEntryHandler() {
                    public void handleDirEntry(SVNDirEntry svnEntry) throws SVNException {
                              if ( svnEntry.getKind()==SVNNodeKind.FILE) {
                                        lstFiles.add(svnEntry);
                              }
                    }
          });

          for ( SVNDirEntry svnEntry:lstFiles ) {
                    System.out.println(svnEntry);
          }
      }
}

Xavier Llorà » data processing

Efficient storage for Python

SVNKit or analyzing SVN content in Java