Martlet: Workflow and Functional Programming

One of the many fun things about programming languages is that you never know when and where a new one -- or a new kind of one -- will pop up next.

Take Martlet, for instance. Marlet is a workflow language for analyzing large datasets whose distribution is continually changing across a number of widely dispersed servers. Designed by Oxford University's Daniel Goodman, Martlet implements a functional programming model that lets you write parallel programs and analyze distributed data without having to be aware of the details of the parallelization. Martlet abstracts the parallelization of the computation and the splitting of the data via functional programming constructs that let programs be written as abstract descriptions that can be adjusted to match the data set and available resources automatically at runtime. In short. this means that inexperienced users can take advantage of the power of distributed computing resources, while reducing the work load on experienced distributed programmers. To allow the evaluation of the language and programming model, a supporting middleware has been constructed using web services supported by Apache Axis and Jakarta Tomcat.

To gauge its effectiveness, Marlet is being tested with the ClimatePrediction.net project, a distributed computing project similar to SETI@home. With CliminatePrediction.net, users download a model of the Earth’s climate and run it for approximately 50 model years , then return results read from their model to one of multiple servers. The output of these models creates a data set that is distributed across the servers.

"Existing workflow languages are not up to the task because they implement a style of programming where the number of data inputs and the paths of data flow through the workflow are set when the workflow is submitted. This makes them unable to cope with subsequent changes to the dataset," says Goodman. Functional programming constructs let the workflow adjust to the requirements of the data at runtime and changes to the way in which a dataset is split can be accommodated dynamically, so removing the need for users to keep adjusting their workflows.

Martlet has potential for use in many science applications which distribute data between servers in a similar way to climateprediction.net. Its development also suggests that there could be other powerful new algorithms awaiting discovery once people start to think in terms of this alternative programming model. "Daniel's work has shown how work on core computer science can be used to meet the exciting challenges generated by e-Science applications. He has demonstrated how taking a different approach to organising the way in which tasks are executed can produce scientific results much more quickly," says Professor Paul Watson of Newcastle University.

Posted by Jon Erickson at 11:49 AM  Permalink