May 02, 2007
Multicore and Java: Yes, They Can Work Together
Jonathan Erickson
A Java development framework to unleash the power of multicore processing
DDJ: Joining us today is Mike Hoskins, CTO of Pervasive Software. Mike, do Java developers have special needs when it comes to developing concurrent software for multicore processors?
MH: Absolutely! Combine exploding data volumes, shrinking time windows and the need for continuously better data analytics and the Java programmer is already in a crunch. Throw in multicore hardware and the expectations are incredible. With all of those cores, software built today must scale to take advantage of them. The java.util.concurrent library provides an excellent base to build multi-threaded applications with, but the learning curve is steep. What Java programmers need is a new way to create scalable applications without being bogged down by multi-threading programming details.
DDJ: Can you give us brief discussion on and example of a "hyper-parallel data-intensive application" that your new Pervasive DataRush tool was designed to build?
MH: We are currently working on a proof of concept with a health care data processing client. The client's application consumes many gigabytes of client data a day. The data is validated, transformed,
aggregated and then loaded into database for query processing. The validation is complex as the data is hierarchal and has many relationships and business rules that must be verified. Traditionally this is done on a transaction by transaction basis and as a result is very time consuming. Currently these operations are measured in days rather than minutes or even hours. The data comes in many forms, so breadth of connectivity is important.
The Pervasive DataRush framework is a great fit for this client for several reasons. First, the volume of data is high and they are experiencing trouble keeping up with demand. In short, they have more data to process than they can handle using their current architecture. The ability of Pervasive DataRush to consume data volumes beyond traditional programming models and scale accordingly is essential. Next is the complexity of the problem. The amount of data intensive operations involved in validating the business rules as well as the data format rules is extremely large, but it is repeatable, making these operations great candidates for parallel work. Next, the customer has multicore machines in house, but are not generally taking advantage of them. This non-efficient use of hardware is a recurring theme among most customers we speak with. Finally this customer uses a Java environment for its solutions, Pervasive DataRush is a natural fit for them. All of these factors combined make a great case for using an application development framework such as Pervasive DataRush.
DDJ: From what I can tell, you have -- or will -- be using Eclipse tools with Pervasive DataRush. Did you have to make any unique extensions of plug-ins specifically for multicore?
MH: We are in the process of building a first-rate editor within Eclipse for Pervasive DataRush. One of the founding concepts of Pervasive DataRush is to enable the creation of easily scalable, data processing applications without requiring in-depth knowledge of multi-threaded programming. With the graphical editor we are building in the Eclipse framework, a Java programmer will be able to drag-and-drop DataRush components onto a canvas and link them together with ease. Pervasive DataRush is built on the foundation of Java technology, so we plan on taking full advantage of the powerful Java tools within
Eclipse, and extending where needed to support Pervasive DataRush. This functionality will be delivered in standard Eclipse plug-ins, easily downloaded and installed in an existing Eclipse instance.
DDJ: What is the biggest challenge from your perspective that developers in general face when writing software for multicore systems?
MH: The biggest challenge faced by the Java programmer is our current way of doing things. We have lived in a single-threaded world for way too long, relying on processor speed ups to come to our rescue. Those days appear to be over. Processor vendors have turned to multicore technology to continue the performance march. Software needs to catch up. But, the average programmer continues to be faced with shorter project time lines and increased pressure to produce high-quality software. There is no time in their schedule to take a few months to master multi-threaded programming skills.
DDJ: Is there a web site readers can go to for more information on these topics?
MH: Yes. The Pervasive DataRush site is a great place to start. You can find blogs, articles, and links to other sites covering this topic. Pervasive DataRush is now in beta release and available there as a free download for Java developers to play with. The Wikipedia site also has several topics on dataflow and dataflow programming that give good background material on the concepts behind Pervasive DataRush.
