We've all heard about the simplicity and power of the EJB 3.0 specification. And because this has proven to be true, we can't help but think that performance must be rather poor. After all, all that simplicity must come at a price.

With this in mind, we set out to test EJB 3.0's performance using Oracle's implementation of the specification. Although the implementation we used is a developer preview, where the focus is typically on product stability instead of performance, our expectations are that the performance will be below or in the best case the same as previous versions of the EJB specification.

We thought that the best way to test was to do a few things that developers typically do with EJB 2.1 and then try out the equivalent with EJB 3.0. This would let us compare performance rather than deal with raw numbers that probably wouldn't mean much. We tested the Data Transfer Object (DTO) design pattern, the Session façade design pattern, and the use of the Container Managed Relationships (CMR) functionality of the entity beans.

The application we used, as well as the test harness and the methodology, is described in J2EE Performance Testing by Peter Zadrozny (Expert Press, 2002). The harness is a simple dispatcher servlet that executes each discrete test case based on the JazzCat application, a catalog of jazz recordings. The database schema includes tables for bands, musicians, instruments, tracks, and albums. The application also handles the storage and retrieval sessions recording and takes of a track.

Our test environment was based on Oracle Application Server EJB 3.0 Preview (Developer Preview 3), and Sun Java HotSpot Client VM (build 1.5.0_03-b07, mixed mode, sharing). We used Oracle Database 10g Enterprise Edition Release 10.1.0.2.0. We used the default settings on all of the software.

The users were simulated with The Grinder 3.0 Beta 25 (http://grinder.sf.net), with a sample size of 5,000 milliseconds. Each test run lasted eight minutes; we ignored the first three minutes to allow the test to stabilize. Each simulated user ran a test script that called the corresponding test case 10 times. The test scripts were executed continuously in a sequential fashion for the duration of the test run. Two things are worth noting: there was a separate HTTP session for each execution of the test scripts, and there was no sleep time between each call to the test case. The latter was done to create a highly stressful situation so we could see how EJB 3.0 behaved when pushed to its limits.

To get a complete picture of the performance, we used two key indicators: Aggregate Average Response Time (AART), to reflect the end user perspective, and Total Transactional Rate (TTR) or throughput, to reflect the load on the systems involved.

We used three Dell PowerEdge 2850 computers with dual Intel Xeon (HT) at 3.4GHz with 4GB of memory, running Microsoft Windows Server 2003, Standard Edition. We used one of the computers to generate the load with The Grinder, another to run Oracle Containers for J2EE (OC4J), and the last to run the database. All three were connected to a switched network in which the only traffic was generated by the tests themselves.

Data Transfer Object
Using JazzCat application to test the DTO design pattern, we created a servlet that lists all the albums in our test database via an entity bean, where each row in the listing shows the field values of the album entity. We started the EJB 2.1 test by programming this functionality without using DTO. In this case, which we called "DTOOff", our test servlet would retrieve a list of all the album entities, and then get the individual field values for each entity via the entity's accessor methods.

Using the DTO design pattern, our test servlet, called "DTOOn", makes a single method call (getData). The entity bean constructs a corresponding DTO, loads it with the entity's field values, and returns the object to the servlet. Now the servlet can access the fields on the local object.

In EJB 3, the entity beans have been replaced by Plain Old Java Objects (POJOs) and all the traditional CRUD and query operations are performed with the Entity Manager (EM). Queries can be defined as standalone named queries, which are predefined in the bean class, or dynamic queries that can be constructed using the query method.

In the JazzCat application, the dispatch servlet of the test harness calls the corresponding test servlet, which in turn looks up the entity manager.

The entity manager then executes the named query that was defined for the Albums class. Here's the definition of this class and the named query:

@Entity
@Table(name="ALBUMS")
@NamedQuery(name="findAll",queryString="Select object(a) from Albums a")
public class Albums {
//......
}

Once the named query returns the list of all the albums, it's then printed out as an HTML table using the getter methods of Albums POJO.

We ran the tests with 15, 25, 50, 100, 150, 200, and 250 simultaneous users and we found that in both cases, response time and throughput, the difference between EJB 2.1 using DTO and EJB 3.0 was within 4%. This is within the margin of error, so for all practical purposes the performance is similar. To give you an idea of the actual performance, with 250 simultaneous users the average aggregate response time was 1,100 milliseconds and the throughput was an average 225 requests per second. This is very impressive, especially since there was no sleep time in the test scripts.

The DTOOff test case produced an AART 28% higher than DTOOn and EJB 3, and a TTR that was 19% lower. This is because each call to an accessor method on the entity bean is a transaction by itself, whereas using DTO and EJB 3, there is only one transaction. Remember to demarcate your transactions correctly to get better performance.

It's interesting to note the throughput curves for each case (see Figure 1). Remember that the TTR or throughput is a measure of system capacity as a whole (application, JVM, database, OS, hardware). The curves start by going upward, which shows the system is handling the capacity as the requests increase. Then they reach a point of stability where the system has reached its limit and the response time starts to increase dramatically. Finally, when the curves begin to fall, this is where the system can't handle any more load and things start going downhill. In this chart you can see that in all cases, the peak has been reached at 50 simultaneous users; however, on the DTOOff case performance goes downhill at 150 users, whereas EJB 3 and DTOOn hold on longer before going down. Interestingly, EJB 3 holds on a little longer by presenting 12% better throughput than DTOOn with 250 users. This is likewise for the response time, where EJB 3 has 13% better number than DTOOn with 250 users. You can say that under heavy loads, EJB 3.0 tends to perform better than EJB 2.1, at least in this test case.