JTA 1.1 Benchmark Testing Do's and Don'ts
I've decided to write a full article on JTA 1.1 implementation and the TransactionManager, that incudes the 5 common programming bottlenecks in a JTA 1.1 TransactionManager implementation. Once finished with many confusing posts that seem to make no sense to common man. I will put it all together into one useful guide for J2EE Application Server/JTA 1.1 implementors.
I've read several articles on the internet where a person doing benchmark testing on multiple JTA 1.1 MQ Messaging implementations for XA, and they perform load testing with different message sizes. This is a huge problem since the testers are not actually testing the TransactionManager performance, or the JTA 1.1 implementation. A person may question how can we know the implementor did not somehow do something to tie the JMS to the TransactionManager for performance improvement or some other means, and that is not possible if the implementation strictly follows the JTA 1.1 specification. In JTA 1.1 implementation performance for XA Transactions should only be performance/load tested without passing in an actual XAResource object which is the Object that connects to a Database or JMS implementation. One exception does exist to passing in an XAResource, and that is testing a transaction with multiple XAResources, as the loading and unloading of the XAResources, and generating their branch qualifier for the Xid Object will take time, however, the Xid portion can likely be ignored due to the performance of that Object being tested during the Global Unique Identifier phase of the Transaction. Testing multiple XA Resources will require multiple XAResource Dummy objects which identify themseves differently via their hashcode, and equals methods (need to verify this is the correct implementation Object for these methods to be tested, It may be Transaction, in which case I need to modify this. Will modify anyways when I confirm this).
Just like XAResource the Synchronization Object implementations are independent of the JTA 1.1 implementation and many are proprietary or only function properly on a particular server. Using a non-dummy Synchronization Object for testing will only interfere with accurate test results of the JTA 1.1 TransactionManager implementation, and therefore should be avoided during testing. It is important to keep in mind that multiple Synchronization Objects can be registered so testing the speed of loading/unloading Synchronization methods with a well implemented Dummy Synchronization shall be sufficient for performance testing execution of these Objects. Do not test non-dummy methods as actual implementations of the Synchronization Objects would give test results that are interfered with by the Synchronization Objects.
Lastly, a few test scenarios exists. The TransactionManager in the JTA 1.1 specification is implemented to run only one Transaction at a time per Thread. All calls to the TransactionManager require instantiation of the TransactionManager Object followed by a call to begin() to start the Transaction and commit() to end the Transaction. A basic load test will exercise begin and commit synchronously like the following for each Thread:
<code>
TransactionManager tm = TransactionManager.getInstance();
System.out.println("Starting TransactionManager Load Test...");
long startTime = System.nanoTime();
for(int j = 0;j<20000000;j++) { tm.begin(); tm.commit();
}
long endTime = System.nanoTime();
System.out.println("TransactionManager Load Test completed...");
System.out.println("The TransactionManager performed 20000000 transactions in " + endTime-startTime + " nanoseconds, averaging " + ((double)(endTime-startTime)) / 1E9 + " transactions per second.");
// shutdown the TransactionManager (this could be any code to stop the TransactionManager)
tm.shutdown();
</code>
Copyright 2011 Aaron Schiffman. All rights reservered.