Onyx Development Tools has created a series of performance benchmark tests using embedded Java databases including H2, Derby, Onyx Database, and HSQL. The purpose of these tests is to compare the performance of various JPA ORMs vs. the Onyx Database platform.
Other than the Onyx Database environment, all embedded databases use the JPA library and their respective drivers. Onyx is a NoSQL database and an ORM (Object Relational Map) therefore it does not require a JPA implementation. This test only submits embedded Java databases. Further NoSQL benchmark comparisons and Graph Database performance reviews will be posted on a later date.
The data model has been set up to represent a production-like application. In this case, the data model is used to track NFL Player statistics. It has been pulled from one of our examples. The data model contains various relationships, indexes, and entities with attributes.
If you have a comment regarding our findings, or suggestions for benchmark test methodology, we would like to hear from you! Contact Onyx Development Tools, LLC.
The benchmarks were designed to test the speed and scalability of embedded databases. In each scenario, the database was pushed beyond a basic use load and used to determine how each individual database performs when it is pushed beyond a simple storage and throughput. In each test, Onyx Database performed better than each of the ORM database implementations. Based on our results, listed below are the databases ordered from best to worst.
|Single Threaded Insertion||Multi Threaded Insertion||Delete Transaction||Update Transaction||Random Transactions *|
|Onyx Database||7,981 operations/second Finished in 2.506 seconds||8,357 operations/second Finished in 5.983 seconds||33,501 operations/second Finished in 0.597 seconds||13,514 operations/second Completed in 1.480 seconds||9,080 operations/second Finished in 3.304 seconds|
|H2||82 operations/second Finished in 245.011 seconds||130 operations/second Finished in 385.436 seconds||614 operations/second Finished in 32.582 seconds||444 operations/second Finished in 45.085 seconds||721 operations/second Finished in 38.117 seconds|
|Derby||70 operations/second Finished in 284.827 seconds||150 operations/second Finished in 333.791 seconds||652 operations/second Finished in 30.665 seconds||423 operations/second Finished in 47.242 seconds||721 operations/second Finished in 41.605 seconds|
|HSQL||29 operations/second Finished in 695.781 seconds||22 operations/second Finished in 2,232.081 seconds||105 operations/second Finished in 190.529 seconds||135 operations/second Finished in 147.971 seconds||132 operations/second Finished in 227.699 seconds|
The tests were designed to measure insertion, read, delete, update, and multi-thread concurrency throughput. In addition to basic transactions, included is a test designed to simulate a realistic load by selecting a random distribution of events. The test was run on a 2015 MacBook Pro with El Capitan. The MacBook Pro consisted of 16 GB 1600 MHz DDR3, 2.5 GHz Intel Core i7, and SSD.
Each database benchmark test contains a reasonable ramp up followed by a fixed amount of iterations of given task(s). In most cases, the test was configured to have a maximum of 8 concurrent threads. In addition to that the JVM arguments included an -Xmx2048 argument. The performance tests were run 3 times and the average value was recorded for each scenario.