Posts Tagged performance
Gatling for rails app @ http://www.brug.be
Last week I had to review an hibernate powered application due to his poor performance and various instability issues.
In my post Don’t Get Caught Hibernating, I’ve assumed that jdbc connectivity and caching were properly configured.
Well in fact this is not always the case.
Don’t use Hibernate built-in connection pool
Hibernate’s own connection pooling algorithm is, however, quite rudimentary. It is intended to help you get started and is not intended for use in a production system, or even for performance testing. You should use a third party pool for best performance and stability. Just replace the hibernate.connection.pool_size property with connection pool specific settings. This will turn off Hibernate’s internal pool. For example, you might like to use c3p0.
Strangely hibernate log this at info level (not warning) when instantiating the session factory
log.info("Using Hibernate built-in connection pool (not for production use!)");
Instead of using this ‘naive’ implementation, you should configure the session factory to use one of the better javax.sql.DataSource implementation :
- C3P0: add the extra hibernate.c3p0.* properties and that’s it !
- commons-dbcp : perhaps the older implementation, need to adjust the dependencies depending the java version your are using.
- tomcat7-jdbc : simpler implementation than commons-dbcp and contains more features 😉
Note also that other implementations are available.
Don’t forget to
- tune the datasource pool size to fit your production needs
- choose and configure the validation mechanism for opened connections and various strategies testOnBorrow, testWhileIdle,…
- specify the isolation level : it will take the default which is sometimes too high like “repeatable read”
- specify also timeouts, max connection age
- enable preparedStatement cache
As you review your database connection pooling, it may be easy to also instrument it with jamon datasource.
You will gain visibility in your various jdbc accesses. This can also help to identify the ideal pool size (take a look at MaxActive and AvgActive)
Don’t use Hibernate default cache implementation
From hibernate documentation:
Hashtable (not intended for production use) org.hibernate.cache.HashtableCacheProvider memory
The HashtableCacheProvider isn’t a production ready implementation :
- no max size
- no invalidation (lru,…)
- no time to live, time-idle
and can be :
- considered as a memory leak,
- source of OutOfMemoryError,
- out of date data : caches not aware of changes made to the persistent store by another application
As the documentation states, various implementation exists.
From my experience Ehcache is quite easy to setup and ready to scale (cluster).
Don’t forget to disable its phone home mechanism
Application performance can’t be summarized to an average and a standard deviation. Most performance issues aren’t so clear… jamonapi can help identifying your bottlenecks
Same average, same standard deviation, not same reality
Most application performance solutions are collecting performance data and only keep average and standard deviation (stddev). But application performance rarely follows normal distribution. Two samples with the same average and stddev doesn’t imply happy users.
Let’s suppose you have a first release of your app and see an histogram like this one
Most users are happy with a average of 1.9 seconds and standard deviation of 0.6 seconds
Let’s introduce our version 2.0 of the application. Our monitoring still shows an average of 1.9 seconds and standard deviation of 0.6 seconds.
But you receive a lot of feedback : 50% of your end-users are complaining about bad performance… what’s going on ?
on the left the happy users… and on the right your unhappy end-users !
Hopefully you can easily instrument your application with jamon and discover this distribution.
Jamon is to System.currentTimeMillis() what log4j is to System.out.println()
Jamon collects “stop/start” events and aggregates the logarithmic distribution of these events/monitor.
- 0-10ms. 11-20ms. 21-40ms. 41-80ms. 81-160ms. 161-320ms. 321-640ms.
- 641-1280ms. 1281-2560ms. 2561-5120ms.
- 5121-10240ms. 10241-20480ms. >20480ms.
It also keeps for each monitor additional informations like :
- Avg ms.
- Total ms.
- Std Dev ms.
- Min ms.
- Max ms.
- Avg Active
- Max Active
- First access
- Last access
The active, avg active and max active shows the degree of concurrency of your monitor.
Jamon feature and advantages :
- easy installation : drop 3 jars, a bunch of jsp that’s it
- production ready with low overhead
- a servlet filter to monitor url time response by just modifying the web.xml
- datasource wrapper to gather sql statistics (just an extra bean in your application)
- spring integration via aop JamonPerformanceMonitorInterceptor
- for non web application like batch processing or junit, you can write the jamon stats to a csv via a jmx console or a the end of the process.
Real life usage
0. mesure don’t guess
1. enable it in production
2. sort by total time
3. detect what can be improved in these use case : db indexes, hibernate batch-size,…
4. fix and rollout in production
5. goto 1 😉
Alternative to jamon
- codahale metrics looks really promising with implementation for Gauges, Ratio Gauges, Percent Counters, Histogram, Meters,… integration with Guice, Jetty, Log4j, Apache HttpClient, Ehcache, Logback, Spring, Ganglia, Graphite
- javasimon : quantiles, hierarchichal, nanoseconds,… but jdk 1.6 and I’m stuck with websphere 😦
- datasource-proxysql no distribution but can summarize sql interactions per http request.But can be linked with other librairies