Performance : when average is not enough ?

Application performance can’t be summarized to an average and a standard deviation. Most performance issues aren’t so clear… jamonapi can help identifying your bottlenecks

Same average, same standard deviation, not same reality

Most application performance solutions are collecting performance data and only keep average and standard deviation (stddev). But application performance rarely follows normal distribution. Two samples with the same average and stddev doesn’t imply happy users.

Let’s suppose you have a first release of your app and see an histogram like this one

Most users are happy with a average of 1.9 seconds and standard deviation of 0.6 seconds

Let’s introduce our version 2.0 of the application. Our monitoring still shows an average of 1.9 seconds and standard deviation of 0.6 seconds.

But you receive a lot of feedback : 50%  of your end-users are complaining about bad performance… what’s going on ?

on the left the happy users… and on the right your unhappy end-users !
Hopefully you can easily instrument your application with jamon and discover this distribution.

Jamon is to System.currentTimeMillis() what log4j is to System.out.println()


Jamon collects “stop/start” events and aggregates the logarithmic distribution of these events/monitor.

  • 0-10ms. 11-20ms. 21-40ms. 41-80ms. 81-160ms. 161-320ms. 321-640ms.
  • 641-1280ms. 1281-2560ms. 2561-5120ms.
  • 5121-10240ms. 10241-20480ms. >20480ms.

It also keeps for each monitor additional informations like :

  • Hits
  • Avg ms.
  • Total ms.
  • Std Dev ms.
  • Min ms.
  • Max ms.
  • Active
  • Avg Active
  • Max Active
  • First access
  • Last access

The active, avg active and max active shows the degree of concurrency of your monitor.

Jamon feature and advantages :

  • easy installation : drop 3 jars, a bunch of jsp that’s it
  • production ready with low overhead
  • a servlet filter to monitor url time response by just modifying the web.xml
  • datasource wrapper to gather sql statistics (just an extra bean in your application)
  • spring integration via aop JamonPerformanceMonitorInterceptor
  • for non web application like batch processing or junit, you can write the jamon stats to a csv via a jmx console or a the end of the process.

Real life usage

0. mesure don’t guess
1. enable it in production
2. sort by total time
3. detect what can be improved in these use case : db indexes, hibernate batch-size,…
4. fix and rollout in production
5. goto 1😉

Alternative to jamon

  • codahale metrics looks really promising with implementation for Gauges, Ratio Gauges, Percent Counters, Histogram, Meters,… integration with Guice, Jetty, Log4j, Apache HttpClient, Ehcache, Logback, Spring, Ganglia, Graphite
  • javasimon : quantiles, hierarchichal, nanoseconds,… but jdk 1.6 and I’m stuck with websphere😦
  • datasource-proxysql no distribution but can summarize sql interactions per http request.But can be linked with other librairies

, ,

  1. #1 by Dharma on June 26, 2012 - 8:50 am

    Hi Thanks for providing about Jmonperfromance monitor.

    Can you help by providing the sample project with code and any architecture daigram ?
    Thanks in advance.

  1. Don’t Get Caught Hibernating (again) « Don't Make the Same Mistake Twice
  2. Jenkins as monitoring platform of the poor « Don't Make the Same Mistake Twice

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: