Application profiling (Comparing 4 tools in my bag)

I have not been writing technical blog for very long. On the other hand, James have been writing quite a few technical blogs recently, and some of them are really interesting.

For today, I will probably write on my recent toys which I have been playinig around - profiling tools. The application that I am profiling is a J2EE application, multi-tier, blah blah blah.

Before we proceed, let me state my intention for this tool: I want to identify the performance hotspots within my application and thereby giving me the starting point to do refactoring work, or recoding of modules entirely.

So, I hope this is a simple enough objective for me... or is it?

Needless to say, even though I have previous experience on some commercial tools, this time round I try to stay away from some of my familiar ones - JProbe, OptimizeIt and even the free one which I like - Netbeans Profiler.

The "problem" with Netbeans Profiler in my present context is that my application runs on BEA's JRockit JVM and on JDK 1.4.2. To be fair, NB profiler have a specially instrumented JVM for JDK 1.4, but hack... I am not using Netbeans here in my team. I am using Eclipse.

So, the "natural" choice was to go for TPTP. There is also another option that I have: Mission Control from BEA. The Mission Control 2.0 ships with the JRockit VM.

---

Here are some of my experiences on Eclipse TPTP:

The good

1. It snaps into the IDE which my team is using which is Eclipse
2. I think it will continue to mature
3. We can profile a JRockit VM

The bad

1. The installation process is not that straight forward (besides downloading all the relevant and dependent plug-ins), at one point foiled my Eclipse installation. It was wise of me to do a backup of my hard-disk and I recovered everything to the original state. I hate to furge around trying to figure out what's wrong with the installation, when my original intention is to profile my application which I think is the problem - I don't need another problem
2. There is a separate agent controller and guess what? yes, it does not work on some old JRockit JVM. I have to figure that out and after banging my heads for a few days, I stumbled upon the possibility that perhaps it is due to a VM bug... so I went ahead to download the latest JRockit JVM and viola! it works

The ugly

1. This whole idea of having a agent controller is ridiculous. There are so many tools that don't need it and I have additional setup to do
2. Ridiculously intrusive - it drags my application to its knees in some use cases

Conclusion: I still find it a reasonable tool to use if you are doing a JPetstore. Not suitable if you are working on a large application. I think the tool needs more time to mature. And I think it will improve over time. I am not throwing in the towel for this one.

---

Now about Mission Control 2.0

The good

1. Almost brainless in the installation. There is only the standard profiling agent to be configured at the VM's runtime options - no big deal.
2. Light enough - as compared to TPTP. It didn't drag my application to its knees

The bad

1. Even though I can discover two of my agents running on a single box, only one of them could be profiled properly. The background is that I am running both the web and application tier on the same box. I think the agent must have gotten themselves confused...

The ugly

1. Absolutely ridiculous in the filtering of classes to be profiled and viewed for analysis. I cannot imagine a tool coming from such a reputable company and requires me to enter the full package and classname to be profiled (imagine, I have to type in java.util.Vector... and there are so many other classes I want to profile - all entered manually!!!) The fact is that when I do a profiling of my application, I want to identify the hotspots of my codes - if I supply the full resolved package and classname, that will mean I already know where the problem is! I was defeated by such a ridiculous design.

Conclusion: Not very useful to me (might be to other folks).

Now, I tried something commercial tool which I know might not be fair to compare with the previous highlighted tools. But I was determined to find something that I can use in the development (i.e. free or at least provide a development license).... BTW, I did try YourKit Java Profiler - it is not easy in installation and didn't work well for my application, so bye bye.

---

The next tool that I want to bring into the scene is JXInsight.

The good

1. A more complicated installation process compared to Mission Control, but close to what I should expect even out of JProbe or OptimizeIt, overall still manageable.
2. It supports so many platform (both OS and application container) and VM... unlike Netbeans Profiler...
   

The bad

1. Ghee, the documentation is really not well organized. Folks from JINSPIRED should really sit up on this one - you have a good tool, the problem is that you have people who want to use your tool. Your objective is to make the road easier. In many aspects, you have done so, but not from the Installation guide and User manual. It is frustrating to read it, not well index, the PDF files (included in the installation) does not have clickable indexes or content page. It can be seen that these folks are engineers.
2. The manuals (which is really the help files too), do not really shows step by step on how to use the different views and perspective. Newbies to these tool can get discouraged if they cannot use all the information presented to them - it is thrown to them.

The ugly

1. The user manual writer should really put in BOLD that the user requires to not only start the VM with all the required options and configuration setup but actually runs a transaction through the application BEFORE you can see anything on the console. Most of the other tools usually get snap into the agent and display heap graphs and things like that. This one doesn't - it polls and reports that it cannot find anything within the network.... until you hit your first transaction through the system and viola! all the lights are blinking.

Conclusion: I am using the developer edition now in my development environment, it has a time limit of 45 mins for profiling. Good enough for most purposes, and the most important thing - it works on my application and did not drag it to its knees. It takes a while before you can interpret what you see on those views and as I said, the documents did help that much. Overall, I am quite happy with this tool and it will certainly not be lying on the floor of my toolbox when I do profiling.

---

For those cool profiling tools maker out there, may I give some inputs to what a user wants:

1. Easy setup
2. Light (i.e. don't drag my application down anymore than it is now!!!) The reason why I am using your tool is to investigate the concerns of my application - not your tool
3. Please write the documentation well. People read them. If you want wide usage, it has to be easy to use and easy to understand the vast amount of data presented. Not everybody is a D.E (Distinguished Engineer) who can get productivity at the snap of the fingers. Tools should be written for people - people with a mission and task and it does not help to burden them further.
4. Yes, provide a developer's license is a great way. I am very willing to buy the commercial production license -and yes, I am the technical decision maker.
5. Support a wide range of OS, VM and application container. We don't just work on one project for the rest of our lives. We want to build on the knowledge of the tool, so, if the tool can also support another OS, and VM and container which my next project is working on... wow... I will embrace it almost like a religion (which translates to loyalty to the tool).
   

So, have fun profiling away.