Eye on performance: Wait leaks

By Jack Shirazi & Kirk Pepperdine
2005-04-28


Spotting a Wait Leak
A thread dump shows the symptoms of the wait leak, but what is important is what is missing from the thread dump -- the thread that is not notifying the waiting threads. So you need to add in some extra contextual information to help identify a wait leak. Typically, there are two failure modes that might get reported -- a deadlock, or gradual degradation in application responsiveness.

Let's first consider the standard deadlock-type problem report: The application is no longer doing anything (although it might still be responsive to a user initiated event) -- the application is partially or completely frozen. The symptoms of a wait leak resemble the symptoms of a garden-variety deadlock report, except there is no sign of deadlock in the stack dumps. If you see that, consider that you might have a wait leak.

The second scenario is a gradually overloaded, less responsive application. In this case, more and more threads enter the wait-leak state over time, which means that more and more threads (which are supposed to be doing something) simply sit there and do no work. Eventually, the application becomes clogged with threads waiting for a notification that will never come. Eventually, some resource will be exhausted -- perhaps a depleted thread pool, or an out-of-memory error from too many threads, or just a non-responsive application as the application finally reaches the equivalent of the first kind of deadlock-type symptom. This is possibly an easier wait leak to diagnose, because you can compare stack dumps over time and see that the number of some particular Object.wait() stack (possibly using the same locks) just keeps increasing. One production example we saw had a server that gradually produced slower responses, until, near the end, 43 wait-leak stacks become 108 wait-leak stacks just a couple of minutes later, shortly after which the server no longer responded to requests.

Tutorial Pages:
» Gain a Better Understanding of this Curious Race Condition
» Wait Leaks
» Simulating a Wait Leak
» Spotting a Wait Leak
» The Final Word
» Resources


First published by IBM developerWorks

Related Tutorials: » All about JAXP, Part 1 » Make Database Queries Without the Database » Load List Values for Improved Efficiency » 2 Ways To Implement Session Tracking » A Simple Way to Read an XML File in Java » Develop Aspect-Oriented Java Applications with Eclipse and AJDT