How to find %WA in top

[hrdGn2007]

I've come across a very busy systems that is suffering from excessive waits. For example, the system is experiencing <10%CPU, <5%Idle and >90%WA as shown in top. Additionally, vmstat was showing blocked over 10 most of the time. Curiously, however, the % wait does seem to change much if we throw extra process that are io intensive ( such as tar ). My belief is that there is a utility used in the application that is driving the system nuts.

My question is, how do I find that offending process and what,exactly, is the resource it's abusing. I'm not a kernel developer, but in my reading of a couple of modern operating system books, there are memory structures that keep track of what processes are using what resources.

Is there a utility out there already that I can use?

I'd like to avoid hunting through code and resorting to guess work.

Thanks in advance,
hrdGn2007