I installed munin and monit. Thank you! I am close to solving this but here is the situation now.
The munin memory chart suggests to me that I have a memory leak in an application. It rises linearly until the swap is nearly maxed out then drops off shortly before mysql is noted to fail.
Check it out: http://picella.com/wiki/docs/memory.pdf
I have a number of php scripts running ... perhaps I have persistent connections somewhere? I have written a number of my own php scripts and I can see in mysql that one there are quite a few processes open by one of the database users that my scripts rely on.
Does anyone know how to track this down? Shouldn't php scripts close all open connections at the termination of the script?
See the attachement