Ok maybe have it solved??
As I followed the extended debian setup I also installed the ddos script that runs... So I went to http://deflate.medialayer.com/
to uninstall it - also noted that a email could be sent to the root so that ip's blocked would be listed... anyway currently it is 4 hours later and not a single drop out of ventrilo... So i'm also hoping not a drop out from any of my services running (email,apache,etc)
Fingers crossed that this was the issue.
I will post back in a day or so if this fixes the problem else I'll post back sooner if it isn't