Maybe i'm wrong but it sounds like your connection. I had previous server on 1/125 adsl and it was always slow. Considering 125 is about 25kb/s having 5 visitors on website can make a huge load for line.. I only had 1 website on that server..
Check your upload connection.. If you have 12 sites on it it doesn't require a lot of traffic..
Uh oh, get munin stats and check the traffic.