Oddly enough ive noticed that while on the local network, ping times go from 1ms to 27-47ms when the server isn't responding, but only on the primary IP. It doesn't ever slow down on the secondary IP (on eth0:1)
munin doesnt show any excessive loads on anything that should be causing this (and the server is 2x Quad core Xeon 8gb ram, Gig-E) So is it possible that a service is causing the eth0 address to respond slowly, but not the eth0:1? Or is it a network issue. The server always gets all services outbound, just never inbound. Ive already replaced the switch, since i had a second lying around, just to narrow it down.
Any ideas with this new Info?