Okay.. I've flushed the iptables again, and remade them with the Firewall option in WebMin. (Webmin > Networking > Linux Firewall)
All open ports are now pointing to the correct IP's, and the system is like normal :-)
For all people who have a slow SSH login and / or FTP (and yes even Email login), have a look at your IPtables! If there is a misstake in it, this can cause the slowdown!
Now my next problem... Interupts and context switches.. They are going crazy on the server here!