Sadly, I am still experiencing this problem. Even after running the swap-clearing script in a weekly cronjob.
I strongly suspect that console-kit-daemon is the culprit here. Just now, it was reportedly using almost 50% of the system's ram. After killing the process, memory usage decreased dramatically.
Googling about it, I believe this daemon is meant for desktop installations and has no real function on a server. When I try to remove it, it says the following packages will be removed: consolekit dbus dbus-x11. Is it safe to remove these packages on an ispconfig server?