Well the free -m output explains it all
total used free shared buffers cached
Mem: 1003 974 29 0 150 723
-/+ buffers/cache: 100 903
Swap: 2588 0 2588
That's 723MB cashed memory, so my box is pretty healthy. I was over paranoid because I'm setting up this box to sell shared hosting and I don't want to screw people over if the box starts to swap aggressively.