First I would recommend that you remove nginx from that setup. Your current hardware is able to run at least 100 or more wordpress blogs.
After you removed nginx, we can start to debug your setup to find out why it uses so much ram.
- Which php mode have you selected for the websites?
- Is suexec enabled?
- Post the output of: