This chart should bring some realities home.
This particular server has 4GB RAM, and is losing almost 10% of that to ClamAV. ClamAV is a great tool - perhaps still the best on Linux, but this is simply the price you have to pay to run it.
4GB RAM would be the very least I'd put up with to run an internet-facing email server these days.
Remember, Just because you can doesn't mean you should.