What I want is to isolate you problem (not to find the exactly settings). If you can manage to do that, next you know where to work (if you have changes after this, you can see if you have problems only with apache, or with other process, or whatever)
When I am blind about a problem, always worked for me to discard little pieces one at a time.
I give special atention to your case, I will use debian with apache2 in production server in a few days, finding a solution for you will be great for me too. I will keep working on your problem.
Right know, in my test server, with just 2 test sites with almost no traffic, my ram consumption has growth from 340mb to 1.4gb max in 2 days (with no swaping because I have 4gb ram).
I think that apache2 debian package is not prepared out of the box to host vhosts... we will see for your health and mine :P