If there is a signed cert on one site and a self signed cert on another, the results appear to be inconsistent where the SSL data served is a strange hybrid between the two.
This depends on the browser that you use. Take a look at wikipedia and search for sni ssl to get a list which browsers support sni.
Beside that, the behaviour of your system depends on the settings that you have made in the ispconfig interface and the things you mentioned above are already avilable, you just have not enabled them. You can disable sni under System > server config > web if you dont want to allow multiple ssl sites on one IP or if you can not ensure that all users use a sni capable browser and you can assign a IP address to one customer if you want to ensure that no other customer uses it.
As a genaral note, I use sni on several customer servers, it workks fine and the results are consistent.