I have tried it with Firefox, and I see your point.
It's definately nothing to do with ISPConfig though. It's to do with the Certification Authority who provided the SSL Certificate. I think it's basically because Firefox doesn't have the root certificates for OVH Secure Certification Authority, whoever they are.
Unless I've missed something here, I think the only resolution is to obtain an SSL Certificate from a reputable provider such as Thawte or Verisign (yes, I know Verisign own Thawte now! ;-) Thawte do a reasonably priced budget certificate called SSL-123. But that's still paying twice, unless you can get a refund.
If you go for a less well known SSL provider, unfortunately you run the risk of the CA not being recognised by some of the browsers. In this case, it seems to work with IE7 and Safari, but not in Firefox or Opera.