I finally had some time to look into the issue with the SSL-connection and the Android app for ISP config.
It looks like my server-config is still not correct, regarding the installation of the certificates.
When I visit my server's config page on a PC, I get the correct site and certificate (with Chrome, IE and Firefox).
But when I visit the same url with the browser on my Android phone, I get an certificate-error and when I look at the certificate, it appears to be the certificate for another site running on the same server.
So it seems the issues with the ISPconfig app indeed are to be related to an issue with SSL and certificates.
I hope you can make it easier to install a certificate for the ISP-config host, for a later version of ISP config. It's still not entirely clear to me what went wrong with installing it.
But for now, I've created a special vhost like you described on the ISPconfig-app page for "self-signed" certificates and it works again.
Edit: It appears that the stock-browser in Android 2.3 doesn't support SNI
and that seems to be what's causing this issue.