ISPConfig uses pho-fcgi and not php-fpm.
I would like to have ability use different php versions for different sites, + different php.ini for each site.
Thats a function of ISPConfig. So no manual configuration needed.
Just follow the ISPConfig 3 perfect setup guide and you have a fully featured hosting server with php-cgi, php-fcgi, suphp and mod_php support builin. There is also no need to compile any php binarys, in these days, the Linux distriobutions are offering already php binaries for purposes like cgi, cli, mod_php etc. in their repositorys.
So all you have to do is follow one of these tutorials:
I recommedn to use the debian 6 tutorial.