This looks like a problem with the php binary that you installed, it simply does not seem to support the fastcgi mode. You can either install the php binarys from debian again or you try to compile your own binaries. The php binraies from debian are secure and up to date even if they are not the bleeding edge release, but if you need a stable system it is highly recommended to use the official debian binaries.
If you want to know more about how fastcgi is configured, take a look in the vhost file of the website in /etc/apache2/sites-enabled/
The difference of cgi and fcgi is that for cgi a instance of php is started when a page is requested and for fcgi php runs in some kind of daemon mode which is faster when you have a lot of requests.