Well, the path in the action is not the server path but "web path". You are actually pointing it to /my/directory/usr/lib/cgi-bin/php5
what I did is the following:
I copied the php4 binary (I use php4 now a cgi for a few things that don't run yet on php5) to my websites cgi folder:
Then I set accordings permissions to the cgi-bin folder and the binary.
and then I just put an .htaccess file within that web - whereever I want to use php as php4.
AddHandler php-script .php
Action php-script /cgi-bin/php4
So for you, just put the php5 binary in a cgi-bin folder, make sure the folder's and the binary's permissions are ok and then you can fine-grain directory by directory where you want to use what
Oh, regarding the php.ini - I have now to edit /etc/php4/cgi/php.ini to alter my php4-cgi ini