The problem is most likely that you dont have enough ram and that your vserver is not allowed to have additional swap space.
Its worth a try to free some ram before you try to install ISPConfig. Stop the services that are not nescessary during installation:
apache, bind, postfix, courier (all courier daemons: pop3, pop3s, imap, imaps and the auth daemon) or dovecot, proftpd or vsftpd
and then try to install ispconfig again.