Another approach that is more generic (can be used with any port/service) is to use the IPT_RECENT module that comes with netfilter:

For example I have the following lines in my iptables config:

iptables -N SSH_CHECK
iptables -A INPUT -p tcp --dport 22 -m state --state NEW -j SSH_CHECK
iptables -A SSH_CHECK -m state --state NEW -m recent --set --name SSH
iptables -A SSH_CHECK -m state --state NEW -m recent --update --seconds 60 --hitcount 4 --name SSH
iptables -A SSH_CHECK -m state --state NEW -m recent --rcheck --seconds 60 --hitcount 4 --name SSH -j DROP

which basically kick-bans the source IP for 60 seconds if more than 3 connections are attempted in a 60 second limit.

I've found this to be 100% effective.