If you sometimes have to ssh in from computers that are outside of your control and therefore don't have access to your private key you can use S/Key authentication. S/Key is a one time password system that uses a program on your PDA to generate the passwords. That way if someone sniffs or key logs your password it will not work twice.

Also, if you get tired of typing ssh -p 31337 you can add that to your user level configuration file like this:
~/.ssh/config:

Host hostname
  Port 31337
  User username