Make sure to grab your desktop's ssh fingerprint and verify it whenever you ssh in from the outside. Else you have no idea whether or not you're being man-in-the-middle'd.
ssh-keygen -lf /etc/ssh/ssh_host_rsa_key.pub
Also make sure you have a strong username. Once script kiddies find your IP and port, they'll run down an alphabetical list of common first names and try to log in. So if your name is Chris or something, and you use that as your username, that's a chunk of the battle lost right there.
Use an aggressive login grace period. I think on many distributions it's like 120 seconds. For internet facing machines, I crank it down to 10 seconds, which is about 8 seconds more than I need.
If you don't think you'll use sftp/scp on the host, consider disabling it, or at least enabling logging. Else if someone nabs your username/password somehow, they can download all of your carefully collected, useless junk without you knowing (in any kind of convenient fashion) that they're even logged on.
Subsystem sftp /usr/lib/openssh/sftp-server -l INFO
You can specify what ciphers are valid, which can be nice if ever you run into some jackass ssh client that tries arcfour first, which I think is still available by default in some distributions.
Ciphers aes128-cbc,aes192-cbc,aes256-cbc
Root login can be restricted to key only, but seriously if you're logging in as root, just give me your computer now. But seriously, prohibit-password will allow root to login via ssh keys. I mainly mention it because it can be super handy for internal use. If you have to allow root login from the outside world, ssh keys are the way to do it. If you don't have to allow root login, well, don't. You can always ssh in, change this setting, and restart the service should you need it.
#PermitRootLogin prohibit-password
PermitRootLogin no
And finally...Awww Dexter_Kane beat me to it. VPNs are nice. Personally I have both VPN and ssh available to the outside world. Sometimes admins restrict one but not the other, so having 2 options to get out of a network and into mine is nice.