I ran some more scans. Two were intensive zenmap scans from another host to my server and gateway. The other is a nmap scan through a proxy to my IP.
I did a scan from http://mxtoolbox.com
and the results are much different.
25 smtp Thread was being aborted.
80 http Thread was being aborted.
443 https No connection could be made because the target machine actively refused it xx.xx.xx.xx:443
So perhaps it's a firewall issue. I'll try opening those up and see what happens.