Okay, so I told FileZilla (my FTP client) to only use ports 49152 to 65534. Everything else default. I put "AllowForeignAddress on" and the PassivePorts to the same range. I port forwarded these ports on my router to my server.. (192.168.1.254)
It seems to work, I'm just careful to connect/disconnect as rarely as possible, and it seems to be alright.
When I have a chunk of time I'll see if this actually did anything beneficial or if it's only better because I'm connecting less frequently over all. But I don't really want to connect/ disconnect a lot and see if it messes up unless I'm not going to be around for a few days.
So in conclusion, I have no idea if I fixed it or not, and I don't want to find out, b/c it works now.. but if I have a chunk of time I can afford to possibly lose by checking, I'll let you guys know.