Ok, I think I figured it out.
When I try to FTP to my website using my IP address there is no name for ISPConfig to resolve to a website. For example, when I added www.mydomain.com
to my hosts file and tried to log in, it would let me log in just fine but just trying to log in with the IP it was giving me an invalid username and password error.
Without the name of the website my server could not resolve to the FTP request to the user's list for that website. So the FTP request to my server was still accepted, it was just accepted and resolved to the ISPConfig server itself instead of to the website I was trying to log into.