Well, the mail client you're using to retrieve mails. Example programs are Oulook, Thunderbird, Eudora to name a few. I assume you're running that on your personal PC. There's no reason to run a POP3 client on your server. POP3 is to retrieve email. If you want to access mail locally on your server, you should use a mail client/tool like 'Mutt'.
And my server is directly connected to the internet
Yes, as I noticed during my testing. That's oke.