Well basically what I did is, like you said it could be the mail client, instead of installing squirrelmail client I googled a bit for RoundCube performance and found this:
from here, I checked a few settings, changed them properly and now my list of mails loads very quickly.
What I changed was:
'imap_cache' => 'db' # how/where and if to cache imap
'messages_cache' => true # caches the mesages into mysql which I found out its acctually faster than reading directly from mail directory
'imap_auth_type' => 'LOGIN' # which is my imap auth type, this setting skips RoundCubes one step towards authentication
'smtp_auth_type' => 'LOGIN' # so does this one
So yeah, basically this worked for me. Thanks Till.
By the way, I checked my server load and it was around 1.0 which is why I focused on the mail client.