suddenly can not login: Username or Password wrong.1

Discussion in 'Installation/Configuration' started by skysky, Mar 20, 2020.

  1. skysky

    skysky Member

    Hi

    I suddenly can not login ISPconfig 3 latest ver. recently I changed my CentOS root password, but I don't think it is related.

    I am very sure the user and password I input is correct, but I got this error: Username or Password wrong.1
    https://monosnap.com/file/xIMR5UjOfvW48wHOOCT8z1mc6nCDA9

    I tried the "lost password" button, but:
    The lost password function is not available for this user.

    what can I do?
     
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    Check that MySQL / MariaDB is started
     
  3. skysky

    skysky Member

    Hi Till
    It's is running, and all website has no problem.
    # mysqladmin -umysql ping
    mysqld is alive
     
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    Did you maybe alter the password of the ispconfig MySQL user?
     
  5. skysky

    skysky Member

    ok, I found out the cause of the issue. the server was hacked again and the ispconfig DB data gone and showing: To recover your lost Database and avoid leaking it... send bitcoin to .....
    ........

    the exact hack to ispconfig DB happened in May 2019.
    https://www.howtoforge.com/communit...ogin-and-ftp-access-failed.82038/#post-389307
    last year, I already changed all password to very long and complicated one. but the same issue still happens this year. what else can I do?
     
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    I guess you have to examine the access.log files of your server and try to find out how this happened. Also take care that all programs that you use to access MySQL use SSL encryption e.g. phpmyadmin. Check if mysql is listening on the external network interface or localhost, if you don't need external access and it currently listening externally, change it to localhost. Another possibility is that a desktop system which is used by a system administrator has a trojan. Which CentOS version do you use, if it's CentOS 6, then you should consider upgrading to a current version. Of course, the issue could be a vulnerability in ISPConfig itself, but this is not very likely as we would see hundreds or thousands of similar issues here in the forum then, and that's not the case. So it's more likely that it's an issue specific to your system.
     
  7. skysky

    skysky Member

    thanks for the advise. I am running centos 7 already. I am trying to check mysql config file to see if
    bind-address=127.0.0.1
    but when I vi /etc/mysql/my.cnf, it's empty.
    /etc/my.cnf ~/.my.cnf is ready only. why is that?

    # mysql --help | grep "Default options" -A 1
    Default options are read from the following files in the given order:
    /etc/mysql/my.cnf /etc/my.cnf ~/.my.cnf
     
  8. skysky

    skysky Member

    Hi
    I really need some help as my server is hacked again that ispconfig mysql DB encrypted causing I can no longer login ispconfig backend. and the only thing in DB is the message asking for bitcoin.:(

    I have secured my server SSH login with key, and no one can login with this key. so the hacking was not through ssh I guess.
    phpmyadmin was disabled, so not the cause.
    mysql is not listening on the external network interface.

    whatelse I should check?
     

    Attached Files:

    Last edited: Oct 16, 2020
  9. till

    till Super Moderator Staff Member ISPConfig Developer

    Are the tow mentioned databases all databases on this server, or are there other ones that are not affected?
     
  10. skysky

    skysky Member

    only these 2 DB affected, not all DB encrypted.
    I am downloading the access.log from /var/log/httpd/, and the files are huge. I don't know how to trace the issue.

    I have all the server DB backup, and I import back the ispconfig DB via phpmyadmin. but somehow I still can not login ispconfig showing login error.
     

    Attached Files:

    Last edited: Oct 16, 2020
  11. till

    till Super Moderator Staff Member ISPConfig Developer

    Ok, but as far as I can see, you don't have any other databases, which means all databases are affected. The MySQL and information_schema are system databases, which explains why they are not affected as you won't be able to read their message in case they would have encrypted them.
     
  12. skysky

    skysky Member

    all the important DB are on different server. How could they get into my server when I have done below:
    • SSH and FTP are using key to login, so they can not login without the key
    • Mysql is not open to external IP
    • ISPconfig and phpmyadmin web login also protected by .htaccess, only allow IP can login
    • antivirus ClamAV enabled.
    • server are patched regularly.
    • CMS are being updated reqularly.
    I import back the ispconfig DB, and I can now login again. but I don't know how it was hacked in, and I don't know if they change or place any files. should I roll back to previous server backup? but I have LE SSL certs renewed, if roll back, all SSL may fail.
     
    Last edited: Oct 16, 2020
  13. till

    till Super Moderator Staff Member ISPConfig Developer

    The first important thing to note is that all databases on this server were hacked, this indicates that the attacked used the MySQL root user or a user with similar permissions. If outside access to MySQL is closed, then the user must have been able to run shell commands (not necessarily as shell root user, but maybe as user of one of the hosted websites) or that the server has been compromised by a rootkit. Plus the attacker must be able to get the mysql root password somehow, possible reasons might be a insecure password used somewhere else or the password was stored on the server in a file that's world readable or the password was stored in a backup and this backup was accessible to the attacker. It might also be that an attacker from previous attack as left a backdoor which e.g. gets reactivated by a cronjob as it's not the first time that this happened on your system.

    Even with all updates installed in our cms, it might be that you run a plugin or theme on one of these cms which is vulnerable e.g. because the author does not provide updates anymore.

    The best solution would probably be a reinstall, if possible. then take care to disable shell exec functions in all websites that don't need them, and use different mysql passwords and ensure that config files from websites which contain MySQL passwords are chmod 0600.
     
  14. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    Also schedule an update to ISPConfig 3.2, there are security fixes that could come into play. Check what other services you might have running on the machine, either listening on the network or even listening on localhost (as you run websites on this server, a hacked site can connect to any service listening on the localhost addr). Keep a copy of your logs files to pore over and try to narrow a timeframe when this happened (though actual access to your systems could have been quite some time prior).
     
  15. skysky

    skysky Member

    thanks for the advise. I think restoring my backup may be meaningless as my backup may have the same virus file. reinstall everything to a new server is huge work, and I want to try more ways first as I did not see other issues in the past 3 attacks except the ispconfig DB being encrypted. (I fixed it by importing back the DB backup now)

    Here is what I am going to do with your advise
    1. check Access log, not sure can help to identify the issue in this case?
    2. change the mysql root password
    3. disable shell exec functions in all websites that don't need them (not sure how to find these files)
    4. rootkit scan with rkhunter -c (nothing found)
    4. Check what other services running on the machine, either listening on the network or even listening on localhost
    5. update or remove old cms plugin or theme
    6. update to ISPConfig 3.2

    the hardest thing is If I can not find out the exact cause (how exactly it happened) , I will never know if the problem is fixed until it happens again.
    I google the message the hacker sent me to see how other people trace the issue:

    To recover your lost databases and avoid leaking it: visit http://hn4o6s5nc7763.onion and enter your unique token bce7db0d and pay the required amount of Bitcoin to get it back. Databases that we have: c2jfile_manager, dbispconfig. Your databases are downloaded and backed up on our servers. If we dont receive your payment in the next 9 Days, we will sell your database to the highest bidder or use them otherwise. To access this site you have use the tor browser https://www.torproject.org/projects/torbrowser.html

    and I found:
    https://www.bitcoinabuse.com/reports/12edUCaxy2gGPMRjcJMGmgxip75jZ8QaJg
    https://medium.com/@kalana.16/tracing-the-origin-of-a-real-life-mysql-hack-5a2a6f21fe66
    https://www.howtoforge.com/communit...oid-leaking-it-send-us-0-1-bitcoin-btc.81962/
     
    Last edited: Oct 16, 2020
  16. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    It's probably less work that exhaustively searching a compromised system for all places that Bad Things(tm) may be hiding. But, do both - the latter is an excellent learning process, and reinstalling has a higher reassurance of less backdoors. You could even change distributions and get to newer software versions.

    It could help if the initial vulnerability(ies) exploited were via the web. Also keep in mind XSS family of vulnerabilities, the access log could show requests made by you yourself (eg. a stored xss attack could have been in place for some time before you did whatever triggered it).

    Change all db passwords, especially other users with high levels of permissions like 'ispconfig' and 'debian-sys-maint'. There is also sensitive info in the data they stole, definitely reset all ISPConfig account passwords (including any remote users for the api), database account passwords, change dkim keys, replace any ssl certificates you uploaded under the SSL tab, and you'll have password hashes of many things, etc. ftp accounts, shell and email accounts, website stats. And there's more (rspamd interface password, and surely more) things to uncover as you browse. Change all shell user passwords (both system level accounts and those within ISPConfig), and audit all users in /etc/passwd.

    You edit the numerous php.ini files on your system, there is one for each mode of each php version you have. Be sure you use a comprehensive list of functions to disable (there are quite a few), and keep in mind when you edit php.ini for your CLI version(s) that you need many of them for normal use (eg. if you disable exec() for the system default php cli, most of ISPConfig's back-end will cease to function).

    Setting open_basedir helps some, too.
    Throw in maldet, and be sure you have a good signature set in clamav when it runs. ISPProtect is another option to try, as are chrootkit, lynis, ossec, and others. They may or may not find anything. Some of them can work much better by setting them up on a clean baseline system, so they can detect when things change, rather than having to positively identify the changed thing.

    Check cron jobs and init scripts, and of course check for web-accessible backdoors left anywhere. Sometimes even php's .user.ini is used to reinfect things (via auto prepend).

    Also remove old site copies (dev/testing sites, backups, etc.).


    One last note, you have a database named c2jfile_manager, and web-based file managers are difficult to write securely; be sure to consider that app as candidate for being exploited. PHPmyadmin is more likely as it was only your databases which were affected, but worth checking.
     
  17. skysky

    skysky Member

    I found these very suspicious files in mysql folder, I guess they are source of the hack.
    They are from 2018. so my backup won't help as my backup is not before 2018.
    I del them already. Can I assume my system is clear now?
     

    Attached Files:

    • log.png
      log.png
      File size:
      117.1 KB
      Views:
      8
    Last edited: Oct 17, 2020
  18. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    No. Those wouldn't be the initial means of compromise, but left there afterwards. Perhaps they could cause later reinfection via some means, or perhaps they are something like Windows executables, that wouldn't even run on your Linux box. Did you examine them at all to see what they may have been? I'm just curious.
     
    Th0m likes this.

Share This Page