Blocked jobqueue

Discussion in 'Installation/Configuration' started by BobGeorge, Jul 25, 2017.

  1. BobGeorge

    BobGeorge Member HowtoForge Supporter

    The jobqueue keeps blocking as I'm putting my multi-server cluster together. I've tried the htf-common-issues script and it reports nothing untoward. I've tried running the server.sh script manually - with loglevel "debug" - and that's revealing nothing.

    But then I tried "mysql -u ispcsrv2 -h node1.domain.tld" from node0 - I looked in the database with phpmyadmin to see that the user is "ispcsrv2" and that there is no password (field is blank). And I'm getting an "access denied".

    Generally, MySQL / MariaDB as installed on Ubuntu 16.04 has been a pain because it locks down everything by default. As the multi-server install instructions suggest, I created "root" users for every IP address and hostname. I cleared the "plugin" that's installed that prevents remote logins. I also created a 'root'@'%' user so that you ought to be able to login as root from anywhere (with the correct password).

    I can see, with phpMyAdmin, that the master server (node0) has created the "ispcsrv2" and "ispconfig" users. Node1 has an "ispconfig" user.

    And, yes, I gave it over 15 minutes waiting for the job to disappear. It was definitely blocked. Indeed, I tried uninstalling and reinstalling (including manually clearing out the user entries with phpMyAdmin too) and it still happens.

    I think what's happening is that the master server is getting an "access denied" to the other node, like I am when I try to login in manually from the command line, and this means the job blocks forever as it can never be completed.

    Edit - Yes, when I run "uninstall.php" to try it again, then it says "access denied" with logging in as "[email protected]" from node1. And the node doesn't disappear on node0's interface. So it's basically not getting through to the master server to let it know it's uninstalling.
     
    Last edited: Jul 25, 2017
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    The master node does not connect to a slave. Slaves are connecting to the master. So when you want to debug this, you have ti run server.sh on the slave.
     
  3. BobGeorge

    BobGeorge Member HowtoForge Supporter

    I have run server.sh on the slave and it's repeatedly saying:
    Access denied for user 'ipscsrv2'@'node1.domain.tld' (using password:YES).

    But that does highlight something interesting. The user "ispcsrv2" is on node0, not node1. I'm accessing from node1, but the user is on node0. Is it trying to access "ispcsrv2" on its own database or something? And, if so, why?
     
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    The ispcsrv* users exist on the master only and slaves connect with them to the master to fetch data. The 'local' database user that ISPConfig uses is always the user 'ispconfig'.

    Either you entered a wrong hostname when asked for the master server during install of the slave or there is something wrong in the /etc/hosts file on the master or the slave.
     
  5. BobGeorge

    BobGeorge Member HowtoForge Supporter

    The hosts file is correct and identical on both machines.

    I'll reinstall again and make double sure that I've got the master hostname correct, I guess.
     
  6. BobGeorge

    BobGeorge Member HowtoForge Supporter

    Okay, I've reinstalled and, on the slave, "server.sh" is saying:

    PHP Warning: mysqli_connect(): (HY000/1045): Access denied for user 'ispcsrv2'@'slave.domain.tld' (using password: YES) in db_mysql.inc.php on line 79

    And also the same error on line 85 five times in a row.

    From the slave, I've tried "sudo mysql -u ipscsrv2 -h master.domain.tld" and it logs in just fine.

    I notice that the error message says "using password: YES" but there is no password when I login. And I can see, via phpMyAdmin, that the password field is blank for both "ispcsrv2" entries (one the IP address, which is correct, and the other the hostname, which is correct).

    The error messages suggest that the slave is trying to login to the slave, not the master. The host should be "master.domain.tld" - which I've manually tested works - but it says "slave.domain.tld" in the message.
     
  7. till

    till Super Moderator Staff Member ISPConfig Developer

    Which hostname is set for the master database connection in the file /usr/local/ispconfig/server/lib/config.inc.php on the slave?
     
  8. BobGeorge

    BobGeorge Member HowtoForge Supporter

    $conf['dbmaster_host'] = 'master.domain.tld';

    I do notice that it's got a random password listed ('dbmaster_password'), but the password field is blank for "ispcsrv2" when I look at it with phpMyAdmin.
     
  9. BobGeorge

    BobGeorge Member HowtoForge Supporter

    Okay, I think I might just have solved it.

    I noticed that none of these user accounts that ISPConfig was creating had passwords in phpMyAdmin - the password field was blank - but they are trying to set and use passwords, as I can see this with "server.sh" output.

    So I checked the "plugin" field on the MySQL users and it's also blank. So I put "mysql_native_password" into the plugin field before adding the slave and now the user accounts have non-blank password fields and the job queue actually empties.

    For some reason, on my set up, when new users are created, the "plugin" field is blank and not "mysql_native_password". But without "mysql_native_password", to explicitly tell it to use passwords for authorisation, it seems that MySQL won't let you set a password for a new user or login with the password properly.

    A bit annoying that I've got to manually enter that into the database myself - I'll have to investigate if there's a way to set this globally somehow - but it does seem to work now that I've done that.
     
    Jesse Norell likes this.

Share This Page