jobsqueue stuck after server move

Discussion in 'Installation/Configuration' started by chris weinhaupl, Mar 1, 2022.

  1. Hey all,
    Had to move the ispconfig server to a new subnet. After the move ispconfig wont process the jobqueue. I made some ispconfig like updating the IP address.
    What may I have missed that prevents the jobqueue from processing ispconfig updates?

    Thank you
     
  2. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    Check the ip addr in /etc/hosts and both db_host and dbmaster_host in /usr/local/ispconfig/{server,interface}/lib/config.inc.php, and ensure you have granted permissions in mysql for connections from the new ip address (should still work via the hostname if /etc/hosts is correct, but set for the ip addr as well to be safe).
     
    Th0m likes this.
  3. Thanks for the thoughts.

    Check the ip addr in /etc/hosts
    localhost is 127.0.0.1

    /usr/local/ispconfig/interface/lib/config.inc.php
    db_host is 'localhost'
    dbmaster_host is blank ''

    I can log into the mysql from the command line using the creds in the config.inc.php file.
    [email protected] etc

    What also is interesting is that I can log into the ISPconfig web user interface. I see the settings etc. So I am thinking I am connected to the dbispconfig at this point.
    What I cant seem to do it make updated/changes so now I have 5 tasks in the jobsqueue.
     
  4. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Is /etc/hosts only one line? It should have 127.0.1.1 also and IP of host with hostname. If a multiserver setup, IP and hostname of all hosts in the setup.
     
  5. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    Sorry, that syntax may be confusing, it means check both /usr/local/ispconfig/interface/lib/config.inc.php (which seems to be correct) and /usr/local/ispconfig/server/lib/config.inc.php (which acts broken).
     
  6. Thanks again.
    Hmmmm


    [email protected]:/usr/local/ispconfig# cat /etc/hosts
    127.0.0.1 localhost.localdomain localhost
    ::1 localhost6.localdomain6 localhost6

    # The following lines are desirable for IPv6 capable hosts
    ::1 localhost ip6-localhost ip6-loopback
    fe00::0 ip6-localnet
    ff02::1 ip6-allnodes
    ff02::2 ip6-allrouters
    ff02::3 ip6-allhosts
    10.19.45.125 zabbix.mgt.int
    10.14.14.27 webserver.domain.com <-- we updated this


    I blanked the passwords below. The db_password is different from the db_master_password

    [email protected]:/home/support# less /usr/local/ispconfig/interface/lib/config.inc.php | grep db
    $conf['db_type'] = 'mysql';
    $conf['db_host'] = 'localhost';
    $conf['db_port'] = '3306';
    $conf['db_database'] = 'dbispconfig';
    $conf['db_user'] = 'ispconfig';
    $conf['db_password'] = '****************************************';
    $conf['db_charset'] = 'utf8'; // same charset as html-charset - (HTML --> MYSQL: "utf-8" --> "utf8", "iso-8859-1" --> "latin1")
    $conf['db_new_link'] = false;
    $conf['db_client_flags'] = 0;
    define('DB_TYPE',$conf['db_type']);
    define('DB_HOST',$conf['db_host']);
    define('DB_PORT',$conf['db_port']);
    define('DB_DATABASE',$conf['db_database']);
    define('DB_USER',$conf['db_user']);
    define('DB_PASSWORD',$conf['db_password']);
    define('DB_CHARSET',$conf['db_charset']);
    $conf['dbmaster_type'] = 'mysql';
    $conf['dbmaster_host'] = '';
    $conf['dbmaster_port'] = '';
    $conf['dbmaster_database'] = 'dbispconfig';
    $conf['dbmaster_user'] = '';
    $conf['dbmaster_password'] = '****************************************';
    $conf['dbmaster_new_link'] = false;
    $conf['dbmaster_client_flags'] = 0;
    $conf['start_db'] = true;
    [email protected]:/home/support#



    [email protected]:/usr/local/ispconfig/server/lib# less config.inc.php | grep db
    $conf['db_type'] = 'mysql';
    $conf['db_host'] = 'localhost';
    $conf['db_port'] = '3306';
    $conf['db_database'] = 'dbispconfig';
    $conf['db_user'] = 'ispconfig';
    $conf['db_password'] = '****************************************';
    $conf['db_charset'] = 'utf8'; // same charset as html-charset - (HTML --> MYSQL: "utf-8" --> "utf8", "iso-8859-1" --> "latin1")
    $conf['db_new_link'] = false;
    $conf['db_client_flags'] = 0;
    define('DB_TYPE',$conf['db_type']);
    define('DB_HOST',$conf['db_host']);
    define('DB_PORT',$conf['db_port']);
    define('DB_DATABASE',$conf['db_database']);
    define('DB_USER',$conf['db_user']);
    define('DB_PASSWORD',$conf['db_password']);
    define('DB_CHARSET',$conf['db_charset']);
    $conf['dbmaster_type'] = 'mysql';
    $conf['dbmaster_host'] = '';
    $conf['dbmaster_port'] = '';
    $conf['dbmaster_database'] = 'dbispconfig';
    $conf['dbmaster_user'] = '';
    $conf['dbmaster_password'] = '****************************************';
    $conf['dbmaster_new_link'] = false;
    $conf['dbmaster_client_flags'] = 0;
    $conf['start_db'] = true;
    [email protected]:/usr/local/ispconfig/server/lib#





    [email protected]:/usr/local/ispconfig/server/lib# mysql -u ispconfig -p
    Enter password:
    Welcome to the MySQL monitor. Commands end with ; or \g.
    Your MySQL connection id is 384
    Server version: 5.7.37-0ubuntu0.18.04.1 (Ubuntu)
    Copyright (c) 2000, 2022, Oracle and/or its affiliates.
    Oracle is a registered trademark of Oracle Corporation and/or its
    affiliates. Other names may be trademarks of their respective
    owners.

    Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

    mysql> show databases;
    +--------------------+
    | Database |
    +--------------------+
    | information_schema |
    | dbispconfig |
    +--------------------+
    2 rows in set (0.00 sec)

    mysql>
     
  7. Quick update
    I did some testing.

    Remove IP from "IP Addresses" web page by selecting the garbage can on the right
    Looking in the phpmyadmin I can see that the same IP address is now gone from the database.
    The IP had been in the server_ip table before i deleted the ip

    in the jobsqueue I still have the task
    2022-03-04 00:16 webserver.domain.com Delete server_ip
     
  8. just updated to php.7.4 from php.7.2
    same issue in that the commands writing to the database are fine, but the jobtask still stays in queue.
    I tried disabling and re-enabling a website and that does not write to the disk the updates to the apache2 conf files
     
  9. one of my sites has a failed cert which started this entire process.
    I have removed the symlink for the site from /etc/apache2/sites-enabled/ so I could start apach2
     
  10. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    Those should be the same in a single server. Or maybe db_master_password should be empty.

    Run /usr/local/ispconfig/server/server.sh and see if it outputs anything useful. If you can access the ispconfig interface, set the server log level to debug first.
     
  11. I have a single server.
    I have access to the ispconfig web interface

    [email protected]:/etc/apache2/sites-enabled# /usr/local/ispconfig/server/server.sh
    04.03.2022-10:53 - WARNING - There is already a lockfile set, but no process running with this pid (19881). Continuing.
    PHP Warning: include_once(/usr/local/ispconfig/server/plugins-enabled/server_services_plugin.inc.php): failed to open stream: No such file or directory in /usr/local/ispconfig/server/lib/classes/plugins.inc.php on line 65
    PHP Warning: include_once(): Failed opening '/usr/local/ispconfig/server/plugins-enabled/server_services_plugin.inc.php' for inclusion (include_path='.:/usr/share/php') in /usr/local/ispconfig/server/lib/classes/plugins.inc.php on line 65
    PHP Fatal error: Uncaught Error: Class 'server_services_plugin' not found in /usr/local/ispconfig/server/lib/classes/plugins.inc.php:67
    Stack trace:
    #0 /usr/local/ispconfig/server/server.php(172): plugins->loadPlugins('all')
    #1 {main}
    thrown in /usr/local/ispconfig/server/lib/classes/plugins.inc.php on line 67
     
  12. till

    till Super Moderator Staff Member ISPConfig Developer

    Run the command:

    php --version

    and ensure that you did not accidentally installed PHP 8 or 8.1 as the default version. You said you updated to PHP 7.4, it might be that you missed installing some PHP packages, the default PHP version of a server should always stay at the version that ships with the OS and should not be changed.

    According to the errors, it might be that parts of the ISPConfig source code have been deleted on your server. Run:

    ispconfig_update.sh --force

    to do an ISPConfig update and let the updater also reconfigure services.
     
  13. we might be onto something.


    [email protected]:/etc/apache2/sites-enabled# ispconfig_update.sh --force
    Select update method (stable,nightly,git-develop) [stable]:
    sh: /usr/local/ispconfig/server/scripts/update_runner.sh: No such file or directory
    [email protected]:/etc/apache2/sites-enabled#

    [email protected]:/etc/apache2/sites-enabled# ispconfig_update.sh
    Select update method (stable,nightly,git-develop) [stable]:
    There are no updates available for ISPConfig 3.2.7p1
    If you are sure you want to update to stable anyway, please use --force parameter
    DOWNGRADING MAY CAUSE ISSUES!
    [email protected]:/etc/apache2/sites-enabled#



    php check:
    I was still running this 7.2 version
    [email protected]:/etc/apache2/sites-enabled# php --version
    PHP 7.2.24-0ubuntu0.18.04.11 (cli) (built: Mar 2 2022 17:52:35) ( NTS )
    Copyright (c) 1997-2018 The PHP Group
    Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
    with Zend OPcache v7.2.24-0ubuntu0.18.04.11, Copyright (c) 1999-2018, by Zend Technologies

    Then I did a php update that installed 8.1
    I removed php 8.1
    I installed php7.4

    [email protected]:/etc/apache2/sites-enabled# php --version
    PHP 7.4.28 (cli) (built: Feb 17 2022 16:06:19) ( NTS )
    Copyright (c) The PHP Group
    Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.28, Copyright (c), by Zend Technologies


     
  14. okay, server screwed.
    recovered from backup last week.

    ubuntu 18.04
    ispconfig just updated
    Help > version reports: ISPConfig Version: 3.2.7p1
    Monitor still reports:
    Server: webserver.domain.com (Ubuntu 18.04.6 LTS (Bionic Beaver)) ISPConfig 3.1.15p2

    I have moved the single isponfig to a new IP.
    I restarted ispconfig and would not start, missing crt for a SSL for one of the sites.
    I removed the offending 100-* from /etc/apache2/sites-enabled/
    apache2 started
    Now I can see ispconfig web interface and log in
    Made an update and the update is stuck in the jobqueue

    I ran a few commands to see whats up:


    [email protected]:/etc/mysql# update-alternatives --config php
    There is only one alternative in link group php (providing /usr/bin/php): /usr/bin/php7.2
    Nothing to configure.

    [email protected]:/etc/mysql# update-alternatives --config php-cgi
    There is only one alternative in link group php-cgi (providing /usr/bin/php-cgi): /usr/bin/php-cgi7.2
    Nothing to configure.

    [email protected]:/etc/mysql# /usr/local/ispconfig/server/server.sh
    04.03.2022-16:44 - WARNING - There is already a lockfile set, but no process running with this pid (20084). Continuing.
    PHP Warning: include_once(/usr/local/ispconfig/server/plugins-enabled/server_services_plugin.inc.php): failed to open stream: No such file or directory in /usr/local/ispconfig/server/lib/classes/plugins.inc.php on line 65
    PHP Warning: include_once(): Failed opening '/usr/local/ispconfig/server/plugins-enabled/server_services_plugin.inc.php' for inclusion (include_path='.:/usr/share/php') in /usr/local/ispconfig/server/lib/classes/plugins.inc.php on line 65
    PHP Fatal error: Uncaught Error: Class 'server_services_plugin' not found in /usr/local/ispconfig/server/lib/classes/plugins.inc.php:67
    Stack trace:
    #0 /usr/local/ispconfig/server/server.php(172): plugins->loadPlugins('all')
    #1 {main}
    thrown in /usr/local/ispconfig/server/lib/classes/plugins.inc.php on line 67

    [email protected]:/etc/mysql# php --version
    PHP 7.2.24-0ubuntu0.18.04.11 (cli) (built: Mar 2 2022 17:52:35) ( NTS )
    Copyright (c) 1997-2018 The PHP Group
    Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
    with Zend OPcache v7.2.24-0ubuntu0.18.04.11, Copyright (c) 1999-2018, by Zend Technologies
    [email protected]:/etc/mysql#
     
  15. till

    till Super Moderator Staff Member ISPConfig Developer

    So parts of ISPConfig must have been deleted already before this backup was taken. Please run the command:

    ls -la /usr/local/ispconfig/server/plugins-available/

    and post the result.
     
  16. [email protected]:/etc/mysql# ls -la /usr/local/ispconfig/server/plugins-available/
    total 880
    drwxr-x--- 2 root root 4096 Mar 4 15:01 .
    drwxr-x--- 13 root root 4096 Mar 4 15:01 ..
    -rwxr-x--- 1 root root 180826 Mar 4 15:00 apache2_plugin.inc.php
    -rwxr-x--- 1 root root 11497 Mar 4 15:00 apps_vhost_plugin.inc.php
    -rwxr-x--- 1 root root 3794 Mar 4 15:00 aps_plugin.inc.php
    -rwxr-x--- 1 root root 15197 Mar 4 15:00 backup_plugin.inc.php
    -rwxr-x--- 1 root root 11201 Mar 4 15:00 bind_dlz_plugin.inc.php
    -rwxr-x--- 1 root root 24159 Mar 4 15:00 bind_plugin.inc.php
    -rwxr-x--- 1 root root 15594 Mar 4 15:00 cron_jailkit_plugin.inc.php
    -rwxr-x--- 1 root root 11872 Mar 4 15:00 cron_plugin.inc.php
    -rwxr-x--- 1 root root 9273 Mar 4 15:00 firewall_plugin.inc.php
    -rwxr-x--- 1 root root 4731 Mar 4 15:00 ftpuser_base_plugin.inc.php
    -rwxr-x--- 1 root root 6595 Mar 4 15:00 getmail_plugin.inc.php
    -rwxr-x--- 1 root root 1965 Mar 4 15:00 iptables_plugin.inc.php
    -rwxr-x--- 1 root root 11964 Mar 4 15:00 maildeliver_plugin.inc.php
    -rwxr-x--- 1 root root 12702 Mar 4 15:00 maildrop_plugin.inc.php
    -rwxr-x--- 1 root root 6643 Mar 4 15:00 mailman_plugin.inc.php
    -rwxr-x--- 1 root root 17137 Mar 4 15:00 mail_plugin_dkim.inc.php
    -rwxr-x--- 1 root root 26819 Mar 4 15:00 mail_plugin.inc.php
    -rwxr-x--- 1 root root 25834 Mar 4 15:00 mongo_clientdb_plugin.inc.php~
    -rwxr-x--- 1 root root 39362 Mar 4 15:00 mysql_clientdb_plugin.inc.php
    -rwxr-x--- 1 root root 13585 Mar 4 15:00 network_settings_plugin.inc.php
    -rwxr-x--- 1 root root 178352 Mar 4 15:00 nginx_plugin.inc.php
    -rwxr-x--- 1 root root 11948 Mar 4 15:00 nginx_reverseproxy_plugin.inc.php
    -rwxr-x--- 1 root root 6991 Mar 4 15:00 openvz_plugin.inc.php
    -rwxr-x--- 1 root root 4358 Mar 4 15:00 pma_symlink_plugin.inc.php
    -rwxr-x--- 1 root root 5046 Mar 4 15:00 postfix_filter_plugin.inc.php
    -rwxr-x--- 1 root root 18732 Mar 4 15:00 postfix_server_plugin.inc.php
    -rwxr-x--- 1 root root 23926 Mar 4 15:00 powerdns_plugin.inc.php
    -rwxr-x--- 1 root root 21822 Mar 4 15:00 rspamd_plugin.inc.php
    -rwxr-x--- 1 root root 21012 Mar 4 15:00 shelluser_base_plugin.inc.php
    -rwxr-x--- 1 root root 25959 Mar 4 15:00 shelluser_jailkit_plugin.inc.php
    -rwxr-x--- 1 root root 2771 Mar 4 15:00 software_update_plugin.inc.php
    -rwxr-x--- 1 root root 5510 Mar 4 15:00 squid_plugin.inc.php
    -rwxr-x--- 1 root root 4349 Mar 4 15:00 webmail_symlink_plugin.inc.php
    -rwxr-x--- 1 root root 13655 Mar 4 15:00 webserver_plugin.inc.php
    -rwxr-x--- 1 root root 18244 Mar 4 15:00 xmpp_plugin.inc.php
    -rwxr-x--- 1 root root 5300 Mar 4 15:00 z_php_fpm_incron_reload_plugin.inc.php
    [email protected]:/etc/mysql#
     
  17. till

    till Super Moderator Staff Member ISPConfig Developer

    Seams as if only the server services plugin has been deleted, run these commands to download and install it.

    Code:
    cd /usr/local/ispconfig/server/plugins-available/
    wget https://git.ispconfig.org/ispconfig/ispconfig3/-/raw/develop/server/plugins-available/server_services_plugin.inc.php
    Then run:

    Code:
    /usr/local/ispconfig/server/server.sh
    again to check if it starts working now.
     
  18. Running the /usr/local/ispconfig/server/server.sh after the download makes my ssh connection drop.

    Checked back at the wget file, the permissions are set differently to the rest of the php files.
    [email protected]:/usr/local/ispconfig/server/plugins-available# ls -al server_services_plugin.inc.php
    -rw-r--r-- 1 root root 5501 Mar 7 02:42 server_services_plugin.inc.php


    And a thank you for your time helping here.
     

Share This Page