Urgent: Job stuck

Discussion in 'General' started by vilhena, Sep 26, 2012.

  1. vilhena

    vilhena New Member

    Hello.

    I have a multi server Ispconfig configuration.

    One master and one slave (webservers), and a DBserver. Each server has its own local ispconfig database.

    Today I tried to change the PHP open_basedir configuration in one website, and that job stayed queued for like 30 minutes, after that I erased it from the master database (table sys_datalog).

    And now everytime I make a change on a website, that jobs stays stuck in the job queue, which means I can't make any change!

    I really really need some help here, this ispconfig is hosting more or les 60 websites.

    Thanks in advance.

    Kind regards,
    Ricardo Vilhena
     
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    Bad idea, never edit or change any data in that table manually as it might destroy the data integrity of the system.

    http://www.howtoforge.com/forums/showthread.php?t=58408
     
  3. vilhena

    vilhena New Member

    Damn :/

    Linux Distribution and version: Red Hat Enterprise Linux Server release 6.2 (Santiago)
    ISPConfig Version: 3.0.4.3

    I can't change the log level because the job stays stuck as the others do.

    Any clue on how to solve this?

    Thanks.

    Regards,
    Ricardo Vilhena
     
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    Thats not related as the loglevel is not part of the jobs in the jobqueue. Please change the loglevel as described in the faq by loggin into the interface of the master server and setting it to debug for the affected slave server.
     
  5. vilhena

    vilhena New Member

    I did what you said, and them I run the server.sh in the two servers (master and slave)

    Here's the command output:

    Master:

    [[email protected] log]# /usr/local/ispconfig/server/server.sh
    26.09.2012-13:27 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    26.09.2012-13:27 - DEBUG - Found 3 changes, starting update process.
    26.09.2012-13:27 - DEBUG - Calling function 'ssl' from plugin 'apache2_plugin' raised by event 'web_domain_update'.
    26.09.2012-13:27 - DEBUG - Calling function 'update' from plugin 'apache2_plugin' raised by event 'web_domain_update'.
    26.09.2012-13:27 - DEBUG - exec: chmod 755 /var/www/clients/client6/web29
    26.09.2012-13:27 - DEBUG - exec: chmod 755 /var/www/clients/client6/web29/cgi-bin
    26.09.2012-13:27 - DEBUG - exec: chmod 755 /var/www/clients/client6/web29/log
    26.09.2012-13:27 - DEBUG - exec: chmod 755 /var/www/clients/client6/web29/ssl
    26.09.2012-13:27 - DEBUG - exec: chmod 755 /var/www/clients/client6/web29/web
    26.09.2012-13:27 - DEBUG - exec: chmod 777 /var/www/clients/client6/web29/tmp
    26.09.2012-13:27 - DEBUG - exec: chown root:root /var/www/clients/client6/web29
    26.09.2012-13:27 - DEBUG - exec: chown web29:client6 /var/www/clients/client6/web29/cgi-bin
    26.09.2012-13:27 - DEBUG - exec: chown root:root /var/www/clients/client6/web29/log
    26.09.2012-13:27 - DEBUG - exec: chown web29:client6 /var/www/clients/client6/web29/tmp
    26.09.2012-13:27 - DEBUG - exec: chown web29:client6 /var/www/clients/client6/web29/ssl
    26.09.2012-13:27 - DEBUG - exec: chown web29:client6 /var/www/clients/client6/web29/web
    26.09.2012-13:27 - DEBUG - exec: chown web29:client6 /var/www/clients/client6/web29/log/error.log
    26.09.2012-13:27 - DEBUG - Enable SSL for: teste-joomla.ul.pt
    26.09.2012-13:27 - DEBUG - Writing the vhost file: /etc/httpd/conf/sites-available/teste-joomla.ul.pt.vhost
    26.09.2012-13:27 - DEBUG - Apache status is: 1
    26.09.2012-13:27 - DEBUG - Calling function 'restartHttpd' from module 'web_module'.
    [Wed Sep 26 14:27:20 2012] [warn] The Alias directive in /etc/httpd/conf.d/phpmyadmin.conf at line 11 will probably never match because it overlaps an earlier Alias.
    [Wed Sep 26 14:27:20 2012] [warn] The Alias directive in /etc/httpd/conf.d/phpmyadmin.conf at line 12 will probably never match because it overlaps an earlier Alias.
    [Wed Sep 26 14:27:20 2012] [warn] NameVirtualHost 10.100.25.101:80 has no VirtualHosts
    [Wed Sep 26 14:27:20 2012] [warn] NameVirtualHost 10.100.25.101:443 has no VirtualHosts
    26.09.2012-13:27 - DEBUG - Apache online status after restart is: 1
    26.09.2012-13:27 - DEBUG - Processed datalog_id 1017
    26.09.2012-13:27 - DEBUG - Calling function 'update' from plugin 'apps_vhost_plugin' raised by event 'server_update'.
    26.09.2012-13:27 - DEBUG - Calling function 'update' from plugin 'network_settings_plugin' raised by event 'server_update'.
    26.09.2012-13:27 - DEBUG - Network configuration disabled in server settings.
    26.09.2012-13:27 - DEBUG - Calling function 'update' from plugin 'postfix_server_plugin' raised by event 'server_update'.
    26.09.2012-13:27 - DEBUG - Processed datalog_id 1018
    26.09.2012-13:27 - DEBUG - Calling function 'update' from plugin 'apps_vhost_plugin' raised by event 'server_update'.
    26.09.2012-13:27 - DEBUG - Calling function 'update' from plugin 'network_settings_plugin' raised by event 'server_update'.
    26.09.2012-13:27 - DEBUG - Network configuration disabled in server settings.
    26.09.2012-13:27 - DEBUG - Calling function 'update' from plugin 'postfix_server_plugin' raised by event 'server_update'.
    26.09.2012-13:27 - DEBUG - Processed datalog_id 1019
    26.09.2012-13:27 - DEBUG - Calling function 'restartHttpd' from module 'web_module'.
    [Wed Sep 26 14:27:24 2012] [warn] The Alias directive in /etc/httpd/conf.d/phpmyadmin.conf at line 11 will probably never match because it overlaps an earlier Alias.
    [Wed Sep 26 14:27:24 2012] [warn] The Alias directive in /etc/httpd/conf.d/phpmyadmin.conf at line 12 will probably never match because it overlaps an earlier Alias.
    [Wed Sep 26 14:27:24 2012] [warn] NameVirtualHost 10.100.25.101:80 has no VirtualHosts
    [Wed Sep 26 14:27:24 2012] [warn] NameVirtualHost 10.100.25.101:443 has no VirtualHosts
    26.09.2012-13:27 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    finished.



    Slave:
    ]# /usr/local/ispconfig/server/server.sh
    26.09.2012-13:27 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    26.09.2012-13:27 - DEBUG - No Updated records found, starting only the core.
    26.09.2012-13:27 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    finished.



    I've looked at the job queue and it was empty, so I tried to test a change in a website configuration (changed the php-mode from Mod-php to fast-cgi), and the job is still in queue.

    Thanks.

    Regards,
    Ricardo Vilhena
     
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    Please rerun the server.sh on the slave if there is a job in the queue. ensure that you commented out the cronjob before you do that as described in the faq.
     
  7. vilhena

    vilhena New Member

    I've rerun the job on the slave, here's the output:

    26.09.2012-13:40 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    26.09.2012-13:40 - DEBUG - No Updated records found, starting only the core.
    which: no tw_cli in (/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin)
    which: no fail2ban-client in (/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin)
    which: no fail2ban in (/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin)
    /sbin/iptables
    /sbin/ip6tables
    26.09.2012-13:40 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    finished.


    The job is still queued (on the master), should I run the server.sh in the master as well?

    Thanks.

    Regards,
    Ricardo Vilhena
     
  8. till

    till Super Moderator Staff Member ISPConfig Developer

    No.

    Ok, so there are several possible resaons for this:

    1) take a look at the sys_datalog table with phpmyadmin and find the job, it must be the last job, the one with the highest datalog_id. The table has a column for the server_id, does this server_id match with the server_id in the /usr/local/ispconfig/server/lib/config.inc.php file on the slave server?

    2) If 1) is ok, then write down the datalog_id of the record, open the "server" table in the ispconfig db on the master server, select the record for the affected slave server and compare the number in the updated column with the datalog_id of the record, the number in updated has to be lower then the datalog_id, as only jobs were datalog_id > value in updated column get executed on the slave.
     
  9. vilhena

    vilhena New Member

    The sys_datalog table in the slave is empty. In the master there are multiple entries in the sys_datalog table. I've found the last one and server_id is 1,
    but the server_id in the /usr/local/ispconfig/server/lib/config.inc.php on the slave is 3, and in the master is 1.

    Thanks.

    Regards,
    Ricardo Vilhena
     
  10. vilhena

    vilhena New Member

    I've just found out that a colleague of mine, yesterday by mistake, erased the crontab of the root user in the master server, do you think that this could be the reason, in the first place, of the stuck jobs?

    Thanks.

    Regards,
    Ricardo Vilhena
     
  11. till

    till Super Moderator Staff Member ISPConfig Developer

    That is very likely as no jobs will get processed when the cronjob is not running.

    Is server 3 a mirror of server 1?
     
  12. vilhena

    vilhena New Member

    Yes.

    Thanks.

    Regards,
    Ricardo Vilhena
     
  13. vilhena

    vilhena New Member

    So, if I run the server.sh in the master, I will force the execution of the stuck job, it the job queue will be ok. Then I will activate the crontab of the root and try to do another job.

    Do you think this will solve it? How can I know if the data integrity of the system is ok?

    Thanks.

    Regards,
    Ricardo Vilhena
     
    Last edited: Sep 26, 2012
  14. vilhena

    vilhena New Member

    Well, maybe it's too early to say, but everything seems ok, I've already tested a new job and it was executed.


    Thanks.

    Regards,
    Ricardo Vilhena
     

Share This Page