Urgent: Job stuck

[vilhena]

Hello.

I have a multi server Ispconfig configuration.

One master and one slave (webservers), and a DBserver. Each server has its own local ispconfig database.

Today I tried to change the PHP open_basedir configuration in one website, and that job stayed queued for like 30 minutes, after that I erased it from the master database (table sys_datalog).

And now everytime I make a change on a website, that jobs stays stuck in the job queue, which means I can't make any change!

I really really need some help here, this ispconfig is hosting more or les 60 websites.

Thanks in advance.

Kind regards,
Ricardo Vilhena

[till]

after that I erased it from the master database (table sys_datalog).

Bad idea, never edit or change any data in that table manually as it might destroy the data integrity of the system.

And now everytime I make a change on a website, that jobs stays stuck in the job queue, which means I can't make any change!

http://www.howtoforge.com/forums/showthread.php?t=58408

[vilhena]

Damn :/

Linux Distribution and version: Red Hat Enterprise Linux Server release 6.2 (Santiago)
ISPConfig Version: 3.0.4.3

I can't change the log level because the job stays stuck as the others do.

Any clue on how to solve this?

Thanks.

Regards,
Ricardo Vilhena

[till]

I can't change the log level because the job stays stuck as the others do.

Thats not related as the loglevel is not part of the jobs in the jobqueue. Please change the loglevel as described in the faq by loggin into the interface of the master server and setting it to debug for the affected slave server.

[vilhena]

I did what you said, and them I run the server.sh in the two servers (master and slave)

Here's the command output:

Master:

[root@ul-clr-srv01 log]# /usr/local/ispconfig/server/server.sh
26.09.2012-13:27 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
26.09.2012-13:27 - DEBUG - Found 3 changes, starting update process.
26.09.2012-13:27 - DEBUG - Calling function 'ssl' from plugin 'apache2_plugin' raised by event 'web_domain_update'.
26.09.2012-13:27 - DEBUG - Calling function 'update' from plugin 'apache2_plugin' raised by event 'web_domain_update'.
26.09.2012-13:27 - DEBUG - exec: chmod 755 /var/www/clients/client6/web29
26.09.2012-13:27 - DEBUG - exec: chmod 755 /var/www/clients/client6/web29/cgi-bin
26.09.2012-13:27 - DEBUG - exec: chmod 755 /var/www/clients/client6/web29/log
26.09.2012-13:27 - DEBUG - exec: chmod 755 /var/www/clients/client6/web29/ssl
26.09.2012-13:27 - DEBUG - exec: chmod 755 /var/www/clients/client6/web29/web
26.09.2012-13:27 - DEBUG - exec: chmod 777 /var/www/clients/client6/web29/tmp
26.09.2012-13:27 - DEBUG - exec: chown root:root /var/www/clients/client6/web29
26.09.2012-13:27 - DEBUG - exec: chown web29:client6 /var/www/clients/client6/web29/cgi-bin
26.09.2012-13:27 - DEBUG - exec: chown root:root /var/www/clients/client6/web29/log
26.09.2012-13:27 - DEBUG - exec: chown web29:client6 /var/www/clients/client6/web29/tmp
26.09.2012-13:27 - DEBUG - exec: chown web29:client6 /var/www/clients/client6/web29/ssl
26.09.2012-13:27 - DEBUG - exec: chown web29:client6 /var/www/clients/client6/web29/web
26.09.2012-13:27 - DEBUG - exec: chown web29:client6 /var/www/clients/client6/web29/log/error.log
26.09.2012-13:27 - DEBUG - Enable SSL for: teste-joomla.ul.pt
26.09.2012-13:27 - DEBUG - Writing the vhost file: /etc/httpd/conf/sites-available/teste-joomla.ul.pt.vhost
26.09.2012-13:27 - DEBUG - Apache status is: 1
26.09.2012-13:27 - DEBUG - Calling function 'restartHttpd' from module 'web_module'.
[Wed Sep 26 14:27:20 2012] [warn] The Alias directive in /etc/httpd/conf.d/phpmyadmin.conf at line 11 will probably never match because it overlaps an earlier Alias.
[Wed Sep 26 14:27:20 2012] [warn] The Alias directive in /etc/httpd/conf.d/phpmyadmin.conf at line 12 will probably never match because it overlaps an earlier Alias.
[Wed Sep 26 14:27:20 2012] [warn] NameVirtualHost 10.100.25.101:80 has no VirtualHosts
[Wed Sep 26 14:27:20 2012] [warn] NameVirtualHost 10.100.25.101:443 has no VirtualHosts
26.09.2012-13:27 - DEBUG - Apache online status after restart is: 1
26.09.2012-13:27 - DEBUG - Processed datalog_id 1017
26.09.2012-13:27 - DEBUG - Calling function 'update' from plugin 'apps_vhost_plugin' raised by event 'server_update'.
26.09.2012-13:27 - DEBUG - Calling function 'update' from plugin 'network_settings_plugin' raised by event 'server_update'.
26.09.2012-13:27 - DEBUG - Network configuration disabled in server settings.
26.09.2012-13:27 - DEBUG - Calling function 'update' from plugin 'postfix_server_plugin' raised by event 'server_update'.
26.09.2012-13:27 - DEBUG - Processed datalog_id 1018
26.09.2012-13:27 - DEBUG - Calling function 'update' from plugin 'apps_vhost_plugin' raised by event 'server_update'.
26.09.2012-13:27 - DEBUG - Calling function 'update' from plugin 'network_settings_plugin' raised by event 'server_update'.
26.09.2012-13:27 - DEBUG - Network configuration disabled in server settings.
26.09.2012-13:27 - DEBUG - Calling function 'update' from plugin 'postfix_server_plugin' raised by event 'server_update'.
26.09.2012-13:27 - DEBUG - Processed datalog_id 1019
26.09.2012-13:27 - DEBUG - Calling function 'restartHttpd' from module 'web_module'.
[Wed Sep 26 14:27:24 2012] [warn] The Alias directive in /etc/httpd/conf.d/phpmyadmin.conf at line 11 will probably never match because it overlaps an earlier Alias.
[Wed Sep 26 14:27:24 2012] [warn] The Alias directive in /etc/httpd/conf.d/phpmyadmin.conf at line 12 will probably never match because it overlaps an earlier Alias.
[Wed Sep 26 14:27:24 2012] [warn] NameVirtualHost 10.100.25.101:80 has no VirtualHosts
[Wed Sep 26 14:27:24 2012] [warn] NameVirtualHost 10.100.25.101:443 has no VirtualHosts
26.09.2012-13:27 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
finished.



Slave:
]# /usr/local/ispconfig/server/server.sh
26.09.2012-13:27 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
26.09.2012-13:27 - DEBUG - No Updated records found, starting only the core.
26.09.2012-13:27 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
finished.



I've looked at the job queue and it was empty, so I tried to test a change in a website configuration (changed the php-mode from Mod-php to fast-cgi), and the job is still in queue.

Thanks.

Regards,
Ricardo Vilhena

[till]

Please rerun the server.sh on the slave if there is a job in the queue. ensure that you commented out the cronjob before you do that as described in the faq.

[vilhena]

I've rerun the job on the slave, here's the output:

26.09.2012-13:40 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
26.09.2012-13:40 - DEBUG - No Updated records found, starting only the core.
which: no tw_cli in (/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin)
which: no fail2ban-client in (/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin)
which: no fail2ban in (/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin)
/sbin/iptables
/sbin/ip6tables
26.09.2012-13:40 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
finished.


The job is still queued (on the master), should I run the server.sh in the master as well?

Thanks.

Regards,
Ricardo Vilhena

[till]

should I run the server.sh in the master as well?

No.

Ok, so there are several possible resaons for this:

1) take a look at the sys_datalog table with phpmyadmin and find the job, it must be the last job, the one with the highest datalog_id. The table has a column for the server_id, does this server_id match with the server_id in the /usr/local/ispconfig/server/lib/config.inc.php file on the slave server?

2) If 1) is ok, then write down the datalog_id of the record, open the "server" table in the ispconfig db on the master server, select the record for the affected slave server and compare the number in the updated column with the datalog_id of the record, the number in updated has to be lower then the datalog_id, as only jobs were datalog_id > value in updated column get executed on the slave.

[vilhena]

The sys_datalog table in the slave is empty. In the master there are multiple entries in the sys_datalog table. I've found the last one and server_id is 1,
but the server_id in the /usr/local/ispconfig/server/lib/config.inc.php on the slave is 3, and in the master is 1.

Thanks.

Regards,
Ricardo Vilhena

[vilhena]

I've just found out that a colleague of mine, yesterday by mistake, erased the crontab of the root user in the master server, do you think that this could be the reason, in the first place, of the stuck jobs?

Thanks.

Regards,
Ricardo Vilhena