jobs not processing

[Toucan]

This evening one of my servers, which happens to be a slave filled it's hard disk up by mistake causing the server to grind to halt.

In an attempt to free up the disk space I set the master to delete a number of test sites. However, because the slave was in such a bad way it didn't respond and now a number jobs are sat in the job queue waiting to process.

After freeing up disk space the slave has now returned to normal operation, however there are still the same jobs stuck in the queue.

I can now SSH into the slave without problem. Is there a way I can prompt the pair to start the jobs again?

[till]

Debug instructions: http://www.howtoforge.com/forums/showthread.php?t=58408

[Toucan]

Sorry, yes should have given you a bit more information.

Debian Lenny
ISP Config3.0.4.6


ok, set the master to debug, commented out the line and ran /usr/local/ispconfig/server/server.sh

and got this back at the command line:

badbuntu:~# /usr/local/ispconfig/server/server.sh
13.10.2012-22:05 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
13.10.2012-22:05 - DEBUG - No Updated records found, starting only the core.
/usr/bin/fail2ban-client
/sbin/iptables
/sbin/ip6tables
13.10.2012-22:05 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
finished.

I did the same on the slave and this time now it updated all the records and cleared the jobs. So that's great!

The only problem I have now is the monitor is still not updating.

The slaver server is still shown as critical and in red showing mysql offline, although it mysql is in fact online.


The most recent entries in the log is looking like this:

2012-10-13 22:19 	badbison.co.uk 	Debug 	Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 	
2012-10-13 22:19 	badbison.co.uk 	Debug 	No Updated records found, starting only the core. 	
2012-10-13 22:19 	badbison.co.uk 	Debug 	Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 	
2012-10-13 22:19 	badbison.badbison.com 	Debug 	Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 	
2012-10-13 22:19 	badbison.badbison.com 	Debug 	No Updated records found, starting only the core. 	
2012-10-13 22:19 	badbison.badbison.com 	Debug 	Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 	
2012-10-13 22:19 	badbison.badbison.com 	Debug 	Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 	
2012-10-13 22:19 	badbison.badbison.com 	Debug 	No Updated records found, starting only the core. 	
2012-10-13 22:19 	badbison.badbison.com 	Debug 	Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 	
2012-10-13 22:18 	badbison.co.uk 	Debug 	Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 	
2012-10-13 22:18 	badbison.co.uk 	Debug 	No Updated records found, starting only the core. 	
2012-10-13 22:18 	badbison.co.uk 	Debug 	Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 	
2012-10-13 22:16 	badbison.co.uk 	Debug 	Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 	
2012-10-13 22:16 	badbison.co.uk 	Debug 	No Updated records found, starting only the core. 	
2012-10-13 22:16 	badbison.co.uk 	Debug 	Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock

So the only real problem left is to get the monitor to update itself?

Thanks

[Toucan]

In due course the monitor also updated, although monitor on my set is often a little off.

[till]

I did the same on the slave and this time now it updated all the records and cleared the jobs. So that's great!

Then the cron daemon was not running on the slave.

The monitor updating its records at specific times, so you can not update it by running the script manually if you dont do this at the exact minute were its processing a monitor job. When you start your cron daemon again, then it will update the monitor automatically after some time.

[Toucan]

A similar problem seems to have come back where jobs are not processing on the slave...

running the script manually, the slave complains it cannot connect to the master:

25.10.2012-00:45 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
25.10.2012-00:45 - WARNING - Unable to connect to master server.
25.10.2012-00:45 - DEBUG - No Updated records found, starting only the core.
/usr/bin/fail2ban-client
/sbin/iptables
/sbin/ip6tables
25.10.2012-00:45 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
finished.

[till]

This means that it is not possible to connect frm slave to master mysql database with the login details for the master database from config.inc.php. Reasons can be:

Firewall
Changed hostname
Changed mysql login details
Mysql not listening on the external network interface on the master
Changed ip address

[Toucan]

Thanks Till

I restarted the slave and then ran the script again but it didn't seem to make any difference.

I restarted the master and then suddenly the jobs seem to start processing.

I'll keep an eye on this.