HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials

HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials (http://www.howtoforge.com/forums/index.php)
-   General (http://www.howtoforge.com/forums/forumdisplay.php?f=25)
-   -   jobs not processing (http://www.howtoforge.com/forums/showthread.php?t=59073)

Toucan 12th October 2012 23:50

jobs not processing
 
This evening one of my servers, which happens to be a slave filled it's hard disk up by mistake causing the server to grind to halt.

In an attempt to free up the disk space I set the master to delete a number of test sites. However, because the slave was in such a bad way it didn't respond and now a number jobs are sat in the job queue waiting to process.

After freeing up disk space the slave has now returned to normal operation, however there are still the same jobs stuck in the queue.

I can now SSH into the slave without problem. Is there a way I can prompt the pair to start the jobs again?

till 13th October 2012 20:42

Debug instructions: http://www.howtoforge.com/forums/showthread.php?t=58408

Toucan 14th October 2012 00:33

Sorry, yes should have given you a bit more information.

Debian Lenny
ISP Config3.0.4.6


ok, set the master to debug, commented out the line and ran /usr/local/ispconfig/server/server.sh

and got this back at the command line:
Code:

badbuntu:~# /usr/local/ispconfig/server/server.sh
13.10.2012-22:05 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
13.10.2012-22:05 - DEBUG - No Updated records found, starting only the core.
/usr/bin/fail2ban-client
/sbin/iptables
/sbin/ip6tables
13.10.2012-22:05 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
finished.

I did the same on the slave and this time now it updated all the records and cleared the jobs. So that's great!

The only problem I have now is the monitor is still not updating.

The slaver server is still shown as critical and in red showing mysql offline, although it mysql is in fact online.


The most recent entries in the log is looking like this:
Code:

2012-10-13 22:19        badbison.co.uk        Debug        Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock       
2012-10-13 22:19        badbison.co.uk        Debug        No Updated records found, starting only the core.       
2012-10-13 22:19        badbison.co.uk        Debug        Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock       
2012-10-13 22:19        badbison.badbison.com        Debug        Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock       
2012-10-13 22:19        badbison.badbison.com        Debug        No Updated records found, starting only the core.       
2012-10-13 22:19        badbison.badbison.com        Debug        Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock       
2012-10-13 22:19        badbison.badbison.com        Debug        Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock       
2012-10-13 22:19        badbison.badbison.com        Debug        No Updated records found, starting only the core.       
2012-10-13 22:19        badbison.badbison.com        Debug        Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock       
2012-10-13 22:18        badbison.co.uk        Debug        Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock       
2012-10-13 22:18        badbison.co.uk        Debug        No Updated records found, starting only the core.       
2012-10-13 22:18        badbison.co.uk        Debug        Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock       
2012-10-13 22:16        badbison.co.uk        Debug        Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock       
2012-10-13 22:16        badbison.co.uk        Debug        No Updated records found, starting only the core.       
2012-10-13 22:16        badbison.co.uk        Debug        Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock

So the only real problem left is to get the monitor to update itself?

Thanks

Toucan 14th October 2012 18:47

[solved]
 
In due course the monitor also updated, although monitor on my set is often a little off.

till 14th October 2012 19:02

Quote:

I did the same on the slave and this time now it updated all the records and cleared the jobs. So that's great!
Then the cron daemon was not running on the slave.

The monitor updating its records at specific times, so you can not update it by running the script manually if you dont do this at the exact minute were its processing a monitor job. When you start your cron daemon again, then it will update the monitor automatically after some time.

Toucan 25th October 2012 01:35

A similar problem seems to have come back where jobs are not processing on the slave...

running the script manually, the slave complains it cannot connect to the master:
Code:

25.10.2012-00:45 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
25.10.2012-00:45 - WARNING - Unable to connect to master server.
25.10.2012-00:45 - DEBUG - No Updated records found, starting only the core.
/usr/bin/fail2ban-client
/sbin/iptables
/sbin/ip6tables
25.10.2012-00:45 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
finished.


till 25th October 2012 07:51

This means that it is not possible to connect frm slave to master mysql database with the login details for the master database from config.inc.php. Reasons can be:

Firewall
Changed hostname
Changed mysql login details
Mysql not listening on the external network interface on the master
Changed ip address

Toucan 25th October 2012 19:25

Thanks Till

I restarted the slave and then ran the script again but it didn't seem to make any difference.

I restarted the master and then suddenly the jobs seem to start processing.

I'll keep an eye on this.


All times are GMT +2. The time now is 08:23.

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.