HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials

HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials (http://www.howtoforge.com/forums/index.php)
-   General (http://www.howtoforge.com/forums/forumdisplay.php?f=25)
-   -   One of my servers cron's keep freezing (http://www.howtoforge.com/forums/showthread.php?t=52074)

Xenocide 29th March 2011 14:55

One of my servers cron's keep freezing
 
Hi Guys,

I've been scratching my head for a while, wonder if you can help me?

We have a multi server setup that's been working fine for ages. However now one of the webserver's crons run but keep running. The lock file is still there and the php process is still running:

Code:

[root@tupac ~]# ps aufx | grep server
root    20988  0.0  0.0  61164  716 pts/0    S+  13:50  0:00                      \_ grep server
root      5052  0.0  0.0  8704  984 ?        Ss  12:50  0:00      \_ /bin/sh -c /usr/local/ispconfig/server/server.sh >> /var/log/xeno_ispconfig_log
root      5053  0.0  0.0  63844  1224 ?        S    12:50  0:00      |  \_ /bin/sh /usr/local/ispconfig/server/server.sh
root      5090  0.0  3.8 352176 157204 ?      S    12:50  0:00      |      \_ /usr/bin/php -q /usr/local/ispconfig/server/server.php

(I modified the cron slightly so it puts all the output to a file - it's the same without the modification though :) )

Said log is basically full up with
Quote:

29.03.2011-13:45 - DEBUG - There is already an instance of server.php running. Exiting.
29.03.2011-13:46 - DEBUG - There is already an instance of server.php running. Exiting.
29.03.2011-13:47 - DEBUG - There is already an instance of server.php running. Exiting.
29.03.2011-13:48 - DEBUG - There is already an instance of server.php running. Exiting.
29.03.2011-13:49 - DEBUG - There is already an instance of server.php running. Exiting.
29.03.2011-13:50 - DEBUG - There is already an instance of server.php running. Exiting.
29.03.2011-13:51 - DEBUG - There is already an instance of server.php running. Exiting.
The last bit of juicy info in the file is:

Code:

29.03.2011-12:48 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
29.03.2011-12:48 - DEBUG - No Updated records found, starting only the core.
29.03.2011-12:48 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
finished.
29.03.2011-12:49 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
29.03.2011-12:49 - DEBUG - No Updated records found, starting only the core.
29.03.2011-12:49 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
finished.
29.03.2011-12:50 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
29.03.2011-12:50 - DEBUG - Found 2 changes, starting update process.
29.03.2011-12:51 - DEBUG - There is already an instance of server.php running. Exiting.
29.03.2011-12:52 - DEBUG - There is already an instance of server.php running. Exiting.
29.03.2011-12:53 - DEBUG - There is already an instance of server.php running. Exiting.

and before that:

Code:

29.03.2011-11:19 - DEBUG - Calling function 'insert' from plugin 'ftpuser_base_plugin' raised by event 'ftp_user_insert'.
29.03.2011-11:19 - DEBUG - Processed datalog_id 4774
29.03.2011-11:19 - DEBUG - Replicated from master: REPLACE INTO ftp_user -----REMOVED------
29.03.2011-11:19 - DEBUG - Calling function 'insert' from plugin 'ftpuser_base_plugin' raised by event 'ftp_user_insert'.
29.03.2011-11:19 - DEBUG - Processed datalog_id 4779
29.03.2011-11:19 - DEBUG - Calling function 'restartHttpd' from module 'web_module'.
29.03.2011-11:19 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
finished.
29.03.2011-11:20 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
29.03.2011-11:20 - DEBUG - No Updated records found, starting only the core.

I can't find anything which might cause the problem, I've rebooted the server but the problem continues.

If you kill all of the server instances, delete the lock file and run server.sh then it all goes through as you'd expect without any errors.

Any ideas? :)

Thanks in advance.

Xenocide 29th March 2011 18:28

Interestingly i've just managed to make it hang when running it at the commandline:

Code:

[root@tupac web]# ps aufx | grep server
root    14635  0.0  0.0  61164  716 pts/0    S+  17:24  0:00                      \_ grep server
root    19424  0.0  0.0  8704  976 ?        Ss  15:45  0:00      \_ /bin/sh -c /usr/local/ispconfig/server/server.sh >> /var/log/xeno_ispconfig_log
root    19425  0.0  0.0  63844  1224 ?        S    15:45  0:00      |  \_ /bin/sh /usr/local/ispconfig/server/server.sh
root    19448  0.0  6.1 441776 246936 ?      S    15:45  0:00      |      \_ /usr/bin/php -q /usr/local/ispconfig/server/server.php
[root@tupac web]# kill 19424 19425 19448
[root@tupac web]# rm /usr/local/ispconfig/server/temp/.ispconfig_lock
rm: remove regular empty file `/usr/local/ispconfig/server/temp/.ispconfig_lock'? y
[root@tupac web]# /usr/local/ispconfig/server/server.sh
29.03.2011-17:25 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
29.03.2011-17:25 - DEBUG - Found 2 changes, starting update process.
PHP Warning:  date(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'Europe/London' for 'BST/1.0/DST' instead in /usr/local/ispconfig/server/mods-available/monitor_core_module.inc.php on line 850


Xenocide 30th March 2011 14:19

Hi Guys,

Just thought i'd update the thread incase anyone else is having problems:

It turned out the server had a site with a mailer on it who was sending out thousands of spammy emails (should get round to setting up that..) and as such the mailq was filled up with 65 odd thousand messages. I deleted them all with postsuper -d ALL and it's right as rain again.

To find out, have it running and use
Code:

ps aufx
to see a tree of commands being ran, and see what's running under the server.php process. That should give you a good hint to see what's going wrong.


All times are GMT +2. The time now is 08:22.

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.