One of my servers cron's keep freezing

[Xenocide]

Hi Guys,

I've been scratching my head for a while, wonder if you can help me?

We have a multi server setup that's been working fine for ages. However now one of the webserver's crons run but keep running. The lock file is still there and the php process is still running:

[root@tupac ~]# ps aufx | grep server
root     20988  0.0  0.0  61164   716 pts/0    S+   13:50   0:00                      \_ grep server
root      5052  0.0  0.0   8704   984 ?        Ss   12:50   0:00      \_ /bin/sh -c /usr/local/ispconfig/server/server.sh >> /var/log/xeno_ispconfig_log
root      5053  0.0  0.0  63844  1224 ?        S    12:50   0:00      |   \_ /bin/sh /usr/local/ispconfig/server/server.sh
root      5090  0.0  3.8 352176 157204 ?       S    12:50   0:00      |       \_ /usr/bin/php -q /usr/local/ispconfig/server/server.php

(I modified the cron slightly so it puts all the output to a file - it's the same without the modification though )

Said log is basically full up with

29.03.2011-13:45 - DEBUG - There is already an instance of server.php running. Exiting.
29.03.2011-13:46 - DEBUG - There is already an instance of server.php running. Exiting.
29.03.2011-13:47 - DEBUG - There is already an instance of server.php running. Exiting.
29.03.2011-13:48 - DEBUG - There is already an instance of server.php running. Exiting.
29.03.2011-13:49 - DEBUG - There is already an instance of server.php running. Exiting.
29.03.2011-13:50 - DEBUG - There is already an instance of server.php running. Exiting.
29.03.2011-13:51 - DEBUG - There is already an instance of server.php running. Exiting.

The last bit of juicy info in the file is:

29.03.2011-12:48 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
29.03.2011-12:48 - DEBUG - No Updated records found, starting only the core.
29.03.2011-12:48 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
finished.
29.03.2011-12:49 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
29.03.2011-12:49 - DEBUG - No Updated records found, starting only the core.
29.03.2011-12:49 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
finished.
29.03.2011-12:50 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
29.03.2011-12:50 - DEBUG - Found 2 changes, starting update process.
29.03.2011-12:51 - DEBUG - There is already an instance of server.php running. Exiting.
29.03.2011-12:52 - DEBUG - There is already an instance of server.php running. Exiting.
29.03.2011-12:53 - DEBUG - There is already an instance of server.php running. Exiting.

and before that:

29.03.2011-11:19 - DEBUG - Calling function 'insert' from plugin 'ftpuser_base_plugin' raised by event 'ftp_user_insert'.
29.03.2011-11:19 - DEBUG - Processed datalog_id 4774
29.03.2011-11:19 - DEBUG - Replicated from master: REPLACE INTO ftp_user -----REMOVED------
29.03.2011-11:19 - DEBUG - Calling function 'insert' from plugin 'ftpuser_base_plugin' raised by event 'ftp_user_insert'.
29.03.2011-11:19 - DEBUG - Processed datalog_id 4779
29.03.2011-11:19 - DEBUG - Calling function 'restartHttpd' from module 'web_module'.
29.03.2011-11:19 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
finished.
29.03.2011-11:20 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
29.03.2011-11:20 - DEBUG - No Updated records found, starting only the core.

I can't find anything which might cause the problem, I've rebooted the server but the problem continues.

If you kill all of the server instances, delete the lock file and run server.sh then it all goes through as you'd expect without any errors.

Any ideas?

Thanks in advance.

[Xenocide]

Interestingly i've just managed to make it hang when running it at the commandline:

[root@tupac web]# ps aufx | grep server
root     14635  0.0  0.0  61164   716 pts/0    S+   17:24   0:00                      \_ grep server
root     19424  0.0  0.0   8704   976 ?        Ss   15:45   0:00      \_ /bin/sh -c /usr/local/ispconfig/server/server.sh >> /var/log/xeno_ispconfig_log
root     19425  0.0  0.0  63844  1224 ?        S    15:45   0:00      |   \_ /bin/sh /usr/local/ispconfig/server/server.sh
root     19448  0.0  6.1 441776 246936 ?       S    15:45   0:00      |       \_ /usr/bin/php -q /usr/local/ispconfig/server/server.php
[root@tupac web]# kill 19424 19425 19448
[root@tupac web]# rm /usr/local/ispconfig/server/temp/.ispconfig_lock
rm: remove regular empty file `/usr/local/ispconfig/server/temp/.ispconfig_lock'? y
[root@tupac web]# /usr/local/ispconfig/server/server.sh
29.03.2011-17:25 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
29.03.2011-17:25 - DEBUG - Found 2 changes, starting update process.
PHP Warning:  date(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'Europe/London' for 'BST/1.0/DST' instead in /usr/local/ispconfig/server/mods-available/monitor_core_module.inc.php on line 850

[Xenocide]

Hi Guys,

Just thought i'd update the thread incase anyone else is having problems:

It turned out the server had a site with a mailer on it who was sending out thousands of spammy emails (should get round to setting up that..) and as such the mailq was filled up with 65 odd thousand messages. I deleted them all with postsuper -d ALL and it's right as rain again.

To find out, have it running and use

ps aufx

to see a tree of commands being ran, and see what's running under the server.php process. That should give you a good hint to see what's going wrong.