Jobqueue randomly not processing

[talkingnews]

ISPconfig 3.0.3.2 on Ubuntu 10.10 64 on a vps.

Spent the weekend on a wild goose chase trying to track why changes made in the control panel weren't working.

Finally found today that some jobs happen after a minute, some take 5, 10, 20 minutes.

So, I manually ran

/usr/local/ispconfig/server/server.sh

Nothing.

Dug around inside and ran

/usr/bin/php -q /usr/local/ispconfig/server/server.php

and it updated the job queue. Made another change to test.
This time /usr/local/ispconfig/server/server.sh worked.

Next 10 times, both files, despite 2 jobs in the queue, nothing worked. 11th time, it worked.

This is all getting a bit confusing. I've seen mention of turning on debugging in /usr/local/ispconfig/server/lib/config.inc.php but I don't actually see what line that is.

Thank you.

[till]

I've seen mention of turning on debugging in /usr/local/ispconfig/server/lib/config.inc.php but I don't actually see what line that is.

The info you read is for older ispconfig 3 versions only. In current ispconfig 3 releases, you turn on debugging under system > server config by setting the loglevel to debug.

[talkingnews]

Ispconfig 3.0.32, Ubuntu 10.10 64 bit vps, 512Mb RAM

I had to give up with cron entirely when something happened so that when I made a change, when cron kicked in the next minute, I could watch the update process climb up the "top" table until it was using all the memory, and the server ground to a halt.

Commenting out the line

* * * * * /usr/local/ispconfig/server/server.sh > /dev/null 2>> /var/log/ispconfig/cron.log

in crontab allowed my server to live, and I was happily using

sh /usr/local/ispconfig/server/server.sh

to get things moving. Yes, I might have to do it 10-15 times in a row, but at least I knew it would happen.

Now all I get is:

08.03.2011-15:35 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
08.03.2011-15:35 - DEBUG - Found 3 changes, starting update process.
postqueue: warning: Mail system is down -- accessing queue directly

(It's always said that last line, as I don't have any mail services running. Hasn't been a problem before).

In cron.log, the last few lines are:

PHP Notice:  Undefined offset: 1 in /usr/local/ispconfig/server/cron_daily.php on line 95
Warning: Truncating oversized referrer field
Warning: Truncating oversized referrer field
PHP Warning:  Invalid argument supplied for foreach() in /usr/local/ispconfig/server/cron_daily.php on line 216
PHP Warning:  Invalid argument supplied for foreach() in /usr/local/ispconfig/server/cron_daily.php on line 330

However, this isn't much help as it doesn't have a timestamp.

All that's in the queue are a couple of site changes and a new mysql user and database.

2011-03-08 11:31	mydomain.com	Insert	web_database	
2011-03-05 10:15	mydomain.com	Update	web_domain	
2011-03-05 10:12	mydomain.com	Update	web_domain

How can I see what the commands are and what might be blocking it?

[till]

Take a look into the ispconfig database with phpmyadmin, are there any records in the sys_remoteaction database table?

[talkingnews]

Just one:

action_id	server_id	tstamp	action_type	action_param	action_state	response
4	        1	1295540461	os_update	 	ok

[till]

please remove that record in the database and check if that fixes your problem.

[talkingnews]

Yes! That cleared it, thank you, and I could manually run those jobs.

But then I tried activating cron again, made a simple change (turned ssl off), and watched the jobqueue. It never clears. In the processes I see:

root      5620  5618  0 17:55 ?        00:00:00 /bin/sh -c /usr/local/ispconfig/server/server.sh > /dev/null 2>> /var/log/ispconfig/cron.log
root      5621  5620  0 17:55 ?        00:00:00 /bin/sh /usr/local/ispconfig/server/server.sh
root      5622  5621  0 17:55 ?        00:00:00 /usr/bin/php -q /usr/local/ispconfig/server/server.php

and in the cron.log I see:

2011-03-08 18:00	mydomain.com	Debug	There is already an instance of server.php running. Exiting.	
2011-03-08 17:59	mydomain.com	Debug	There is already an instance of server.php running. Exiting.	
2011-03-08 17:58	mydomain.com	Debug	There is already an instance of server.php running. Exiting.	
2011-03-08 17:57	mydomain.com	Debug	There is already an instance of server.php running. Exiting.	
2011-03-08 17:56	mydomain.com	Debug	There is already an instance of server.php running. Exiting.	
2011-03-08 17:55	mydomain.com	Debug	Found 1 changes, starting update process.	
2011-03-08 17:55	mydomain.com	Debug	Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock	
2011-03-08 17:54	mydomain.com	Debug	Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock	
2011-03-08 17:54	mydomain.com	Debug	No Updated records found, starting only the core.

so once again, the only way to get the jon to run would be kill the server.php/sh processes, comment out the line in cron, delete the lock file, and manually run /bin/sh /usr/local/ispconfig/server/server.sh

Not ideal. Is there anything else I can do to fix this?

[till]

There should be no difference if the job is run by cron or manually. I guess it might be a resource problem as your system is a vps and 500MB are not that much RAM. Please post the outout of:

cat /proc/user_beancounters

[talkingnews]

Originally Posted by till

There should be no difference if the job is run by cron or manually. I guess it might be a resource problem as your system is a vps and 500MB are not that much RAM. Please post the outout of:

cat /proc/user_beancounters

I've been asked this before, and here's the problem: I have no access to the "host" physical machine. Therefore, as an "instance" of a vps, /proc/user_beancounters doesn't actually exist.

Still, I am always only using 50% RAM.

[till]

Therefore, as an "instance" of a vps, /proc/user_beancounters doesn't actually exist.

The file /proc/user_beancounters exists in the vps as well if openvz or virtuozzo is used as virtualisation system. So if you dont have that file, it simply means that your provider uses a different virtualisation software.

I still believe that its a resource problem. A server that is installed according to the perfect setup guide that uses normally more then 250 MB RAM.