HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials

HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials (http://www.howtoforge.com/forums/index.php)

- General (http://www.howtoforge.com/forums/forumdisplay.php?f=25)

- - Jobqueue randomly not processing (http://www.howtoforge.com/forums/showthread.php?t=51218)

Jobqueue randomly not processing

ISPconfig 3.0.3.2 on Ubuntu 10.10 64 on a vps.

Spent the weekend on a wild goose chase trying to track why changes made in the control panel weren't working.

Finally found today that some jobs happen after a minute, some take 5, 10, 20 minutes.

So, I manually ran

/usr/local/ispconfig/server/server.sh

Nothing.

Dug around inside and ran

/usr/bin/php -q /usr/local/ispconfig/server/server.php

and it updated the job queue. Made another change to test.
This time /usr/local/ispconfig/server/server.sh worked.

Next 10 times, both files, despite 2 jobs in the queue, nothing worked. 11th time, it worked.

This is all getting a bit confusing. I've seen mention of turning on debugging in /usr/local/ispconfig/server/lib/config.inc.php but I don't actually see what line that is.

Thank you.

Quote:

I've seen mention of turning on debugging in /usr/local/ispconfig/server/lib/config.inc.php but I don't actually see what line that is.

The info you read is for older ispconfig 3 versions only. In current ispconfig 3 releases, you turn on debugging under system > server config by setting the loglevel to debug.

Ispconfig 3.0.32, Ubuntu 10.10 64 bit vps, 512Mb RAM

I had to give up with cron entirely when something happened so that when I made a change, when cron kicked in the next minute, I could watch the update process climb up the "top" table until it was using all the memory, and the server ground to a halt.

Commenting out the line

Code:

* * * * * /usr/local/ispconfig/server/server.sh > /dev/null 2>> /var/log/ispconfig/cron.log

in crontab allowed my server to live, and I was happily using

sh /usr/local/ispconfig/server/server.sh

to get things moving. Yes, I might have to do it 10-15 times in a row, but at least I knew it would happen.

Now all I get is:

Code:

08.03.2011-15:35 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock

08.03.2011-15:35 - DEBUG - Found 3 changes, starting update process.

postqueue: warning: Mail system is down -- accessing queue directly

(It's always said that last line, as I don't have any mail services running. Hasn't been a problem before).

In cron.log, the last few lines are:

Code:

PHP Notice:  Undefined offset: 1 in /usr/local/ispconfig/server/cron_daily.php on line 95

Warning: Truncating oversized referrer field

Warning: Truncating oversized referrer field

PHP Warning:  Invalid argument supplied for foreach() in /usr/local/ispconfig/server/cron_daily.php on line 216

PHP Warning:  Invalid argument supplied for foreach() in /usr/local/ispconfig/server/cron_daily.php on line 330

However, this isn't much help as it doesn't have a timestamp.

All that's in the queue are a couple of site changes and a new mysql user and database.

Code:

2011-03-08 11:31        mydomain.com        Insert        web_database        

2011-03-05 10:15        mydomain.com        Update        web_domain        

2011-03-05 10:12        mydomain.com        Update        web_domain

How can I see what the commands are and what might be blocking it?

Take a look into the ispconfig database with phpmyadmin, are there any records in the sys_remoteaction database table?

Just one:

Code:

action_id        server_id        tstamp        action_type        action_param        action_state        response

4                1        1295540461        os_update                 ok

please remove that record in the database and check if that fixes your problem.

Yes! That cleared it, thank you, and I could manually run those jobs.

But then I tried activating cron again, made a simple change (turned ssl off), and watched the jobqueue. It never clears. In the processes I see:

Code:

root      5620  5618  0 17:55 ?        00:00:00 /bin/sh -c /usr/local/ispconfig/server/server.sh > /dev/null 2>> /var/log/ispconfig/cron.log

root      5621  5620  0 17:55 ?        00:00:00 /bin/sh /usr/local/ispconfig/server/server.sh

root      5622  5621  0 17:55 ?        00:00:00 /usr/bin/php -q /usr/local/ispconfig/server/server.php

and in the cron.log I see:

Code:

2011-03-08 18:00        mydomain.com        Debug        There is already an instance of server.php running. Exiting.        

2011-03-08 17:59        mydomain.com        Debug        There is already an instance of server.php running. Exiting.        

2011-03-08 17:58        mydomain.com        Debug        There is already an instance of server.php running. Exiting.        

2011-03-08 17:57        mydomain.com        Debug        There is already an instance of server.php running. Exiting.        

2011-03-08 17:56        mydomain.com        Debug        There is already an instance of server.php running. Exiting.        

2011-03-08 17:55        mydomain.com        Debug        Found 1 changes, starting update process.        

2011-03-08 17:55        mydomain.com        Debug        Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock        

2011-03-08 17:54        mydomain.com        Debug        Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock        

2011-03-08 17:54        mydomain.com        Debug        No Updated records found, starting only the core.

so once again, the only way to get the jon to run would be kill the server.php/sh processes, comment out the line in cron, delete the lock file, and manually run /bin/sh /usr/local/ispconfig/server/server.sh

Not ideal. Is there anything else I can do to fix this?

There should be no difference if the job is run by cron or manually. I guess it might be a resource problem as your system is a vps and 500MB are not that much RAM. Please post the outout of:

cat /proc/user_beancounters

Quote:

Originally Posted by till (Post 252917)

I've been asked this before, and here's the problem: I have no access to the "host" physical machine. Therefore, as an "instance" of a vps, /proc/user_beancounters doesn't actually exist.

Still, I am always only using 50% RAM.

Quote:

Therefore, as an "instance" of a vps, /proc/user_beancounters doesn't actually exist.

The file /proc/user_beancounters exists in the vps as well if openvz or virtuozzo is used as virtualisation system. So if you dont have that file, it simply means that your provider uses a different virtualisation software.

I still believe that its a resource problem. A server that is installed according to the perfect setup guide that uses normally more then 250 MB RAM.