Jobqueue randomly not processing

Discussion in 'General' started by talkingnews, Feb 1, 2011.

  1. talkingnews

    talkingnews New Member

    ISPconfig 3.0.3.2 on Ubuntu 10.10 64 on a vps.

    Spent the weekend on a wild goose chase trying to track why changes made in the control panel weren't working.

    Finally found today that some jobs happen after a minute, some take 5, 10, 20 minutes.

    So, I manually ran

    /usr/local/ispconfig/server/server.sh

    Nothing.

    Dug around inside and ran

    /usr/bin/php -q /usr/local/ispconfig/server/server.php

    and it updated the job queue. Made another change to test.
    This time /usr/local/ispconfig/server/server.sh worked.

    Next 10 times, both files, despite 2 jobs in the queue, nothing worked. 11th time, it worked.

    This is all getting a bit confusing. I've seen mention of turning on debugging in /usr/local/ispconfig/server/lib/config.inc.php but I don't actually see what line that is.

    Thank you.
     
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    The info you read is for older ispconfig 3 versions only. In current ispconfig 3 releases, you turn on debugging under system > server config by setting the loglevel to debug.
     
  3. talkingnews

    talkingnews New Member

    Ispconfig 3.0.32, Ubuntu 10.10 64 bit vps, 512Mb RAM

    I had to give up with cron entirely when something happened so that when I made a change, when cron kicked in the next minute, I could watch the update process climb up the "top" table until it was using all the memory, and the server ground to a halt.

    Commenting out the line

    Code:
    * * * * * /usr/local/ispconfig/server/server.sh > /dev/null 2>> /var/log/ispconfig/cron.log
    in crontab allowed my server to live, and I was happily using

    sh /usr/local/ispconfig/server/server.sh

    to get things moving. Yes, I might have to do it 10-15 times in a row, but at least I knew it would happen.

    Now all I get is:

    Code:
    08.03.2011-15:35 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    08.03.2011-15:35 - DEBUG - Found 3 changes, starting update process.
    postqueue: warning: Mail system is down -- accessing queue directly
    
    (It's always said that last line, as I don't have any mail services running. Hasn't been a problem before).

    In cron.log, the last few lines are:

    Code:
    PHP Notice:  Undefined offset: 1 in /usr/local/ispconfig/server/cron_daily.php on line 95
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    PHP Warning:  Invalid argument supplied for foreach() in /usr/local/ispconfig/server/cron_daily.php on line 216
    PHP Warning:  Invalid argument supplied for foreach() in /usr/local/ispconfig/server/cron_daily.php on line 330

    However, this isn't much help as it doesn't have a timestamp.

    All that's in the queue are a couple of site changes and a new mysql user and database.

    Code:
    2011-03-08 11:31	mydomain.com	Insert	web_database	
    2011-03-05 10:15	mydomain.com	Update	web_domain	
    2011-03-05 10:12	mydomain.com	Update	web_domain
    How can I see what the commands are and what might be blocking it?
     
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    Take a look into the ispconfig database with phpmyadmin, are there any records in the sys_remoteaction database table?
     
  5. talkingnews

    talkingnews New Member

    Just one:

    Code:
    action_id	server_id	tstamp	action_type	action_param	action_state	response
    4	        1	1295540461	os_update	 	ok	 
     
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    please remove that record in the database and check if that fixes your problem.
     
  7. talkingnews

    talkingnews New Member

    Yes! That cleared it, thank you, and I could manually run those jobs.

    But then I tried activating cron again, made a simple change (turned ssl off), and watched the jobqueue. It never clears. In the processes I see:

    Code:
    root      5620  5618  0 17:55 ?        00:00:00 /bin/sh -c /usr/local/ispconfig/server/server.sh > /dev/null 2>> /var/log/ispconfig/cron.log
    root      5621  5620  0 17:55 ?        00:00:00 /bin/sh /usr/local/ispconfig/server/server.sh
    root      5622  5621  0 17:55 ?        00:00:00 /usr/bin/php -q /usr/local/ispconfig/server/server.php
    and in the cron.log I see:

    Code:
    2011-03-08 18:00	mydomain.com	Debug	There is already an instance of server.php running. Exiting.	
    2011-03-08 17:59	mydomain.com	Debug	There is already an instance of server.php running. Exiting.	
    2011-03-08 17:58	mydomain.com	Debug	There is already an instance of server.php running. Exiting.	
    2011-03-08 17:57	mydomain.com	Debug	There is already an instance of server.php running. Exiting.	
    2011-03-08 17:56	mydomain.com	Debug	There is already an instance of server.php running. Exiting.	
    2011-03-08 17:55	mydomain.com	Debug	Found 1 changes, starting update process.	
    2011-03-08 17:55	mydomain.com	Debug	Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock	
    2011-03-08 17:54	mydomain.com	Debug	Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock	
    2011-03-08 17:54	mydomain.com	Debug	No Updated records found, starting only the core.	
    so once again, the only way to get the jon to run would be kill the server.php/sh processes, comment out the line in cron, delete the lock file, and manually run /bin/sh /usr/local/ispconfig/server/server.sh

    Not ideal. Is there anything else I can do to fix this?
     
  8. till

    till Super Moderator Staff Member ISPConfig Developer

    There should be no difference if the job is run by cron or manually. I guess it might be a resource problem as your system is a vps and 500MB are not that much RAM. Please post the outout of:

    cat /proc/user_beancounters
     
  9. talkingnews

    talkingnews New Member

    I've been asked this before, and here's the problem: I have no access to the "host" physical machine. Therefore, as an "instance" of a vps, /proc/user_beancounters doesn't actually exist.

    Still, I am always only using 50% RAM.
     
  10. till

    till Super Moderator Staff Member ISPConfig Developer

    The file /proc/user_beancounters exists in the vps as well if openvz or virtuozzo is used as virtualisation system. So if you dont have that file, it simply means that your provider uses a different virtualisation software.

    I still believe that its a resource problem. A server that is installed according to the perfect setup guide that uses normally more then 250 MB RAM.
     
  11. talkingnews

    talkingnews New Member

    OK, but I have turned off EVERYTHING to do with mail, spam, smtp, imap and all the bits that come with it. MySQL is tweaked to limit memory usage, so I can't understand how this server.sh uses over 200Mb of my free RAM plus another 512Mb of swap, just to, for example, configure a vhost to be non-ssl and restart apache. And when I do it manually and it works OK, the whole process barely takes 3-4 seconds and the memory bar hardly flutters at all.

    I can run a complex large sound encode using lame on an 80Mb file, but ispconfig needs more than this just to run a shell script? Seems odd.
     
  12. till

    till Super Moderator Staff Member ISPConfig Developer

    Then you can only try to debug the scripts step by steps. There must be a difference in your system when you use the shell and cron. Maybe you installed something manually which is not in the normal paths that are used for that Linux distribution which causes a script to fail. Compare your path variable with the one that is set in the server.sh script.
     
: jobqueue

Share This Page