Lockfile not being deleted

Discussion in 'General' started by outeredge, Nov 17, 2010.

  1. outeredge

    outeredge New Member

    Hello All,

    I wonder if someone might be able to help. It seems that the lockfile on the web server within our multi-server setup is not being removed. In the ISPConfig logs we see:

    2010-11-17 17:11 xxx.xxx.com Debug There is already a lockfile set. Waiting another 10 seconds...
    2010-11-17 17:11 xxx.xxx.com Debug There is already a lockfile set. Waiting another 10 seconds...
    2010-11-17 17:10 xxx.xxx.com Debug No Updated records found, starting only the core.
    2010-11-17 17:10 xxx.xxx.com Debug Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock

    Each time we delete the lock file, the queue is able to run, but before long it's back in there stopping future tasks.

    Thanks

    David
     
  2. falko

    falko Super Moderator ISPConfig Developer

    Please delete the lock file and run
    Code:
    /usr/local/ispconfig/server/server.sh
    manually. Do you get any errors?
     
  3. wiss

    wiss Member

    I've got the same problem.

    I was removed lock file and run server.php manually; this is cron.log of ispconfig:

    Code:
    Resolving www.ispconfig.org... 78.46.59.59
    Connecting to www.ispconfig.org|78.46.59.59|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 2976564 (2,8M) [application/x-gzip]
    Saving to: `ISPConfig-3.0.3.tar.gz'
    
         0K .......... .......... .......... .......... ..........  1%  451K 6s
        50K .......... .......... .......... .......... ..........  3% 1,28M 4s
       100K .......... .......... .......... .......... ..........  5% 1,33M 3s
       150K .......... .......... .......... .......... ..........  6%  843K 3s
       200K .......... .......... .......... .......... ..........  8%  101M 3s
       250K .......... .......... .......... .......... .......... 10% 4,57M 2s
       300K .......... .......... .......... .......... .......... 12% 1,67M 2s
       350K .......... .......... .......... .......... .......... 13% 6,55M 2s
       400K .......... .......... .......... .......... .......... 15% 1,80M 2s
    ...
     2900K ......                                                100% 12,6M=0,6s
    
    2010-11-19 23:36:15 (5,04 MB/s) - `ISPConfig-3.0.3.tar.gz' saved [2976564/2976564]
    
    PHP Deprecated:  Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/imagick.ini on line 1 in Unknown on line 0
    PHP Deprecated:  Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/imap.ini on line 1 in Unknown on line 0
    PHP Deprecated:  Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/mcrypt.ini on line 1 in Unknown on line 0
    hostname: Name or service not known
    PHP Notice:  Undefined offset: 0 in /tmp/ispconfig3_install/install/autoupdate.php on line 89
    PHP Notice:  Undefined variable: hosts in /tmp/ispconfig3_install/install/lib/installer_base.lib.php on line 323
    hostname: Name or service not known
      The value of variable $myhostname is "", but should have been
      a fully qualified domain name; perhaps uname(3) did not provide such.
      You must explicitly assign a FQDN of this host to variable $myhostname
      in /etc/amavis/conf.d/05-node_id, or fix what uname(3) provides as a host's 
      network name!
    PHP Deprecated:  Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/imagick.ini on line 1 in Unknown on line 0
    PHP Deprecated:  Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/imagick.ini on line 1 in Unknown on line 0
    PHP Deprecated:  Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/imap.ini on line 1 in Unknown on line 0
    PHP Deprecated:  Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/mcrypt.ini on line 1 in Unknown on line 0
    PHP Fatal error:  Out of memory (allocated 1512046592) (tried to allocate 261900 bytes) in /usr/local/ispconfig/server/lib/app.inc.php on line 69
    ...
    19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away
    19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away
    19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away
    19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away
    19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away
    19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away
    19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away
    19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away
    19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away
    19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away
    19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away
    19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away
    19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away
    19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away
    ...
    19.11.2010-23:51 - WARNING - DB::connect()-> mysql_select_db MySQL server has gone away
    PHP Fatal error:  Out of memory (allocated 1497104384) (tried to allocate 261900 bytes) in /usr/local/ispconfig/server/lib/app.inc.php on line 69
    
    
    and this is the event log monitor in Ispconfig:

    Code:
    2010-11-19 23:54 	ns1.pcrednet.net 	Debug 	There is already an instance of server.php running. Exiting. 	
    2010-11-19 23:53 	ns1.pcrednet.net 	Debug 	There is already an instance of server.php running. Exiting. 	
    2010-11-19 23:52 	ns1.pcrednet.net 	Debug 	There is already an instance of server.php running. Exiting. 	
    2010-11-19 23:51 	ns1.pcrednet.net 	Debug 	There is already an instance of server.php running. Exiting. 	
    2010-11-19 23:50 	ns1.pcrednet.net 	Debug 	There is already an instance of server.php running. Exiting. 	
    2010-11-19 23:49 	ns1.pcrednet.net 	Debug 	Found 36 changes, starting update process. 	
    2010-11-19 23:49 	ns1.pcrednet.net 	Debug 	Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 	
    2010-11-19 23:49 	ns1.pcrednet.net 	Debug 	There is already an instance of server.php running. Exiting. 	
    2010-11-19 23:48 	ns1.pcrednet.net 	Debug 	There is already an instance of server.php running. Exiting. 	
    2010-11-19 22:35 	ns1.pcrednet.net 	Debug 	Found 34 changes, starting update process. 	
    2010-11-19 22:35 	ns1.pcrednet.net 	Debug 	Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 	
    2010-11-19 22:34 	ns1.pcrednet.net 	Debug 	There is already a lockfile set. Waiting another 10 seconds... 	
    2010-11-19 22:34 	ns1.pcrednet.net 	Debug 	There is already a lockfile set. Waiting another 10 seconds... 	
    2010-11-19 22:34 	ns1.pcrednet.net 	Debug 	There is already a lockfile set. Waiting another 10 seconds... 	
    2010-11-19 22:34 	ns1.pcrednet.net 	Debug 	There is already a lockfile set. Waiting another 10 seconds... 	
    
    Job queue task is waiting for 36hrs...

    Thx in advance.
     
  4. wiss

    wiss Member

    Now; i delete lock file and run manually server.sh.

    Code:
    /usr/local/ispconfig/server# sh server.sh 
    PHP Deprecated:  Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/imagick.ini on line 1 in Unknown on line 0
    PHP Deprecated:  Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/imap.ini on line 1 in Unknown on line 0
    PHP Deprecated:  Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/mcrypt.ini on line 1 in Unknown on line 0
    PHP Warning:  mysql_connect(): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) in /usr/local/ispconfig/server/lib/classes/db_mysql.inc.php on line 78
    PHP Warning:  mysql_errno() expects parameter 1 to be resource, boolean given in /usr/local/ispconfig/server/lib/classes/db_mysql.inc.php on line 63
    PHP Warning:  mysql_error() expects parameter 1 to be resource, boolean given in /usr/local/ispconfig/server/lib/classes/db_mysql.inc.php on line 64
    
    Can't connect to mysql??¿?¿?¿¿?
     
  5. falko

    falko Super Moderator ISPConfig Developer

    IT seems as if MySQL isn't running. What's the output of
    Code:
    netstat -tap
    ?
     
  6. wiss

    wiss Member

    netstat -tap output:


    Code:
    /usr/local/ispconfig/server# netstat -tap
    Active Internet connections (servers and established)
    Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
    tcp        0      0 *:postgresql            *:*                     LISTEN      363/postgres    
    tcp        0      0 localhost.localdoma:953 *:*                     LISTEN      9828/named      
    tcp        0      0 *:smtp                  *:*                     LISTEN      8746/master     
    tcp        0      0 *:https                 *:*                     LISTEN      9791/apache2    
    tcp        0      0 *:2812                  *:*                     LISTEN      13824/monit     
    tcp        0      0 *:2082                  *:*                     LISTEN      9791/apache2           
    tcp        0      0 localhost.localdo:10024 *:*                     LISTEN      8973/amavisd (maste
    tcp        0      0 localhost.localdo:10025 *:*                     LISTEN      8746/master     
    tcp        0      0 *:mysql                 *:*                     LISTEN      9870/mysqld     
    tcp        0      0 localhost.localdo:spamd *:*                     LISTEN      558/spamd.pid   
    tcp        0      0 *:www                   *:*                     LISTEN      9791/apache2    
    tcp        0      0 *:tproxy                *:*                     LISTEN      9791/apache2    
    tcp        0      0 localhost.locald:domain *:*                     LISTEN      9828/named      
    tcp        0      0 *:ftp                   *:*                     LISTEN      9809/pure-ftpd (SER
    tcp        0      0 *:munin                 *:*                     LISTEN      1118/munin-node 
    tcp        0      0 *:ssh                   *:*                     LISTEN      131/sshd        
    tcp        0      0 localhost.localdo:10024 localhost.localdo:36009 TIME_WAIT   -               
    tcp        0      0 localhost.localdo:39708 localhost.localdoma:ssh TIME_WAIT   -               
    tcp        0      0 localhost.localdo:59088 localhost.localdoma:953 TIME_WAIT   -               
    tcp        0      0 localhost.lo:postgresql localhost.localdo:51791 ESTABLISHED 14664/postgres: buc
    tcp        0      0 localhost.localdoma:www localhost.localdo:46782 TIME_WAIT   -               
    tcp        0      0 localhost.localdo:49808 localhost.localdo:pop3s TIME_WAIT   -               
    tcp        0      0 localhost.lo:postgresql localhost.localdo:51771 ESTABLISHED 14634/postgres: buc
    tcp        0      0 localhost.lo:postgresql localhost.localdo:51793 ESTABLISHED 14665/postgres: adm
    tcp        0      0 localhost.localdo:51789 localhost.lo:postgresql ESTABLISHED 14661/Bucardo Contr
    tcp        0      0 localhost.localdo:51771 localhost.lo:postgresql ESTABLISHED 14633/Bucardo Maste
    tcp        0      0 localhost.localdom:smtp localhost.localdo:45235 TIME_WAIT   -               
    tcp        0      0 localhost.localdo:48647 localhost.lo:postgresql TIME_WAIT   -               
    tcp        0      0 localhost.localdo:mysql localhost.localdo:48809 TIME_WAIT   -               
    tcp        0      0 localhost.localdo:51793 localhost.lo:postgresql ESTABLISHED 14663/Bucardo Kid. 
    tcp        0      0 localhost.lo:postgresql localhost.localdo:51789 ESTABLISHED 14662/postgres: buc
    tcp        0      0 localhost.localdo:51791 localhost.lo:postgresql ESTABLISHED 14663/Bucardo Kid. 
    tcp        0      0 localhost.localdo:48648 localhost.lo:postgresql TIME_WAIT   -               
    tcp        0      0 localhost.localdo:51774 localhost.lo:postgresql ESTABLISHED 14633/Bucardo Maste
    tcp        0      0 localhost.lo:postgresql localhost.localdo:51774 ESTABLISHED 14649/postgres: adm
    tcp6       0      0 [::]:postgresql         [::]:*                  LISTEN      363/postgres    
    tcp6       0      0 ::1%137029864:953       [::]:*                  LISTEN      9828/named      
    tcp6       0      0 [::]:imaps              [::]:*                  LISTEN      9733/couriertcpd
    tcp6       0      0 [::]:pop3s              [::]:*                  LISTEN      9761/couriertcpd
    tcp6       0      0 [::]:pop3               [::]:*                  LISTEN      9744/couriertcpd
    tcp6       0      0 [::]:imap2              [::]:*                  LISTEN      9716/couriertcpd
    tcp6       0      0 [::]:domain             [::]:*                  LISTEN      9828/named      
    tcp6       0      0 [::]:ftp                [::]:*                  LISTEN      9809/pure-ftpd (SER
    tcp6       0      0 [::]:ssh                [::]:*                  LISTEN      131/sshd        
    
     
    Last edited: Nov 22, 2010
  7. wiss

    wiss Member

    every time that run server.sh (or server.php), the system try to update;

    How do i change "the flag" to cancel this update?.

    Thx.
     
  8. outeredge

    outeredge New Member

    Hello Falko,

    Back to my original problem, I don't have any errors running server.sh after manually deleting the lock file:

    22.11.2010-11:24 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    22.11.2010-11:24 - DEBUG - No Updated records found, starting only the core.
    22.11.2010-11:24 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    finished.

    It seems that after a while a lock is getting stuck in there and not getting deleted. Is there any way we can debug the cause of this?

    Thanks

    David
     
  9. outeredge

    outeredge New Member

    Trying to find the cause, it seems to happen when there's nothing in the queue for some random reason.

    web-01:/usr/local/ispconfig/server/temp# /usr/local/ispconfig/server/server.sh
    23.11.2010-15:09 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    23.11.2010-15:09 - DEBUG - No Updated records found, starting only the core.
    23.11.2010-15:09 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    finished.
    web-01:/usr/local/ispconfig/server/temp# /usr/local/ispconfig/server/server.sh
    23.11.2010-15:09 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    23.11.2010-15:09 - DEBUG - No Updated records found, starting only the core.
    23.11.2010-15:09 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    finished.
    web-01:/usr/local/ispconfig/server/temp# /usr/local/ispconfig/server/server.sh
    23.11.2010-15:09 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    23.11.2010-15:09 - DEBUG - No Updated records found, starting only the core.
    23.11.2010-15:09 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    finished.
    web-01:/usr/local/ispconfig/server/temp# /usr/local/ispconfig/server/server.sh
    23.11.2010-15:09 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    23.11.2010-15:09 - DEBUG - No Updated records found, starting only the core.
    23.11.2010-15:09 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    finished.
    web-01:/usr/local/ispconfig/server/temp# /usr/local/ispconfig/server/server.sh
    23.11.2010-15:09 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    23.11.2010-15:09 - DEBUG - No Updated records found, starting only the core.
    23.11.2010-15:09 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    finished.
    web-01:/usr/local/ispconfig/server/temp# /usr/local/ispconfig/server/server.sh
    23.11.2010-15:10 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    23.11.2010-15:10 - DEBUG - No Updated records found, starting only the core.
    web-01:/usr/local/ispconfig/server/temp#

    Why does the script not remove the lock despite nothing being in the queue and no errors being shown? Is there a way to debug this further, appreciate your advice.
     
  10. wiss

    wiss Member

  11. wiss

    wiss Member

    what shows ispconfig cron log¿
     
  12. till

    till Super Moderator Staff Member ISPConfig Developer

    check with phpmyadmin if there are any records in the table sys_remoteaction and delete them manually with phpmyadmin.
     
  13. outeredge

    outeredge New Member

    Thanks Till,

    I found the a record for 'ispc_update' with a status of 'pending' in this table so have deleted.

    Will report back if any further issues.
     
  14. outeredge

    outeredge New Member

    Unfortunately the problem still seems to be occuring.

    Although it does look like jobs are eventually being put through, they seem to hang in the queue for a long time whilst the error messages regarding the lockfile persist.
     
  15. outeredge

    outeredge New Member

    We have also noticed that this server is not being monitored correctly, i.e. there are no stats for the RAID State, Load, etc, etc.

    Any ideas?
     
  16. outeredge

    outeredge New Member

    After a lot of digging and debugging of the scripts, it seems that the @ftp_connect($host, $port) line in /usr/local/ispconfig/server/mods-core/monitor_core_module.inc.php is crashing out the script but not throwing a PHP error (because of the @), in the case that the PHP-FTP module is not enabled.

    Can I highly recommend that a requirements page is put up on the ISPConfig site, specifically with reference to the PHP modules required by the software?
     
  17. till

    till Super Moderator Staff Member ISPConfig Developer

    The perfect setup guide is that reference. Everything that is installed there is required by ispconfig.
     
  18. outeredge

    outeredge New Member

    Thanks Till,

    That's a fair point, but some of us don't want to follow the guides in their entirity as we may have custom requirements (for example we are building our own PHP rather than using binaries).

    Although we've based our install on the Perfect Install Guides, when you look at the PHP section, there isn't explicity mention of the php ftp module so it's hard to know to install this in the first place:

    apt-get install apache2 apache2.2-common apache2-doc apache2-mpm-prefork apache2-utils libexpat1 ssl-cert libapache2-mod-php5 php5 php5-common php5-gd php5-mysql php5-imap phpmyadmin php5-cli php5-cgi libapache2-mod-fcgid apache2-suexec php-pear php-auth php5-mcrypt mcrypt php5-imagick imagemagick libapache2-mod-suphp libapache2-mod-ruby

    Maybe a community Wiki might be a good idea too so we (the community) can help build up the documentation for you guys?
     
  19. falko

    falko Super Moderator ISPConfig Developer

    We've tried that so many times already, but hardly anyone is willing or has the time to contribute... :(
     
  20. outeredge

    outeredge New Member

    Thanks Falko,

    Perhaps i'll set one up in the future, if I have time :eek:

    By the way, we now have Litespeed working perfectly alongside ISPConfig 3! A perfect combination :)
     

Share This Page