Apache keeps files "/var/www/clients/.../error.log" open

Discussion in 'Server Operation' started by Tomislav Aurednik, Feb 19, 2018.

  1. Tomislav Aurednik

    Tomislav Aurednik Member HowtoForge Supporter

    Configuration
    - Ubuntu 16.04 server
    - DRBD 8.4.5
    - Ispconfig 3.1.9
    - 2 servers in cluster

    When I try to restart master server it hangs during a process (found in syslog)
    block drbd2: State change failed: Device is held open by someone

    Exactly the same behavior when I manually try to transfer resources to secondary server.
    Since this volume contains only websites (mounts to /var/www), I started to suspect apache. With a help of "lsof" I found out, that a process apache2 keeps open a few hundred files "error.log", which obviously prevent drbd from unmounting a volume.

    apache2 18473 www-data 50w REG 147,2 1477810 22151702 /var/www/clients/client25/web20/log/error.log

    Approx half of these files are zero size.

    Question
    - am I looking in the right direction at all ?
    - if yes, how to force apache not to keep these files open at all times ?
     
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    Normally, apache writes the error.log files to /var/log/ispconfig/httpd/<tmpl_var name='domain'>/error.log and /var/log/ispconfig/httpd/<tmpl_var name='domain'>/ is then mounted to /var/www/.... with a bind mount. Did you alter the vhost template so that apache writes to /var/www directly?
     
  3. Tomislav Aurednik

    Tomislav Aurednik Member HowtoForge Supporter

    Templates are managed by Ispconfig, we didn't manually override any setting. The strange thing is as follows : every instance of apache service keeps open error.log for all sites. Example for one vhost only

    vlogger 6246 root 15w REG 147,2 77655 22157285 /var/www/clients/ client1/web23/log/20180220-access.log
    apache2 2754 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 3127 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 3129 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 3702 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 3718 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 3731 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 3733 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 3735 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 3736 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 4202 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 6815 root 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 6987 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 6988 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 6989 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 7344 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 7946 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 8308 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 10039 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 10040 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 10041 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 10054 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 10055 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 10056 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 10057 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 22756 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 22757 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 23643 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 25525 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 27986 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 29736 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log
    apache2 29742 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log

    I have no idea if this is normal or not.
     
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    That's normal. I just wonder why they appear under /var/www/... and not /var/log/.... When you take a look at one of the vhost files, does the error.log path is /var/www/... or /var/log/.... there?
     
  5. Tomislav Aurednik

    Tomislav Aurednik Member HowtoForge Supporter

    Templates are managed by Ispconfig, we didn't manually override any setting. Where can I check these settings?
     
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    Take a look into the vhost config file.
     
  7. Tomislav Aurednik

    Tomislav Aurednik Member HowtoForge Supporter

    DocumentRoot /var/www/domain.tld/web
    ErrorLog /var/www/domain.tld/log/error.log
    Alias /error/ "/var/www/domain.tld/web/error/"
    where can I change this?
     
  8. till

    till Super Moderator Staff Member ISPConfig Developer

    You are probably using a custom file already as the file that ships with ISPconfig 3.1.11 has the line:

    ErrorLog /var/log/ispconfig/httpd/<tmpl_var name='domain'>/error.log

    inside. Custom config file templates are in the folder /usr/local/ispconfig/server/conf-custom/
     
  9. Tomislav Aurednik

    Tomislav Aurednik Member HowtoForge Supporter

    Unfortunately these folders are "empty" (error, index, install, mail). Only a file "empty.dir" in each folder.
    Interesting thing is that error logs are at expected location

    /var/log/ispconfig/httpd/<domain>/error.log

    only at approximately half of websites, others are found in

    /var/www/<domain>/log/error.log

    Search goes on...
     
  10. Tomislav Aurednik

    Tomislav Aurednik Member HowtoForge Supporter

    I believe I found a reason... It seems that from some point on logs are directed to correct folder.
    In "/etc/apache2/sites-enabled/<domain>.vhost" file is entry ErrorLog. For some sites correct, for most of them not.

    Last year we migrated web server to new platform with fresh install of everything. It seems to me that vhosts added later are OK. I wish there is a way to correct wrong entries using ispconfig and not manual editing of vhost file...
     
  11. till

    till Super Moderator Staff Member ISPConfig Developer

    See: Tools >Resync in ISPConfig.
     
  12. Tomislav Aurednik

    Tomislav Aurednik Member HowtoForge Supporter

    You are a lifesaver :)

    Thanks, I simply forgot about this option.
     
  13. Tomislav Aurednik

    Tomislav Aurednik Member HowtoForge Supporter

    Unfortuntely there's no end of strange situations... in most of the sites everything is OK, but there's a problem with some of them. There are "duplicate" log files in two locations (please see appacteh picture)

    [​IMG]
    Of course if I delete a file on either side, it disappears on both. And I can't find any symbolic link. Can somebody explain what is happening ?
     
  14. till

    till Super Moderator Staff Member ISPConfig Developer

    That's exactöy as it should be. The log dir in /var/log/ispconfig/httpd/domain.tld is mounted to /var/www/.... with a bind mount to allow read-only access by the web user to his log files.
     
  15. Tomislav Aurednik

    Tomislav Aurednik Member HowtoForge Supporter

    Thanks for clearing this out. It seems that original sin was attempt to store logs in /var/www/_website_/log folders (to sync everything using drbd) and not the location originally intended by ispconfig. But there's always "but"...
    which script / process is doing bind mount ? Because for most of the sites now logs exist only in
    /var/log/ispconfig/httpd/_website_/, but no change is reflected in /var/www/_website_/log

    Or to ask in a different way : what I have to do to normalize/initialize situation ?

    Regards
     
  16. till

    till Super Moderator Staff Member ISPConfig Developer

    Logs get stored by ISPConfig to /var/log/ispconfig/httpd/_website_/ and not /var/www/_website_/log as logs shall not be stored in the web folder. The folder /var/www/_website_/log is just mounted with a BIND mount in /etc/fstab and this bind mount is added by the apache or nginx plugin at the time the website gest created. It works like this since ISPConfig 3 exists.
     
  17. Tomislav Aurednik

    Tomislav Aurednik Member HowtoForge Supporter

    Things are getting clearer now. In /etc/fstab I've found entries for bind mounting log folders. But the problem is DRBD... in a moment when entries in fstab should be mounted, the source doesn't exist yet. DRBD mounts partition later on.
    Thanks for clearing things out. Will solve it with some script - at least temporarily, until I find permanent solution.
     
    till likes this.

Share This Page