nginx logs not being compressed anymore

Discussion in 'Installation/Configuration' started by cbj4074, Feb 5, 2017.

  1. cbj4074

    cbj4074 Member

    I posted another thread about a similar problem ( https://www.howtoforge.com/communit...rking-completely-correctly.75191/#post-353989 ), but I'm not yet sure if the two issues are related, fundamentally.

    I noticed excessive disk space consumption on a particular server, and upon investigation, it was due to nginx logs not being compressed each day.

    This had always worked fine, and then on January 13, logs stopped being compressed. Now, every new daily access log is left uncompressed indefinitely.

    The directory output looks like this:

    Code:
    # ls -lah /var/www/domain.tld/log
    total 6.2G
    drwxr-xr-x 2 root  root 4.0K Feb  5 00:00 .
    drwxr-xr-x 8 root  root 4.0K Oct  1 18:05 ..
    -rw-r--r-- 1 root  root 4.5M Dec 17 00:05 20161215-access.log.gz
    -rw-r--r-- 1 root  root 4.5M Dec 18 00:04 20161216-access.log.gz
    -rw-r--r-- 1 root  root 3.0M Dec 19 00:04 20161217-access.log.gz
    -rw-r--r-- 1 root  root 2.3M Dec 20 00:04 20161218-access.log.gz
    -rw-r--r-- 1 root  root 4.7M Dec 21 00:04 20161219-access.log.gz
    -rw-r--r-- 1 root  root 4.6M Dec 22 00:04 20161220-access.log.gz
    -rw-r--r-- 1 root  root 4.8M Dec 23 00:04 20161221-access.log.gz
    -rw-r--r-- 1 root  root 5.0M Dec 24 00:04 20161222-access.log.gz
    -rw-r--r-- 1 root  root 4.5M Dec 25 00:05 20161223-access.log.gz
    -rw-r--r-- 1 root  root 3.7M Dec 26 00:05 20161224-access.log.gz
    -rw-r--r-- 1 root  root 3.1M Dec 27 00:04 20161225-access.log.gz
    -rw-r--r-- 1 root  root 4.5M Dec 28 00:04 20161226-access.log.gz
    -rw-r--r-- 1 root  root 5.5M Dec 29 00:05 20161227-access.log.gz
    -rw-r--r-- 1 root  root 5.5M Dec 30 00:05 20161228-access.log.gz
    -rw-r--r-- 1 root  root 5.2M Dec 31 00:04 20161229-access.log.gz
    -rw-r--r-- 1 root  root 4.7M Jan  1 00:04 20161230-access.log.gz
    -rw-r--r-- 1 root  root 3.1M Jan  2 00:04 20161231-access.log.gz
    -rw-r--r-- 1 root  root 2.6M Jan  3 00:04 20170101-access.log.gz
    -rw-r--r-- 1 root  root 3.9M Jan  4 00:04 20170102-access.log.gz
    -rw-r--r-- 1 root  root 5.0M Jan  5 00:06 20170103-access.log.gz
    -rw-r--r-- 1 root  root 7.6M Jan  6 00:08 20170104-access.log.gz
    -rw-r--r-- 1 root  root  21M Jan  7 00:06 20170105-access.log.gz
    -rw-r--r-- 1 root  root  22M Jan  8 00:09 20170106-access.log.gz
    -rw-r--r-- 1 root  root  21M Jan  9 00:07 20170107-access.log.gz
    -rw-r--r-- 1 root  root  20M Jan 10 00:06 20170108-access.log.gz
    -rw-r--r-- 1 root  root  24M Jan 11 00:08 20170109-access.log.gz
    -rw-r--r-- 1 root  root  24M Jan 12 00:06 20170110-access.log.gz
    -rw-r--r-- 1 root  root  24M Jan 13 00:05 20170111-access.log.gz
    -rw-r--r-- 1 www-data root 341M Jan 13 00:00 20170112-access.log
    -rw-r--r-- 1 www-data root 339M Jan 14 00:00 20170113-access.log
    -rw-r--r-- 1 www-data root 296M Jan 15 00:00 20170114-access.log
    -rw-r--r-- 1 www-data root 280M Jan 16 00:00 20170115-access.log
    -rw-r--r-- 1 www-data root 331M Jan 17 00:00 20170116-access.log
    -rw-r--r-- 1 www-data root 331M Jan 18 00:00 20170117-access.log
    -rw-r--r-- 1 www-data root 330M Jan 19 00:00 20170118-access.log
    -rw-r--r-- 1 www-data root 323M Jan 20 00:00 20170119-access.log
    -rw-r--r-- 1 www-data root 311M Jan 21 00:00 20170120-access.log
    -rw-r--r-- 1 www-data root 263M Jan 22 00:00 20170121-access.log
    -rw-r--r-- 1 www-data root 244M Jan 23 00:00 20170122-access.log
    -rw-r--r-- 1 www-data root 294M Jan 24 00:00 20170123-access.log
    -rw-r--r-- 1 www-data root 303M Jan 25 00:00 20170124-access.log
    -rw-r--r-- 1 www-data root 253M Jan 26 00:00 20170125-access.log
    -rw-r--r-- 1 www-data root 145M Jan 27 00:00 20170126-access.log
    -rw-r--r-- 1 www-data root 121M Jan 28 00:00 20170127-access.log
    -rw-r--r-- 1 www-data root  77M Jan 29 00:00 20170128-access.log
    -rw-r--r-- 1 www-data root  64M Jan 30 00:00 20170129-access.log
    -rw-r--r-- 1 www-data root 111M Jan 31 00:00 20170130-access.log
    -rw-r--r-- 1 www-data root 151M Feb  1 00:00 20170131-access.log
    -rw-r--r-- 1 www-data root 190M Feb  2 00:00 20170201-access.log
    -rw-r--r-- 1 www-data root 201M Feb  3 00:00 20170202-access.log
    -rw-r--r-- 1 www-data root 242M Feb  4 00:00 20170203-access.log
    -rw-r--r-- 1 www-data root 255M Feb  5 00:00 20170204-access.log
    -rw-r--r-- 1 www-data root 189M Feb  5 19:12 access.log
    -rw-r--r-- 1 www-data root  84M Feb  5 18:56 error.log
    -rw-r--r-- 1 root  root 996K Jan 12 00:06 error.log.1.gz
    
    What could have caused this behavior? How shall I troubleshoot it?

    I have examined "/usr/local/ispconfig/server/scripts/create_daily_nginx_access_logs.sh", but there is no mention of compression there:

    Code:
    #!/bin/bash
    PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin
    FILES=/var/log/ispconfig/httpd/*
    for f in $FILES
    do
      mv $f/access.log $f/`date --date='yesterday' "+%Y%m%d"`-access.log &> /dev/null
      touch $f/access.log &> /dev/null
    done
    pkill -USR1 -u root nginx &> /dev/null
    
    What process is normally responsible for compressing old nginx access logs with ISPConfig?
     
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    That's a cron plugin, see /usr/local/ispconfig/server/lib/classes/cron.d/
     
  3. cbj4074

    cbj4074 Member

    Thanks for pointing me in the appropriate direction here, Till.

    I ran the cron script like this:

    Code:
    php -f /usr/local/ispconfig/server/cron_debug.php -- --cronjob=200-logfiles.inc.php
    
    It looks like this "worked", but only the log file from a couple days ago was gzipped (20170204-access.log.gz).

    Here' the new directory listing data:

    Code:
    # ls -lah /var/log/ispconfig/httpd/domain.tld/
    total 6.1G
    drwxr-xr-x 2 root  root 4.0K Feb  7 00:00 .
    drwxr-xr-x 4 root  root 4.0K Oct  1 19:43 ..
    -rw-r--r-- 1 root  root 2.6M Jan  3 00:04 20170101-access.log.gz
    -rw-r--r-- 1 root  root 3.9M Jan  4 00:04 20170102-access.log.gz
    -rw-r--r-- 1 root  root 5.0M Jan  5 00:06 20170103-access.log.gz
    -rw-r--r-- 1 root  root 7.6M Jan  6 00:08 20170104-access.log.gz
    -rw-r--r-- 1 root  root  21M Jan  7 00:06 20170105-access.log.gz
    -rw-r--r-- 1 root  root  22M Jan  8 00:09 20170106-access.log.gz
    -rw-r--r-- 1 root  root  20M Jan 10 00:06 20170108-access.log.gz
    -rw-r--r-- 1 root  root  24M Jan 11 00:08 20170109-access.log.gz
    -rw-r--r-- 1 root  root  24M Jan 12 00:06 20170110-access.log.gz
    -rw-r--r-- 1 root  root  24M Jan 13 00:05 20170111-access.log.gz
    -rw-r--r-- 1 www-data root 341M Jan 13 00:00 20170112-access.log
    -rw-r--r-- 1 www-data root 339M Jan 14 00:00 20170113-access.log
    -rw-r--r-- 1 www-data root 296M Jan 15 00:00 20170114-access.log
    -rw-r--r-- 1 www-data root 280M Jan 16 00:00 20170115-access.log
    -rw-r--r-- 1 www-data root 331M Jan 17 00:00 20170116-access.log
    -rw-r--r-- 1 www-data root 331M Jan 18 00:00 20170117-access.log
    -rw-r--r-- 1 www-data root 330M Jan 19 00:00 20170118-access.log
    -rw-r--r-- 1 www-data root 323M Jan 20 00:00 20170119-access.log
    -rw-r--r-- 1 www-data root 311M Jan 21 00:00 20170120-access.log
    -rw-r--r-- 1 www-data root 263M Jan 22 00:00 20170121-access.log
    -rw-r--r-- 1 www-data root 244M Jan 23 00:00 20170122-access.log
    -rw-r--r-- 1 www-data root 294M Jan 24 00:00 20170123-access.log
    -rw-r--r-- 1 www-data root 303M Jan 25 00:00 20170124-access.log
    -rw-r--r-- 1 www-data root 253M Jan 26 00:00 20170125-access.log
    -rw-r--r-- 1 www-data root 145M Jan 27 00:00 20170126-access.log
    -rw-r--r-- 1 www-data root 121M Jan 28 00:00 20170127-access.log
    -rw-r--r-- 1 www-data root  77M Jan 29 00:00 20170128-access.log
    -rw-r--r-- 1 www-data root  64M Jan 30 00:00 20170129-access.log
    -rw-r--r-- 1 www-data root 111M Jan 31 00:00 20170130-access.log
    -rw-r--r-- 1 www-data root 151M Feb  1 00:00 20170131-access.log
    -rw-r--r-- 1 www-data root 190M Feb  2 00:00 20170201-access.log
    -rw-r--r-- 1 www-data root 201M Feb  3 00:00 20170202-access.log
    -rw-r--r-- 1 www-data root 242M Feb  4 00:00 20170203-access.log
    -rw-r--r-- 1 root  root  17M Feb  6 23:59 20170204-access.log.gz
    -rw-r--r-- 1 www-data root 242M Feb  6 00:00 20170205-access.log
    -rw-r--r-- 1 www-data root 292M Feb  7 00:00 20170206-access.log
    -rw-r--r-- 1 www-data root 1.4M Feb  7 00:07 access.log
    -rw-r--r-- 1 www-data root 2.4K Feb  7 00:06 error.log
    -rw-r--r-- 1 root  root 5.1M Feb  6 23:59 error.log.1.gz
    
    I "fixed" the existing situation with this:

    Code:
    find /var/log/ispconfig/httpd/domain.tld/ \( -name '*.log' \) -exec gzip --verbose {} \;
    
    That frees-up some 9GB of disk space on this server. I'd like to be sure that the log gzipping is functioning in a reliable way going forward.

    Should I simply check on it tomorrow and see if logs continue to be gzipped on schedule?

    And why might it have happened in the first place? This server has never been powered-off for any period of time, or subjected to any other event that would cause scheduled cron scripts not to run (at least to my knowledge).

    Thanks!
     
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    yes.

    I have no idea. If it still does not run, try this: edit the root crontab with:

    crontab -e

    insert a comment like "# test" and save, this should trigger a reload of the crontab. I've seen that cron for whatever reason lost a cronjob and triggering it like this made it being executed again.
     
  5. cbj4074

    cbj4074 Member

    Just so I know whether or not "touching" the crontab fixes the issue, I am going to wait until after midnight on this server (about two hours from now) before doing so.

    In the meantime, the file contains the following at the moment:

    Code:
    MAILTO = root@domain.tld
    * * * * * /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done
    * * * * * /usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done
    0 0 * * * /usr/local/ispconfig/server/scripts/create_daily_nginx_access_logs.sh &> /dev/null
    
    The contents look correct, I assume... we'll see what happens at midnight. If the job doesn't run, I'll touch the crontab and see what happens the following night. :)
     
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    Crontab looks fine.
     
  7. cbj4074

    cbj4074 Member

    Thanks, Till.

    Everything looks correct again today.

    Is it possible that once the log compression is "out of sync", it never recovers without manual intervention?

    I don't know what caused this condition in the first place, but it would be nice if successive cron runs would simply make the state consistent, no matter how out-of-sync it may have become. But that might asking a lot; to do so seems rather complicated!
     
    Last edited: Feb 9, 2017

Share This Page