High Disk IO

Discussion in 'ISPConfig 3 Priority Support' started by olimortimer, May 1, 2014.

  1. olimortimer

    olimortimer Member

    Since the latest update of ISPConfig (not sure if that's connected), I've been finding that at about 6:40 each morning, the server starts using 100% CPU and ~100% Disk IO - this is reported through New Relic, which runs an agent on the server.

    Now I know a little about servers, but not a great deal...

    Attached are a couple of screenshots of top, htop, iostat and iotop. These screenshots are after a reboot - notice how it doesn't show apache or mysql, as the server didn't start the processes for some reason (maybe too busy?). Suppose this is a good thing, as it rules out an issue with apache and mysql etc.

    One thing that does stand out, is 'quotacheck -augm' using 99.99% IO when viewed in iotop. I'm not sure if this is normal or not though.

    My first thought was maybe the HDD is going bad, but this is a VPS so I would hope that my provider would catch that if it was an issue, however I won't rule it out.

    I would really appreciate if someone was able to offer some advise on pinpointing exactly what is going on, and how to sort it out.

    Many thanks in advance!
     

    Attached Files:

  2. till

    till Super Moderator Staff Member ISPConfig Developer

    ispconfig is not running a cronjob at that time and ispconfig is not running the quotacheck command at all, so I guess your problem is not related to ispconfig. You should check the system cronjob to find out which one is running the quotacheck daily.

    As your server is a vps, your problem might be another vm on the same server, so it can be that the issue is not related to your server at all. If another server on the same hardware runs a io intensive task at that time, then your io wait times will rise as the server hardware has not a unlimited io bandwidth, the bandwidht is shared by all vm's.
     
  3. lee.veal

    lee.veal Member HowtoForge Supporter

    I had this same issue on my Amazon hosted multi server setup. I disabled "Mailbox quota statistics" for each server and my IO dropped big time. Hope this helps
     
  4. olimortimer

    olimortimer Member

    I found that it was updatedb.mlocate that was causing the high disk IO usage. I've disabled it, but it is slightly annoying, as I use locate quite often.
     
  5. lee.veal

    lee.veal Member HowtoForge Supporter

    Ubuntu uses ionice to only use idle IO , I would check to see if that is an option for you. Locate is a very useful command


    if [ -x /usr/bin/ionice ] &&
    /usr/bin/ionice -c3 true 2>/dev/null; then
    IONICE="/usr/bin/ionice -c3"
    fi

    $IONICE /usr/bin/updatedb.mlocate
     
  6. olimortimer

    olimortimer Member

    Yeah, that's in the /etc/cron.daily/mlocate file;

    Code:
    #! /bin/bash
    
    set -e
    
    [ -x /usr/bin/updatedb.mlocate ] || exit 0
    
    if which on_ac_power >/dev/null 2>&1; then
        ON_BATTERY=0
        on_ac_power >/dev/null 2>&1 || ON_BATTERY=$?
        if [ "$ON_BATTERY" -eq 1 ]; then
            exit 0
        fi
    fi
    
    ##
    
    LOCKFILE="/var/lib/mlocate/daily.lock"
    
    trap "rm -f $LOCKFILE" EXIT
    
    if [ -e "$LOCKFILE" ]; then
        echo >&2 "Warning: $LOCKFILE present, not running updatedb."
        exit 1
    else
        touch "$LOCKFILE"
    fi
    
    ##
    
    # See ionice(1)
    if [ -x /usr/bin/ionice ] &&
        /usr/bin/ionice -c3 true 2>/dev/null; then
        IONICE="/usr/bin/ionice -c3"
    fi
    
    $IONICE /usr/bin/updatedb.mlocate
     
  7. lee.veal

    lee.veal Member HowtoForge Supporter

    Is ionice installed ?
     
    Last edited: May 8, 2014
  8. olimortimer

    olimortimer Member

    I believe so;

    Code:
    [email protected]:~# ionice
    none: prio 0
     

Share This Page