Unexplained Load increases

Discussion in 'Technical' started by ZeroEnna, Nov 22, 2011.

  1. ZeroEnna

    ZeroEnna Member

    Hi everyone,

    for over two months, I am facing unexplained load increases on my server.

    Take a look at the screenshot.

    Cacti-Screenshot

    At about 2.00, no login is possible, and the remote service board fails to get a video signal as well, because no input is possible.

    Well, is there a tool like Microsoft's perfmon to caputre load, and what application causes it? Maybe I can isolate the cause and fix the problem.

    Thank you in advance

    Kind Regards

    ZeroEnna
     
  2. falko

    falko Super Moderator ISPConfig Developer

    Might have to do with a cron job and/or hard disk I/O. munin might give you more insight.
     
  3. ZeroEnna

    ZeroEnna Member

    Concerning the crontab, this is output of

    Code:
    for user in $(cut -f1 -d: /etc/passwd); do echo $user; crontab -u $user -l; done
    Code:
    root
    0 1 * * * /var/www/kibo.fm/web/sctrans/sc_trans_040/sc_trans stop
    45 * * * * MAILTO=""; lynx --dump "http://kibo.fm/meinkibo/bot.php?action=log&password=d6iKpB$"
    00 01 * * 0 /root/logs_backup.sh
    
    00 02 * * * rsync -rltDv -e ssh /backup/ [email protected]:/users/hikarimedia/
    00 03 * * 0 /root/spamtrain.sh
    30 01 * * * /root/mysqlback.sh
    
    00 03 * * 6 /root/reboot.sh
    23 4 */2 * * /usr/bin/sa-update --no-gpg &> /dev/null
    * * * * * /usr/local/ispconfig/server/server.sh > /dev/null 2>> /var/log/ispconfig/cron.log
    30 00 * * * /usr/local/ispconfig/server/cron_daily.sh > /dev/null 2>> /var/log/ispconfig/cron.log
    daemon
    no crontab for daemon
    bin
    no crontab for bin
    sys
    no crontab for sys
    sync
    no crontab for sync
    games
    no crontab for games
    man
    no crontab for man
    lp
    no crontab for lp
    mail
    no crontab for mail
    news
    no crontab for news
    uucp
    no crontab for uucp
    proxy
    no crontab for proxy
    www-data
    no crontab for www-data
    backup
    no crontab for backup
    list
    no crontab for list
    irc
    no crontab for irc
    gnats
    no crontab for gnats
    nobody
    no crontab for nobody
    libuuid
    no crontab for libuuid
    Debian-exim
    no crontab for Debian-exim
    statd
    no crontab for statd
    zeroenna
    no crontab for zeroenna
    sshd
    no crontab for sshd
    ntp
    no crontab for ntp
    mysql
    no crontab for mysql
    postfix
    no crontab for postfix
    clamav
    no crontab for clamav
    amavis
    no crontab for amavis
    vmail
    no crontab for vmail
    getmail
    */5 * * * * /usr/local/bin/run-getmail.sh > /dev/null 2>> /var/log/ispconfig/cron.log
    ispapps
    no crontab for ispapps
    ispconfig
    no crontab for ispconfig
    web1
    no crontab for web1
    shoutcast
    no crontab for shoutcast
    web3
    no crontab for web3
    postgrey
    no crontab for postgrey
    unreal
    no crontab for unreal
    anope
    no crontab for anope
    eggdrop
    # m h  dom mon dow   command
    */1 * * * * /home/eggdrop/eggdrop/scripts/botchk > /dev/null &
    web5
    no crontab for web5
    web6
    no crontab for web6
    web7
    no crontab for web7
    web8
    no crontab for web8
    web9
    # m h  dom mon dow   command
    #00 16 * * * /var/www/kibo.fm/web/sctrans/sc_trans_040/sc_trans start
    web11
    no crontab for web11
    web12
    no crontab for web12
    web13
    no crontab for web13
    web14
    no crontab for web14
    web15
    no crontab for web15
    web16
    no crontab for web16
    web17
    no crontab for web17
    web18
    no crontab for web18
    davfs2
    no crontab for davfs2
    dellroot
    no crontab for dellroot
    web19
    no crontab for web19
    ircservice
    no crontab for ircservice
    web21
    no crontab for web21
    marcel
    no crontab for marcel
    sctrans
    no crontab for sctrans
    web24
    no crontab for web24
    munin
    no crontab for munin
    messagebus
    no crontab for messagebus
    snmp
    no crontab for snmp
    web26
    no crontab for web26
    tcpdump
    no crontab for tcpdump
    web29
    no crontab for web29
    paranormal
    # Edit this file to introduce tasks to be run by cron.
    #
    # Each task to run has to be defined through a single line
    # indicating with different fields when the task will be run
    # and what command to run for the task
    #
    # To define the time you can provide concrete values for
    # minute (m), hour (h), day of month (dom), month (mon),
    # and day of week (dow) or use '*' in these fields (for 'any').#
    # Notice that tasks will be started based on the cron's system
    # daemon's notion of time and timezones.
    #
    # Output of the crontab jobs (including errors) is sent through
    # email to the user the crontab file belongs to (unless redirected).
    #
    # For example, you can run a backup of all your user accounts
    # at 5 a.m every week with:
    # 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
    #
    # For more information see the manual pages of crontab(5) and cron(8)
    #
    # m h  dom mon dow   command
    
    
    28,58 * * * * /home/paranormal/bots/psi-bot/scripts/botchk
    
    
    
    web31
    no crontab for web31
    minecraft
    no crontab for minecraft
    avahi
    no crontab for avahi
    
     
  4. falko

    falko Super Moderator ISPConfig Developer

    I guess this cron job is the problem:

    Code:
    00 02 * * * rsync -rltDv -e ssh /backup/ [email protected]:/users/hikarimedia/
    It could cause high disk I/O. Can you deactivate it and see if that changes anything?
     
  5. ZeroEnna

    ZeroEnna Member

    Hi,
    unfortunately, this is my backup job, so disabling it for about 3 weeks might cause a problem. Also, the problem occured at about 3 p.m, the task finished at about 2:40 a.m., so I don't see any relations between the increasing load and the task.

    A few days ago I remembered that this had happened before. But in July or August (I think) there has been an update, I don't know what it was anymore, maybe the logs will tell.
    Anyway, after this, I had uptimes of about 60 to 70 days, when a kernel update arrived. After installing it, my uptimes crashed! So either, one of my installed packages has some problems, or something else is going wrong with it, as a patch fixed it, and another patch caused it to happen again.

    Regards

    ZeroEnna
     
  6. falko

    falko Super Moderator ISPConfig Developer

    In your first post you said it happens at 2.00h. (Is it possible that this has something to do with daylight savings time? Do you and your server have the same time?)
     
  7. ZeroEnna

    ZeroEnna Member

    I'm sorry, I was misleading.

    I am currently (at this very moment) having issues again. It started at about noon yesterday and increased slowly to a peak of 7 and staying!

    Please have a look at the munin graphs (They are accessable)

    I have no access to SSH anymore, The connection is established, I get a liogon prompt but then the session freezes with no possibility of entering commands. Also, the Console Redirection (DELL iDRAC) is not returning a display, it stays black.

    The only option I have is a reboot, but I am afraid, I will never ever find out what is causing this! Is there a possibility (as FTP and HTTP are working fine) to write a script that returns the output as a Webpage (e.g. something with exec())?

    Regards

    Zero
     
  8. falko

    falko Super Moderator ISPConfig Developer

Share This Page