Unexplained Load increases

[ZeroEnna]

Hi everyone,

for over two months, I am facing unexplained load increases on my server.

Take a look at the screenshot.

Cacti-Screenshot

At about 2.00, no login is possible, and the remote service board fails to get a video signal as well, because no input is possible.

Well, is there a tool like Microsoft's perfmon to caputre load, and what application causes it? Maybe I can isolate the cause and fix the problem.

Thank you in advance

Kind Regards

ZeroEnna

[falko]

Might have to do with a cron job and/or hard disk I/O. munin might give you more insight.

[ZeroEnna]

Concerning the crontab, this is output of

for user in $(cut -f1 -d: /etc/passwd); do echo $user; crontab -u $user -l; done

root
0 1 * * * /var/www/kibo.fm/web/sctrans/sc_trans_040/sc_trans stop
45 * * * * MAILTO=""; lynx --dump "http://kibo.fm/meinkibo/bot.php?action=log&password=d6iKpB$"
00 01 * * 0 /root/logs_backup.sh

00 02 * * * rsync -rltDv -e ssh /backup/ hikarimedia@rsync.hidrive.strato.com:/users/hikarimedia/
00 03 * * 0 /root/spamtrain.sh
30 01 * * * /root/mysqlback.sh

00 03 * * 6 /root/reboot.sh
23 4 */2 * * /usr/bin/sa-update --no-gpg &> /dev/null
* * * * * /usr/local/ispconfig/server/server.sh > /dev/null 2>> /var/log/ispconfig/cron.log
30 00 * * * /usr/local/ispconfig/server/cron_daily.sh > /dev/null 2>> /var/log/ispconfig/cron.log
daemon
no crontab for daemon
bin
no crontab for bin
sys
no crontab for sys
sync
no crontab for sync
games
no crontab for games
man
no crontab for man
lp
no crontab for lp
mail
no crontab for mail
news
no crontab for news
uucp
no crontab for uucp
proxy
no crontab for proxy
www-data
no crontab for www-data
backup
no crontab for backup
list
no crontab for list
irc
no crontab for irc
gnats
no crontab for gnats
nobody
no crontab for nobody
libuuid
no crontab for libuuid
Debian-exim
no crontab for Debian-exim
statd
no crontab for statd
zeroenna
no crontab for zeroenna
sshd
no crontab for sshd
ntp
no crontab for ntp
mysql
no crontab for mysql
postfix
no crontab for postfix
clamav
no crontab for clamav
amavis
no crontab for amavis
vmail
no crontab for vmail
getmail
*/5 * * * * /usr/local/bin/run-getmail.sh > /dev/null 2>> /var/log/ispconfig/cron.log
ispapps
no crontab for ispapps
ispconfig
no crontab for ispconfig
web1
no crontab for web1
shoutcast
no crontab for shoutcast
web3
no crontab for web3
postgrey
no crontab for postgrey
unreal
no crontab for unreal
anope
no crontab for anope
eggdrop
# m h  dom mon dow   command
*/1 * * * * /home/eggdrop/eggdrop/scripts/botchk > /dev/null &
web5
no crontab for web5
web6
no crontab for web6
web7
no crontab for web7
web8
no crontab for web8
web9
# m h  dom mon dow   command
#00 16 * * * /var/www/kibo.fm/web/sctrans/sc_trans_040/sc_trans start
web11
no crontab for web11
web12
no crontab for web12
web13
no crontab for web13
web14
no crontab for web14
web15
no crontab for web15
web16
no crontab for web16
web17
no crontab for web17
web18
no crontab for web18
davfs2
no crontab for davfs2
dellroot
no crontab for dellroot
web19
no crontab for web19
ircservice
no crontab for ircservice
web21
no crontab for web21
marcel
no crontab for marcel
sctrans
no crontab for sctrans
web24
no crontab for web24
munin
no crontab for munin
messagebus
no crontab for messagebus
snmp
no crontab for snmp
web26
no crontab for web26
tcpdump
no crontab for tcpdump
web29
no crontab for web29
paranormal
# Edit this file to introduce tasks to be run by cron.
#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h  dom mon dow   command


28,58 * * * * /home/paranormal/bots/psi-bot/scripts/botchk



web31
no crontab for web31
minecraft
no crontab for minecraft
avahi
no crontab for avahi

[falko]

I guess this cron job is the problem:

00 02 * * * rsync -rltDv -e ssh /backup/ hikarimedia@rsync.hidrive.strato.com:/users/hikarimedia/

It could cause high disk I/O. Can you deactivate it and see if that changes anything?

[ZeroEnna]

Hi,
unfortunately, this is my backup job, so disabling it for about 3 weeks might cause a problem. Also, the problem occured at about 3 p.m, the task finished at about 2:40 a.m., so I don't see any relations between the increasing load and the task.

A few days ago I remembered that this had happened before. But in July or August (I think) there has been an update, I don't know what it was anymore, maybe the logs will tell.
Anyway, after this, I had uptimes of about 60 to 70 days, when a kernel update arrived. After installing it, my uptimes crashed! So either, one of my installed packages has some problems, or something else is going wrong with it, as a patch fixed it, and another patch caused it to happen again.

Regards

ZeroEnna

[falko]

Originally Posted by ZeroEnna

Also, the problem occured at about 3 p.m, the task finished at about 2:40 a.m., so I don't see any relations between the increasing load and the task.

In your first post you said it happens at 2.00h. (Is it possible that this has something to do with daylight savings time? Do you and your server have the same time?)

[ZeroEnna]

I'm sorry, I was misleading.

I am currently (at this very moment) having issues again. It started at about noon yesterday and increased slowly to a peak of 7 and staying!

Please have a look at the munin graphs (They are accessable)

I have no access to SSH anymore, The connection is established, I get a liogon prompt but then the session freezes with no possibility of entering commands. Also, the Console Redirection (DELL iDRAC) is not returning a display, it stays black.

The only option I have is a reboot, but I am afraid, I will never ever find out what is causing this! Is there a possibility (as FTP and HTTP are working fine) to write a script that returns the output as a Webpage (e.g. something with exec())?

Regards

Zero

[falko]

According to http://www.hikari-media.de/monitorin...ops/index.html you have some disk IO. That could be the problem, but I'm not sure. If it's possible to change the HDDs, you should try that.