My server has crashed unexpectedly for 2 days

[arastirici]

I use Opensuse11

My server has crashed for 2 days. I had to restart by the button.
After the restart, the server works for a time about half a day. and it has crash unexptectedly.

When I look at the /var/log/messages, I see something messages as below.

Oct 7 10:15:09 php logrotate: ALERT exited abnormally with [1]
Oct 7 10:15:09 php logrotate: Reload httpd2 (graceful restart)..done
Oct 7 10:15:09 php logrotate: Reload httpd2 (graceful restart)..done
Oct 7 10:15:09 php logrotate: ^G/usr/bin/mysqladmin: connect to server at 'localhost' failed
Oct 7 10:15:09 php logrotate: error: 'Access denied for user 'root'@'localhost' (using password: NO)'
Oct 7 10:15:09 php logrotate: /etc/logrotate.d/mysql failed, probably because
Oct 7 10:15:09 php logrotate: the root acount is protected by password.
Oct 7 10:15:09 php logrotate: See comments in /etc/logrotate.d/mysql on how to fix this
Oct 7 10:15:09 php logrotate: error: error running non-shared postrotate script for /var/lib/mysql/mysqld.log of '/var/lib/mysql/my
Oct 7 10:15:12 php su: (to nobody) root on none
Oct 7 10:15:32 php syslog-ng[2885]: last message repeated 2 times
Oct 7 09:47:10 php smartd[3017]: Device: /dev/sda, SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 57 to 58
Oct 7 09:47:10 php smartd[3017]: Device: /dev/sda, SMART Usage Attribute: 194 Temperature_Celsius changed from 43 to 42
Oct 7 10:23:21 php kernel: __journal_remove_journal_head: freeing b_frozen_data
Oct 7 10:25:52 php kernel: Bad page state in process 'kswapd0'
Oct 7 10:25:52 php kernel: page:c1800000 flags:0x80000008 mapping:d4e86564 mapcount:0 count:0
Oct 7 10:25:52 php kernel: Trying to fix it up, but a reboot is needed
Oct 7 10:25:52 php kernel: Backtrace:
Oct 7 10:25:52 php kernel: Pid: 20, comm: kswapd0 Tainted: G N 2.6.25.16-0.1-pae #1
Oct 7 10:25:52 php kernel: [<c01071d9>] dump_trace+0x63/0x227
Oct 7 10:25:52 php kernel: [<c0107c8a>] show_trace+0x15/0x29
Oct 7 10:25:52 php kernel: [<c02e2c89>] dump_stack+0x5b/0x65
Oct 7 10:25:52 php kernel: [<c015d9c9>] bad_page+0x55/0x82
Oct 7 10:25:52 php kernel: [<c015e495>] free_hot_cold_page+0x63/0x1b2
Oct 7 10:25:52 php kernel: [<c015e601>] __pagevec_free+0x1d/0x25
Oct 7 10:25:52 php kernel: [<c0160b62>] release_pages+0x14b/0x153
Oct 7 10:25:52 php kernel: [<c0160f18>] __pagevec_release+0x18/0x21
Oct 7 10:25:52 php kernel: [<c0162903>] shrink_inactive_list+0x334/0x33f
Oct 7 10:25:52 php kernel: [<c01629d9>] shrink_zone+0xcb/0xea
Oct 7 10:25:52 php kernel: [<c0162eaa>] kswapd+0x380/0x4ba
Oct 7 10:25:52 php kernel: [<c0135f07>] kthread+0x3b/0x61
Oct 7 10:25:52 php kernel: [<c0106d37>] kernel_thread_helper+0x7/0x10
Oct 7 10:25:52 php kernel: =======================

What may the problem be ?

[damir]

Seems to be hardware related, have you installed anything hardware related?

Can you run free command and df -h and paste the output.

[arastirici]

I haven't installed recently any hardware

php:/ # free
total used free shared buffers cached
Mem: 4139476 388724 3750752 0 22240 198332
-/+ buffers/cache: 168152 3971324
Swap: 5124724 0 5124724


php:/ # df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 97G 5.3G 86G 6% /
udev 2.0G 92K 2.0G 1% /dev
/dev/sda2 39G 750M 36G 3% /var
/dev/sda4 114G 66G 43G 61% /www

[damir]

php:/ # df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 97G 5.3G 86G 6% /
udev 2.0G 92K 2.0G 1% /dev
/dev/sda2 39G 750M 36G 3% /var
/dev/sda4 114G 66G 43G 61% /www

There is no swap partition, now it is possible that you are running out of RAM and because swap is missing system crashes.

[arastirici]

When I run top, swap exist;

top - 12:15:05 up 1:26, 3 users, load average: 0.00, 0.00, 0.00
Tasks: 112 total, 1 running, 110 sleeping, 0 stopped, 1 zombie
Cpu(s): 0.6%us, 0.1%sy, 0.0%ni, 98.1%id, 1.2%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 4139476k total, 370808k used, 3768668k free, 32000k buffers
Swap: 5124724k total, 0k used, 5124724k free, 223684k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 772 304 256 S 0 0.0 0:00.76 init
2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/1
6 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
7 root 15 -5 0 0 0 S 0 0.0 0:00.04 events/0
8 root 15 -5 0 0 0 S 0 0.0 0:00.06 events/1
9 root 15 -5 0 0 0 S 0 0.0 0:00.00 khelper
10 root 15 -5 0 0 0 S 0 0.0 0:00.02 kblockd/0
11 root 15 -5 0 0 0 S 0 0.0 0:00.02 kblockd/1
12 root 15 -5 0 0 0 S 0 0.0 0:00.00 kacpid
13 root 15 -5 0 0 0 S 0 0.0 0:00.00 kacpi_notify
14 root 15 -5 0 0 0 S 0 0.0 0:00.00 cqueue
15 root 15 -5 0 0 0 S 0 0.0 0:00.00 kseriod
16 root 15 -5 0 0 0 S 0 0.0 0:00.36 kondemand/0
17 root 15 -5 0 0 0 S 0 0.0 0:00.44 kondemand/1
18 root 20 0 0 0 0 S 0 0.0 0:00.00 pdflush
19 root 20 0 0 0 0 S 0 0.0 0:00.06 pdflush
20 root 15 -5 0 0 0 S 0 0.0 0:00.00 kswapd0

[damir]

What is the output od fdisk -l

[arastirici]

php:/ # fdisk -l

Disk /dev/sda: 320.0 GB, 320072933376 bytes
255 heads, 63 sectors/track, 38913 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x90909090

Device Boot Start End Blocks Id System
/dev/sda1 * 263 13011 102406342+ 83 Linux
/dev/sda2 18112 23211 40965750 83 Linux
/dev/sda3 23212 23849 5124735 82 Linux swap / Solaris
/dev/sda4 23850 38913 121001580 83 Linux

[damir]

I think the problem is that you have to little of RAM memory, wich after a while runs out and system tries to swap but can't find the swap partition and crashes.

Try from comandline:

swapon /dev/sda3

Than edit /etc/fstab and add following:

/dev/sda3 none swap sw 0 0

Now if memory runs out, OS is gonna swap to hardrive and it shouldn't crash.

The best solution is to add more RAM, because RAM is cheap and it's gonna help your system stay online

[arastirici]

Meanwhile, My server has crashed again.
it has been waiting during the boot. there is "grub loading stage2" on the screen.

What do I have to do ?

[arastirici]

I booted the server by opensuse installer. it repaired problems on the disk because of I had to reset on the button.
I also changed its RAMs.
Now, the system has been up about 24 hours.