Howto resolve sudden server crashes? Possibly spamassassin or clamscan?

[linus]

Hello and thank you for your wonderful Ispconfig! It is a brilliant set of tools!

Now I wonder how I should get rid of problems that causes my server to crash several times a year. It did it again this morning at 5:57:18. When I plugged the monitor there was a dump ending with text "closing the console". I was able to ping the machine. The webpages weren't served and the browser went on waiting. At the crash-time in the maillog there was a usual activity probably from spam-mail. I suspect clamscan and spamassassin, as previously spamassassin has generated segfaults.

How should one resolve errors like this? This is the relevant part of the /var/log/messages-file:

Jan 31 05:57:18 oslo kernel: general protection fault: 0000 [1] SMP
Jan 31 05:57:18 oslo kernel: last sysfs file: /devices/system/cpu/cpu0/cpufreq/scaling_setspeed
Jan 31 05:57:18 oslo kernel: CPU 0
Jan 31 05:57:18 oslo kernel:
Modules linked in: xt_tcpudp xt_state ipt_LOG ip_conntrack_ftp
iptable_mangle iptable_nat ip_nat ip_conntrack nfnetlink iptable_filter
ip_tables x_tables autofs4 hidp rfcomm l2cap bluetooth sunrpc ipv6 dm_mirror
dm_mod video button battery acpi_memhotplug ac lp parport_pc parport snd_via82xx gameport
snd_ac97_codec snd_ac97_bus snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss
floppy ehci_hcd snd_pcm via_velocity ohci1394 serio_raw i2c_viapro i2c_core uhci_hcd ieee1394 snd_timer snd_page_alloc snd_mpu401_uart

Jan 31 05:57:18 oslo kernel: Pid: 23720, comm: clamscan Not tainted 2.6.17-1.2187_FC5 #1
Jan 31 05:57:18 oslo kernel: RIP: 0010:[<ffffffff802075d3>] <ffffffff802075d3>{find_get_page+45}
Jan 31 05:57:18 oslo kernel: RSP: 0018:ffff81007c46dc68 EFLAGS: 00010006
Jan 31 05:57:18 oslo kernel: RAX: 2f2f3a7074746822 RBX: 2f2f3a7074746822 RCX: 0000000000000000
Jan 31 05:57:18 oslo kernel: RDX: 0000000000000000 RSI: 0000000000000100 RDI: ffff81004ec5c4c0
Jan 31 05:57:18 oslo kernel: RBP: ffff81002ceacc30 R08: 0000000000000004 R09: ffffffff8020caf1
Jan 31 05:57:18 oslo kernel: R10: ffff81007c46dde8 R11: 0000000000000246 R12: 0000000000000100
Jan 31 05:57:18 oslo kernel: R13: 00000000000000ff R14: 0000000000000000 R15: ffff81007c46dd88
Jan 31 05:57:18 oslo kernel: FS: 00002aaaaac13bc0(0000) GS:ffffffff8069e000(0000) knlGS:00000000f7fe56c0
Jan 31 05:57:18 oslo kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 31 05:57:18 oslo kernel: CR2: 000000000051c000 CR3: 0000000070b71000 CR4: 00000000000006e0
Jan 31 05:57:18 oslo kernel: Process clamscan (pid: 23720, threadinfo ffff81007c46c000, task ffff81001cdf2860)
Jan 31 05:57:18 oslo kernel: Stack: ffff81007c46dde8 0000000000000000 0000000000000100 ffffffff8020bab4
Jan 31 05:57:18 oslo kernel: ffffffff8020caf1 ffff81007c46de88 ffff81006bfe0200 ffff81006bfe0280
Jan 31 05:57:18 oslo kernel: ffff81002ceacc18 ffff81002ceaca88
Jan 31 05:57:18 oslo kernel: Call Trace: <ffffffff8020bab4>{do_generic_mapping_read+313}
Jan 31 05:57:18 oslo kernel: <ffffffff8020caf1>{file_read_actor+0} <ffffffff8020bee8>{__generic_file_aio_read+351}
Jan 31 05:57:18 oslo kernel: <ffffffff802175ff>{generic_file_aio_read+52} <ffffffff8020c891>{do_sync_read+199}
Jan 31 05:57:18 oslo kernel: <ffffffff80269d8e>{_spin_lock_irqsave+9} <ffffffff8029f419>{autoremove_wake_function+0}
Jan 31 05:57:18 oslo kernel: <ffffffff80269d4b>{_spin_unlock_irq+9} <ffffffff802677b6>{thread_return+0}
Jan 31 05:57:18 oslo kernel: <ffffffff80267814>{thread_return+94} <ffffffff8020ae31>{vfs_read+203}
Jan 31 05:57:18 oslo kernel: <ffffffff802111e3>{sys_read+69} <ffffffff802629ce>{system_call+126}
Jan 31 05:57:18 oslo kernel:
Jan 31 05:57:18 oslo kernel: Code: 8b 00 48 89 da f6 c4 40 74 04 48 8b 53 10 f0 ff 42 08 48 89
Jan 31 05:57:18 oslo kernel: RIP <ffffffff802075d3>{find_get_page+45} RSP <ffff81007c46dc68>
Jan 31 05:57:18 oslo kernel: <3>BUG: sleeping function called from invalid context at include/linux/rwsem.h:43
Jan 31 05:57:18 oslo kernel: in_atomic():0, irqs_disabled():1
Jan 31 05:57:18 oslo kernel:
Jan 31 05:57:18 oslo kernel: Call Trace: <ffffffff80299774>{blocking_notifier_call_chain+31}
Jan 31 05:57:18 oslo kernel: <ffffffff80215bf6>{do_exit+32} <ffffffff8027071f>{kernel_math_error+0}
Jan 31 05:57:18 oslo kernel: <ffffffff8026a9ee>{do_general_protection+254} <ffffffff802638a5>{error_exit+0}
Jan 31 05:57:19 oslo kernel: <ffffffff8020caf1>{file_read_actor+0} <ffffffff802075d3>{find_get_page+45}
Jan 31 05:57:19 oslo kernel: <ffffffff802075cb>{find_get_page+37} <ffffffff8020bab4>{do_generic_mapping_read+313}
Jan 31 05:57:19 oslo kernel: <ffffffff8020caf1>{file_read_actor+0} <ffffffff8020bee8>{__generic_file_aio_read+351}
Jan 31 05:57:19 oslo kernel: <ffffffff802175ff>{generic_file_aio_read+52} <ffffffff8020c891>{do_sync_read+199}
Jan 31 05:57:19 oslo kernel: <ffffffff80269d8e>{_spin_lock_irqsave+9} <ffffffff8029f419>{autoremove_wake_function+0}
Jan 31 05:57:19 oslo kernel: <ffffffff80269d4b>{_spin_unlock_irq+9} <ffffffff802677b6>{thread_return+0}
Jan 31 05:57:19 oslo kernel: <ffffffff80267814>{thread_return+94} <ffffffff8020ae31>{vfs_read+203}
Jan 31 05:57:19 oslo kernel: <ffffffff802111e3>{sys_read+69} <ffffffff802629ce>{system_call+126}


I'd be glad if someone has a clue on how to proceed.

Vielen danke!

[martinfst]

A segfault should not (does not, I have never seen it in >20 year of U/Linux experience) lead to a system crash, unless you have been toying around with device drivers and made a booboo.

For a default installed system I'd run several (extensive) memory checks. Most likely it will appear to be a faulty module. Next to try is a flacky disk. Any read/write errors in your kern.log file?

[linus]

Hello and thank you very much for helping!

Do you have any command line program that you could recommend, sufficient for testing the memory? Memtest86 from a floppy?

Is finding faulty modules possible for regular mortal people?

The only kernel messages I found in /var/log/messages (except for the bootup ones) were (no read or write errors):

Jan 16 00:48:14 oslo kernel: clamscan[339]: segfault at 00000051000000cc rip 00002aaaaaab6df6 rsp 00007fff274384c0 error 4
Jan 16 00:49:22 oslo kernel: spamassassin[520]: segfault at 0000000000000000 rip 0000003ae4d68d94 rsp 00007fff9d937940 error 4

Jan 18 14:51:51 oslo kernel: ispconfig_httpd[23396]: segfault at 00007fff3ebd4ff0 rip 0000003ae4040bc4 rsp 00007fff3ebd4fd0 error 6

Jan 23 15:47:11 oslo kernel: ispconfig_httpd[744]: segfault at 00007fff3ebd4f40 rip 0000003ae4040bc4 rsp 00007fff3ebd4f20 error 6

[martinfst]

Could it be that you mixed 32 and 64 bit libraries? Or you installed / updated to a 64 bit kernel afterwards?

Yes, Memtest86 or the hardware tests from the manufacturer. Usually systems come with a diagnostics CD/Floppy, which does contain a memory test set. Some BIOS versions include a memtest in the bios-setup.