segmentation fault

[Daisy]

newbie screw up. I noticed that my server was running slow and I couldn't figure out what it was so I pulleda windows and rebooted. ACK! It rebooted and everything is screwed. Segmentation errors out the wazoo. i get to a prompt and login and the error I get there is /bin/unicode_start: line 15: 1835 Segmentation fault /bin/kbd_mode -u. Where are the logs and where should I even start to look for what went wrong? Please help!!

running:
fedora core 5
ispconfig
dovecot
postfix
openssh

the howtoforge perfect server setup

p.s. what the heck is a segmentation fault?
p.p.s. I looked in dmesg but don't see any errors. Is that the right place to look?
p.p.s I've got no network either.

[till]

Segmentation faults are often caused by hardware problems, so it might be a problem with your RAM or mainboard that causes the problems.

[Daisy]

I dropped a spare hdd in and installed FC5 and ispconfig and it's working like a champ. I took the two hdd's that were in my machine and put them in a different machine with the same hardware config and still got the segmentation faults so I'm 99% sure it's not hardware. What would be the next step. I had noticed that the RAID 1 wasn't working. I have it setup with LVM RAID 1. Consequently, I can't access the drives independently. They have to be together which sucks. I think I made a mistake using LVM. What would be the next step from here to figure out what went wrong?

[till]

It is possible to recover data from a LVM raid volume. Here is the first draft of a howto that I'am currently writing:

I used a Knoppix 5.1 LiveCD for this tutorial. Download the CD ISO-Image from here and burn it on CD, then connect the harddisk which contains the raid partition(s) to the IDE / ATA controller of your mainboard, put the Knoppix CD in your CD drive and boot from the CD.

The Harddisk I used is a IDE drive that is attached to the first IDE Controller (hda). In my case, the harddisk contained only one partition.

Restoring the Raid

After Knoppix has bootet, Open a shell and execute the command:

sudo su

to become root user.

As I dont have the mdadm.conf file from the original configuration, I create it with this command:

mdadm --examine --scan /dev/hda1 >> /etc/mdadm/mdadm.conf

The result shall be similar like this:

DEVICE partitions
CREATE owner=root group=disk mode=0660 auto=yes metadata=1
MAILADDR root
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=a28090aa:6893be8b:c4024dfc:29cdb07a

Edit the file and add "devices=/dev/hda1,missing" at the end of the line that describes the raid array.

vi /etc/mdadm/mdadm.conf

finally the file looks like this:

DEVICE partitions
CREATE owner=root group=disk mode=0660 auto=yes metadata=1
MAILADDR root
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=a28090aa:6893be8b:c4024dfc:29cdb07a devices=/dev/hda1,missing

The string "/dev/hda1" is the hardware device and "missing" means that the second disk in this raid is not present at the moment.

Edit the file /etc/default/mdadm:

vi /etc/default/mdadm


and change the line:

AUTOSTART=false

to:

AUTOSTART=true


Now we can start our raid setup:

/etc/init.d/mdadm start
/etc/init.d/mdadm-raid start

To check if our raid device is ok, run the command:

cat /proc/mdstat

The output shall look similar to this:


Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [ra id10]
md0 : active raid1 hda1[1]
293049600 blocks [2/1] [_U]

unused devices: <none>


Recovering the LVM setup

The LVM configuration file can not be created by a easy command like the mdadm.conf, but LVM stores one or more copy(s) of the configuration file content at the beginning of the partition. I use the command dd to extract the first part of the partition and write it to a text file:

dd if=/dev/md0 bs=512 count=255 skip=1 of=/tmp/md0.txt

Open the file with a text editor:

vi /tmp/md0.txt

You will find some binary data first and then a configuration file part like this:

VolGroup00 {
id = "evRkPK-aCjV-HiHY-oaaD-SwUO-zN7A-LyRhoj"
seqno = 2
status = ["RESIZEABLE", "READ", "WRITE"]
extent_size = 65536 # 32 Megabytes
max_lv = 0
max_pv = 0

physical_volumes {

pv0 {
id = "uMJ8uM-sfTJ-La9j-oIuy-W3NX-ObiT-n464Rv"
device = "/dev/md0" # Hint only

status = ["ALLOCATABLE"]
pe_start = 384
pe_count = 8943 # 279,469 Gigabytes
}
}

logical_volumes {

LogVol00 {
id = "ohesOX-VRSi-CsnK-PUoI-GjUE-0nT7-ltxWoy"
status = ["READ", "WRITE", "VISIBLE"]
segment_count = 1

segment1 {
start_extent = 0
extent_count = 8942 # 279,438 Gigabytes

type = "striped"
stripe_count = 1 # linear

stripes = [
"pv0", 0
]
}
}
}
}

Create a file /etc/lvm/backup/VolGroup00:

vi /etc/lvm/backup/VolGroup00

and insert the configuration data so the file looks similar to the above example.

Now we can start LVM:

/etc/init.d/lvm start

read in the volume:

vgscan

Reading all physical volumes. This may take a while...
Found volume group "VolGroup00" using metadata type lvm2

pvscan

PV /dev/md0 VG VolGroup00 lvm2 [279,47 GB / 32,00 MB free]
Total: 1 [279,47 GB] / in use: 1 [279,47 GB] / in no VG: 0 [0 ]

and activate the volume:

vgchange VolGroup00 -a y

1 logical volume(s) in volume group "VolGroup00" now active

Now we are able to mount the partition to /mnt/data

mkdir /mnt/data
mount /dev/VolGroup00/LogVol00 /mnt/data/


If you recover data from a harddisk with filenames in UTF-8 format, it might be nescessary to convert them to your current non UTF-8 locale. In may case, the raid harddisk is from a Fedora Core system with UTF-8 encoded filenames. My target locale is ISO-8859-1. In this cases, the perl script cnvmv helps to convert the filenames to the target locale.

Installation of confmv:

cd /tmp
wget http://j3e.de/linux/convmv/convmv-1.10.tar.gz
tar xvfz convmv-1.10.tar.gz
cd convmv-1.10
cp convmv /usr/bin/convmv


To convert all filenames in /mnt/data to ISO-8859-1 locale, run this command:

convmv -f UTF-8 -t ISO-8859-1 -r --notest /mnt/data/*

If you want to test the conversion first, use:

convmv -f UTF-8 -t ISO-8859-1 -r /mnt/data/*


Related Links:


http://j3e.de/linux/convmv/
http://www.linuxjournal.com/article/8874

[Daisy]

while a very cool how to, I was sorta hoping to save the FC5/ISPconfig installation so I don't have to do it all over again. I've got a mess of sites and users and it would like, way suck to have to try to get all their info again. not to mention make me look like a compelte tard. (which I am I'm just pretty good at hiding it.)

[till]

As you posted above, the install on the harddisks wont boot anymore. So the normal way to resolve is to gain access to the data (thyts why I posted the howto) and migrate the users, ISPConfig and webiste data to a new running linux system.

If you search the forum for "move ispconfig", you will find several threads about moving a ISPConfig installation to a new server.

[Daisy]

oh! sorry! My fault. I didn't communicate well enough. It still boots. Gets to a prompt. I can login as root. I just get a ton of failed msg's and segmentation fault alert thingy mabobbers as it's loading. It also has a real hard time shutting down. it won't cut off unless I do a shutdown -r now and then hard power it off while it's reloading the bios.

[Daisy]

definitely 100% sure it's not a hardware issue now. I dropped in a spare hdd and installed FC5 4 days ago and it's still running fine. I'm hoping we can figure out what went wrong with the other installation so we can fix it and keep using it. I really don't want to start all over. It's why I switched to linux. supposedly more stable than windows but so far, it's been more problematic. Running fine one minute and broken the next with no changes made. My windows server was up and running with only 5 reboots in 6 years. It once ran 507 days without error or reboot and I'm really hoping I can get the same kind of performance out of linux. Any other suggestions on where I should start looking? Like I said, I can login as root but my commands are limited because of all the errors. I should be able to use a rescue cd to get around that though right?

[edge]

definitely 100% sure it's not a hardware issue now. I dropped in a spare hdd and installed FC5 4 days ago and it's still running fine.

I'm lost here... Is a "HDD" not hardware?
It sounds to me that your HDD (the old one with the error's) is the problem.

[Daisy]

ahhm, forgot to mention, the "old" hdd's (there were two using linux software raid1 with LVM) are only about 2 months old. I ran diagnostics on them and they didn't show any errors. I can mount them with the new installation of FC5 and access all the data. I'm pretty sure it's not hardware. I think I might of screwed something a while back trying to make the RAID work but I never rebooted and because I just now rebooted the problems showed up.

[Daisy]

any ideas? at all? anyone? Still haven't found the problem and really don't want to start over....

[Daisy]

looking at your tutorial, I'm not sure what to do. I installed FC5 on a clean hdd and then I hooked up one of the hdd's from the broken raid set. I think it's a bit different cause I'm using sata and it shows up different. I do a fdisk -l and it shows this:

[root@server etc]# fdisk -l

Disk /dev/sda: 160.0 GB, 160000000000 bytes
255 heads, 63 sectors/track, 19452 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 19452 156143767+ 8e Linux LVM

Disk /dev/sdb: 320.0 GB, 320072933376 bytes
255 heads, 63 sectors/track, 38913 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 * 1 13 104391 fd Linux raid autodetect
/dev/sdb2 14 38913 312464250 fd Linux raid autodetect

Disk /dev/md0: 106 MB, 106823680 bytes
2 heads, 4 sectors/track, 26080 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/md0 doesn't contain a valid partition table

Disk /dev/md1: 319.9 GB, 319963267072 bytes
2 heads, 4 sectors/track, 78116032 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/md1 doesn't contain a valid partition table

what should I do differently? Also what's a knoppix live cd?

[falko]

Also what's a knoppix live cd?
Take a look here: http://www.knoppix.net/
It's a Debian-based, very powerful Live-CD.

[Daisy]

Till, so I followed your instructions replacing hda1 with sda1. My hdd has two partitions though. sda1 and sda2. The other drive is sdb1 and sdb2. I went through your steps for the RAID part of it and I get an error:

Assembling MD array md0. . . failed (no devices found)
generating udev events for MD arrays. . . done.

What should I do?

[Daisy]

scratch that. Ok, so I added sda2, sdb1, and sdb2 to the mdadm.conf and restarted and it found the raid array. yaay. ok. so, basic tutorial time? what's md0 and md1? so it finds array md1 and it's degraded. I assume that means it's working somewhat but would love to learn more about it.

[Daisy]

went through the LVM stuff and can now access all the data but what do I do with it? Both drives are hooked up to my sata card and the cdrom is IDE so where can I copy the data to so I can reinstall FC5 and get ispconfig reinstalled?

[Daisy]

I borrowed a USB hdd and I've got it mounted but I can't seem to copy all my info. I don't know if I did it right though cause I'm only trying to copy 18GB to a 40GB drive but it keeps telling me I'm out of room. What did I do wrong?

[Daisy]

anyone? Hello?

[till]

If it tells you that you are out of space, then you are most likely out of space. have you considered to compreass tha data in a tar.gz or tar.bz2 file?

[Daisy]

i used the right click properties in konquerer and it says only 4.1 gb are copied over before it runs out of space but it's a 40 gb drive. I used fdisk and mkfs as described here.

http://www.ehow.com/how_1000631_hard-drive-linux.html

and then I mount it to a folder I created following the steps you outlined to mount the LVM from the previous page. could I have gotten something wrong?

and no. I hadn't considered tar. It's like zip right? How does one go about that? I'm really sorry but this is all still really new to me.

[Daisy]

I took the hdd out of the external case and connected to the PC's IDE controller. it recognizes it but still same problem. It copies over about 4 gigs and then says I'm out of space. Suggestions?

[falko]

What's the output of df -h?

[Daisy]

Filesystem Size Used Avail Use% Mounted on
/dev/root 3.4M 28K 3.4M 1% /
/ramdisk 198M 5.0M 193M 3% /ramdisk
/UNIONFS 198M 5.0M 193M 3% /UNIONFS
/dev/hdc 697M 697M 0 100% /cdrom
/dev/cloop 2.0G 2.0G 0 100% /KNOPPIX
/dev/mapper/VolGroup01-LogVol00
286G 55G 217G 21% /mnt/data
/dev/hda1 37G 48M 35G 1% /mnt/forty

[falko]

Is that the output with your external HDD mounted?

[Daisy]

yes. I pulled the hdd out of the case and hooked direct to the IDE controller on the mother board. It's hda1 partition.

[Daisy]

I removed the 40GB drive and added a USB 160GB drive. I formatted and mounted it and this is what it shows:

fdisk shows
Disk /dev/sdc: 160.0 GB
/dev/sdc1 1 19452 156248158+ 83 Linux

df -h shows
/dev/sdc1 99M 5.6M 89M 6% /mnt/backup

any idea what I'm doing wrong?

[till]

How do you format the harddisk? EXT3?

[Daisy]

mkfs -t ext3 /dev/sdc1

[falko]

mkfs -t ext3 /dev/sdc1
That's for creating the filesystem in the partition. But how did you create the partition? With fdisk? Did you specify that the partition should use the whole space on the HDD?

[Daisy]

I followed the tutorial here: http://www.ehow.com/how_1000631_hard-drive-linux.html

At the command prompt, type "fdisk /dev/sdc" replacing the "sdc" with the letters for your drive. Upon opening, fdisk may give you a couple of warnings, all of which can be ignored. It then gives you a prompt that looks like this: Command (m for help):

Enter "p" to see the partition table of the drive. The first line of output from the "p" command will also tell you size of the drive. This is a good way to double check that you are working with the correct drive.

If there are any partitions already on the drive they will be listed as the last lines of the "p" command. On our example, this looks like "/dev/sdc1" followed by some information about the partition's size and file system.

To delete any existing partitions, press "d" then enter. It will ask you which partition number you wish to delete. The number of the partition is the number that follows sdc, so on our example system we enter 1. If there are multiple partitions repeat the "d" command for each one. You can always view the partition table again with the "p" command.

Once you have deleted all existing partitions on the drive you are ready to make a new one. Type "n" and hit enter. Then press "p" to create a primary partition. It asks you for a partition number, enter "1." Now you are asked which cylinder the partition should start at, the beginning of the drive is the default, so just hit Enter. Then you are asked for the last cylinder, the end of the drive is default so you can just press Enter again.

Now you are back at fdisk's command prompt. Use the "p" command to check the partition table. You should now see your new partition at the bottom of the output. In the example it lists "/dev/sdc1."

You now need to set the filesystem type for your new partition with the "t" command. You are asked for the Hex code of the filesystem you wish to use. We wll use the standard Linux ext2 filesystem which is "83." If you are doing something special and know of a particular filesystem that you need to use, you can press "L" to see all the codes, which are one or two characters made up of the numbers 0-9 and the letters a-f.

Now just issue the "w" command to write your new partition table and exit fdisk.

[falko]

That sounds correct, but still something seems to be wrong with your HDD...

[Daisy]

I've tried it with a 40GB PATA and a 160GB SATA. Same problem on both. The 160 had a copy of fedora installed successfully on it with the full 160GB available before I repartitioned and formatted.

[Daisy]

I went and bought a NEW 300GB hdd and am having the same problem. This is in a totally different machine too.