Failed during pvmove, howto: set up software raid 1 on running LVM system

[sbh77]

Hi...

After succesfully following the guide on installing software RAID 1 on a running ubuntu 10.04 system (no LVM), I decided to do the same on a server machine with LVM installed:
http://www.howtoforge.com/how-to-set...n-ubuntu-10.04

The configuration as shown in the howto is identical to our server except for the server name. Of course I used our server name whenever it was written server1.

After the command
$ pvmove -i 2 /dev/sda5 /dev/md1
reached 100%, it displayed:
ABORTING: Can't find mirror LV in servername for /dev/sda5

After this I couldn't do anything anymore. all LVM commands resulted with: "Command failed with status code 5."
And the system became read-only.

I had to reboot, and of course, changing the grub and fstab resulted in an unbootable system.

So now I need to either continue the process, or restore the system.

Is there a way to restore the previous state? I assume I need to load the live CD, install mdm, mount sda through lvm and somehow restore the system from the command line, but I have no idea how to do all this... anyone?

Thanks,
Danny

[sbh77]

Still no luck. I'm trying to restore the system to its former state, I need to mount sda in the live cd environment:

$ pvscan
Couldn't find device with uuid '5YzXQo-8F15-lq66-O316-nufN-hlWJ-T7nW7O'.
PV /dev/sda5 VG MadaaServer lvm2 [465.52 GiB / 465.52 GiB free]
PV unknown device VG MadaaServer lvm2 [465.52 GiB / 52.00 MiB free]
Total: 2 [931.04 GiB] / in use: 2 [931.04 GiB] / in no VG: 0 [0 ]

The unknown device is the RAID md0 that was installed during the guide steps.

Trying to activate the VG:

$ vgchange -a y
Couldn't find device with uuid '5YzXQo-8F15-lq66-O316-nufN-hlWJ-T7nW7O'.
Couldn't find device with uuid '5YzXQo-8F15-lq66-O316-nufN-hlWJ-T7nW7O'.
Refusing activation of partial LV root. Use --partial to override.
Couldn't find device with uuid '5YzXQo-8F15-lq66-O316-nufN-hlWJ-T7nW7O'.
Refusing activation of partial LV swap_1. Use --partial to override.
0 logical volume(s) in volume group "MadaaServer" now active

Using Partial:

$ vgchange -a y --partial
Partial mode. Incomplete volume groups will be activated read-only.
Couldn't find device with uuid '5YzXQo-8F15-lq66-O316-nufN-hlWJ-T7nW7O'.
Couldn't find device with uuid '5YzXQo-8F15-lq66-O316-nufN-hlWJ-T7nW7O'.
Couldn't find device with uuid '5YzXQo-8F15-lq66-O316-nufN-hlWJ-T7nW7O'.
2 logical volume(s) in volume group "MadaaServer" now active

$ vgscan
Reading all physical volumes. This may take a while...
/dev/mapper/MadaaServer-root-missing_0_0: read failed after 0 of 4096 at 488984477696: Input/output error
/dev/mapper/MadaaServer-root-missing_0_0: read failed after 0 of 4096 at 488984535040: Input/output error
/dev/mapper/MadaaServer-root-missing_0_0: read failed after 0 of 4096 at 0: Input/output error
/dev/mapper/MadaaServer-root-missing_0_0: read failed after 0 of 4096 at 4096: Input/output error
/dev/mapper/MadaaServer-root-missing_0_0: read failed after 0 of 4096 at 0: Input/output error
/dev/MadaaServer/root: read failed after 0 of 4096 at 488984477696: Input/output error
/dev/MadaaServer/root: read failed after 0 of 4096 at 488984535040: Input/output error
/dev/MadaaServer/root: read failed after 0 of 4096 at 0: Input/output error
/dev/MadaaServer/root: read failed after 0 of 4096 at 4096: Input/output error
/dev/MadaaServer/root: read failed after 0 of 4096 at 0: Input/output error
/dev/mapper/MadaaServer-swap_1-missing_0_0: read failed after 0 of 4096 at 10808655872: Input/output error
/dev/mapper/MadaaServer-swap_1-missing_0_0: read failed after 0 of 4096 at 10808713216: Input/output error
/dev/mapper/MadaaServer-swap_1-missing_0_0: read failed after 0 of 4096 at 0: Input/output error
/dev/mapper/MadaaServer-swap_1-missing_0_0: read failed after 0 of 4096 at 4096: Input/output error
/dev/mapper/MadaaServer-swap_1-missing_0_0: read failed after 0 of 4096 at 0: Input/output error
/dev/MadaaServer/swap_1: read failed after 0 of 4096 at 10808655872: Input/output error
/dev/MadaaServer/swap_1: read failed after 0 of 4096 at 10808713216: Input/output error
/dev/MadaaServer/swap_1: read failed after 0 of 4096 at 0: Input/output error
/dev/MadaaServer/swap_1: read failed after 0 of 4096 at 4096: Input/output error
/dev/MadaaServer/swap_1: read failed after 0 of 4096 at 0: Input/output error
/dev/mapper/MadaaServer-root-missing_0_0: read failed after 0 of 4096 at 0: Input/output error
/dev/MadaaServer/root: read failed after 0 of 4096 at 0: Input/output error
/dev/mapper/MadaaServer-swap_1-missing_0_0: read failed after 0 of 4096 at 0: Input/output error
/dev/MadaaServer/swap_1: read failed after 0 of 4096 at 0: Input/output error
Couldn't find device with uuid '5YzXQo-8F15-lq66-O316-nufN-hlWJ-T7nW7O'.
Found volume group "MadaaServer" using metadata type lvm2

Trying to mount:

$ mount -t autofs /dev/MadaaServer/root /mnt/madaa
mount: wrong fs type, bad option, bad superblock on /dev/mapper/MadaaServer-root,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so

Okay, so I decided to try to remove the PV of the RAID from the LG.

$ vgchange -an
/dev/mapper/MadaaServer-root-missing_0_0: read failed after 0 of 4096 at 0: Input/output error
/dev/MadaaServer/root: read failed after 0 of 4096 at 0: Input/output error
/dev/mapper/MadaaServer-swap_1-missing_0_0: read failed after 0 of 4096 at 0: Input/output error
/dev/MadaaServer/swap_1: read failed after 0 of 4096 at 0: Input/output error
/dev/mapper/MadaaServer-root-missing_0_0: read failed after 0 of 4096 at 488984477696: Input/output error
/dev/mapper/MadaaServer-root-missing_0_0: read failed after 0 of 4096 at 488984535040: Input/output error
/dev/mapper/MadaaServer-root-missing_0_0: read failed after 0 of 4096 at 0: Input/output error
/dev/mapper/MadaaServer-root-missing_0_0: read failed after 0 of 4096 at 4096: Input/output error
/dev/mapper/MadaaServer-root-missing_0_0: read failed after 0 of 4096 at 0: Input/output error
/dev/MadaaServer/root: read failed after 0 of 4096 at 488984477696: Input/output error
/dev/MadaaServer/root: read failed after 0 of 4096 at 488984535040: Input/output error
/dev/MadaaServer/root: read failed after 0 of 4096 at 0: Input/output error
/dev/MadaaServer/root: read failed after 0 of 4096 at 4096: Input/output error
/dev/MadaaServer/root: read failed after 0 of 4096 at 0: Input/output error
/dev/mapper/MadaaServer-swap_1-missing_0_0: read failed after 0 of 4096 at 10808655872: Input/output error
/dev/mapper/MadaaServer-swap_1-missing_0_0: read failed after 0 of 4096 at 10808713216: Input/output error
/dev/mapper/MadaaServer-swap_1-missing_0_0: read failed after 0 of 4096 at 0: Input/output error
/dev/mapper/MadaaServer-swap_1-missing_0_0: read failed after 0 of 4096 at 4096: Input/output error
/dev/mapper/MadaaServer-swap_1-missing_0_0: read failed after 0 of 4096 at 0: Input/output error
/dev/MadaaServer/swap_1: read failed after 0 of 4096 at 10808655872: Input/output error
/dev/MadaaServer/swap_1: read failed after 0 of 4096 at 10808713216: Input/output error
/dev/MadaaServer/swap_1: read failed after 0 of 4096 at 0: Input/output error
/dev/MadaaServer/swap_1: read failed after 0 of 4096 at 4096: Input/output error
/dev/MadaaServer/swap_1: read failed after 0 of 4096 at 0: Input/output error
Couldn't find device with uuid '5YzXQo-8F15-lq66-O316-nufN-hlWJ-T7nW7O'.
/dev/mapper/MadaaServer-root-missing_0_0: read failed after 0 of 4096 at 0: Input/output error
/dev/MadaaServer/root: read failed after 0 of 4096 at 0: Input/output error
/dev/mapper/MadaaServer-swap_1-missing_0_0: read failed after 0 of 4096 at 0: Input/output error
/dev/MadaaServer/swap_1: read failed after 0 of 4096 at 0: Input/output error
Couldn't find device with uuid '5YzXQo-8F15-lq66-O316-nufN-hlWJ-T7nW7O'.
/dev/mapper/MadaaServer-root-missing_0_0: stat failed: No such file or directory
Path /dev/mapper/MadaaServer-root-missing_0_0 no longer valid for device(252,0)
/dev/block/252:0: stat failed: No such file or directory
Path /dev/block/252:0 no longer valid for device(252,0)
/dev/disk/by-id/dm-uuid-LVM-9qiT5OFMvSrlu8Kn28sZ6E3I5mJ3mqQ2biNPC0KnjwxEEYOLOy o1SgU2Zv2eJvPm-missing_0_0: stat failed: No such file or directory
Path /dev/disk/by-id/dm-uuid-LVM-9qiT5OFMvSrlu8Kn28sZ6E3I5mJ3mqQ2biNPC0KnjwxEEYOLOy o1SgU2Zv2eJvPm-missing_0_0 no longer valid for device(252,0)
/dev/disk/by-id/dm-name-MadaaServer-root-missing_0_0: stat failed: No such file or directory
Path /dev/disk/by-id/dm-name-MadaaServer-root-missing_0_0 no longer valid for device(252,0)
/dev/MadaaServer/root: stat failed: No such file or directory
Path /dev/MadaaServer/root no longer valid for device(252,1)
/dev/disk/by-id/dm-uuid-LVM-9qiT5OFMvSrlu8Kn28sZ6E3I5mJ3mqQ2biNPC0KnjwxEEYOLOy o1SgU2Zv2eJvPm: stat failed: No such file or directory
Path /dev/disk/by-id/dm-uuid-LVM-9qiT5OFMvSrlu8Kn28sZ6E3I5mJ3mqQ2biNPC0KnjwxEEYOLOy o1SgU2Zv2eJvPm no longer valid for device(252,1)
/dev/disk/by-id/dm-name-MadaaServer-root: stat failed: No such file or directory
Path /dev/disk/by-id/dm-name-MadaaServer-root no longer valid for device(252,1)
/dev/block/252:1: stat failed: No such file or directory
Path /dev/block/252:1 no longer valid for device(252,1)
/dev/mapper/MadaaServer-root: stat failed: No such file or directory
Path /dev/mapper/MadaaServer-root no longer valid for device(252,1)
/dev/mapper/MadaaServer-swap_1-missing_0_0: read failed after 0 of 4096 at 0: Input/output error
/dev/MadaaServer/swap_1: read failed after 0 of 4096 at 0: Input/output error
Couldn't find device with uuid '5YzXQo-8F15-lq66-O316-nufN-hlWJ-T7nW7O'.
/dev/disk/by-id/dm-name-MadaaServer-root-missing_0_0: stat failed: No such file or directory
Path /dev/disk/by-id/dm-name-MadaaServer-root-missing_0_0 no longer valid for device(252,0)
/dev/mapper/MadaaServer-root: stat failed: No such file or directory
Path /dev/mapper/MadaaServer-root no longer valid for device(252,1)
/dev/mapper/MadaaServer-swap_1-missing_0_0: read failed after 0 of 4096 at 0: Input/output error
/dev/MadaaServer/swap_1: read failed after 0 of 4096 at 0: Input/output error
Couldn't find device with uuid '5YzXQo-8F15-lq66-O316-nufN-hlWJ-T7nW7O'.
/dev/disk/by-id/dm-name-MadaaServer-root-missing_0_0: stat failed: No such file or directory
Path /dev/disk/by-id/dm-name-MadaaServer-root-missing_0_0 no longer valid for device(252,0)
/dev/mapper/MadaaServer-root: stat failed: No such file or directory
Path /dev/mapper/MadaaServer-root no longer valid for device(252,1)
/dev/mapper/MadaaServer-swap_1-missing_0_0: stat failed: No such file or directory
Path /dev/mapper/MadaaServer-swap_1-missing_0_0 no longer valid for device(252,2)
/dev/disk/by-id/dm-name-MadaaServer-swap_1-missing_0_0: stat failed: No such file or directory
Path /dev/disk/by-id/dm-name-MadaaServer-swap_1-missing_0_0 no longer valid for device(252,2)
/dev/block/252:2: stat failed: No such file or directory
Path /dev/block/252:2 no longer valid for device(252,2)
/dev/disk/by-id/dm-uuid-LVM-9qiT5OFMvSrlu8Kn28sZ6E3I5mJ3mqQ2iiA7qa5BrdwOIVMfq1 LRiTegMvp5WDHw-missing_0_0: stat failed: No such file or directory
Path /dev/disk/by-id/dm-uuid-LVM-9qiT5OFMvSrlu8Kn28sZ6E3I5mJ3mqQ2iiA7qa5BrdwOIVMfq1 LRiTegMvp5WDHw-missing_0_0 no longer valid for device(252,2)
/dev/MadaaServer/swap_1: stat failed: No such file or directory
Path /dev/MadaaServer/swap_1 no longer valid for device(252,3)
/dev/disk/by-id/dm-name-MadaaServer-swap_1: stat failed: No such file or directory
Path /dev/disk/by-id/dm-name-MadaaServer-swap_1 no longer valid for device(252,3)
/dev/disk/by-id/dm-uuid-LVM-9qiT5OFMvSrlu8Kn28sZ6E3I5mJ3mqQ2iiA7qa5BrdwOIVMfq1 LRiTegMvp5WDHw: stat failed: No such file or directory
Path /dev/disk/by-id/dm-uuid-LVM-9qiT5OFMvSrlu8Kn28sZ6E3I5mJ3mqQ2iiA7qa5BrdwOIVMfq1 LRiTegMvp5WDHw no longer valid for device(252,3)
/dev/block/252:3: stat failed: No such file or directory
Path /dev/block/252:3 no longer valid for device(252,3)
/dev/mapper/MadaaServer-swap_1: stat failed: No such file or directory
Path /dev/mapper/MadaaServer-swap_1 no longer valid for device(252,3)
Couldn't find device with uuid '5YzXQo-8F15-lq66-O316-nufN-hlWJ-T7nW7O'.
0 logical volume(s) in volume group "MadaaServer" now active

Trying to remove the physical unknown device:

$ vgreduce --removemissing MadaaServer
Couldn't find device with uuid '5YzXQo-8F15-lq66-O316-nufN-hlWJ-T7nW7O'.
Couldn't find device with uuid '5YzXQo-8F15-lq66-O316-nufN-hlWJ-T7nW7O'.
WARNING: Partial LV root needs to be repaired or removed.
WARNING: Partial LV swap_1 needs to be repaired or removed.
WARNING: There are still partial LVs in VG MadaaServer.
To remove them unconditionally use: vgreduce --removemissing --force.
Proceeding to remove empty missing PVs.

Here I stopped. Should I use the --force directive or not? How can I repair the LV root afterwards?

Do all the read fails and i/o errors mean that the pvmove command which finished with ABORTING after 100% actually corrupted the original /dev/sda?

[sbh77]

Having a backup of the system, I did a vgcfgrestore of the backup inside /etc/lvm/backup.
It seemed to restore the LVM system to its previous definitions, but when trying to mount, I get:

Killed

trying to mount again hangs with no output...

Restarting shows again the same situation after following the guide until the pvmove command.