software RAID 1 mixed up after disk fall-out
 

Hi All,

we have a Debian based LAMP server (Debian-60-squeeze-64), physical, with 2 x 3 TB harddisks configured with 4 partitions in a software RAID 1 mirror.

One day, we get the following warning:

md3 : active raid1 sda4[0](F) sdb4[1]
1822442815 blocks super 1.2 [2/1] [_U]
md2 : active raid1 sda3[0](F) sdb3[1]
1073740664 blocks super 1.2 [2/1] [_U]
md1 : active raid1 sda2[0](F) sdb2[1]
524276 blocks super 1.2 [2/1] [_U]
md0 : active raid1 sda1[0](F) sdb1[1]
33553336 blocks super 1.2 [2/1] [_U]

Seemed like sda is out. The server is then shut down, and a SMART check is run on both harddisks. No errors found.

After the server is booted to the OS again, the sda is visible, but the mirror is degraded, and all partitions are "mixed":

md124 : active (auto-read-only) raid1 sda1[0]
33553336 blocks super 1.2 [2/1] [U_]
md125 : active (auto-read-only) raid1 sda2[0]
524276 blocks super 1.2 [2/1] [U_]
md126 : active (auto-read-only) raid1 sda3[0]
1073740664 blocks super 1.2 [2/1] [U_]
md127 : active raid1 sda4[0]
1822442815 blocks super 1.2 [2/1] [U_]
md3 : active (auto-read-only) raid1 sdb4[1]
1822442815 blocks super 1.2 [2/1] [_U]
md2 : active raid1 sdb3[1]
1073740664 blocks super 1.2 [2/1] [_U]
md1 : active (auto-read-only) raid1 sdb2[1]
524276 blocks super 1.2 [2/1] [_U]
md0 : active (auto-read-only) raid1 sdb1[1]
33553336 blocks super 1.2 [2/1] [_U]

I decide to try to remove the mirror from sda, keeping it only on sdb, but without success:

# mdadm /dev/md127 -r /dev/sda4
mdadm: hot remove failed for /dev/sda4: Device or resource busy
# mdadm /dev/md127 --fail /dev/sda4
mdadm: set /dev/sda4 faulty in /dev/md127
# mdadm /dev/md127 -r /dev/sda4
mdadm: hot remove failed for /dev/sda4: Device or resource busy

What I suppose is that the mirrored partitions were kept on sda, but when it came up and running after the reboot, it could "see" that the partitions are out of sync, and the partitions are kept on both sda og sdb now.

How do I remove the partitions on the sda, and resync them from the sdb again, without loosing any data?

There is also another thing that happened - some partitions were not mounted after the reboot.

Here is the hard disk configuration before mirroring degraded:

Filesystem Size Used Avail Use% Mounted on
---------------------------------------------------------
/dev/md2 1016G 58G 908G 6% /
tmpfs 32G 0 32G 0% /lib/init/rw
udev 32G 172K 32G 1% /dev
tmpfs 32G 0 32G 0% /dev/shm
/dev/md1 496M 38M 434M 8% /boot
/dev/md3 1.7T 554G 1.1T 34% /home

... and hard disk configuration after the degradation & reboot:

Filesystem Size Used Avail Use% Mounted on
---------------------------------------------------------
/dev/md2 1016G 58G 908G 6% /
tmpfs 32G 0 32G 0% /lib/init/rw
udev 32G 188K 32G 1% /dev
tmpfs 32G 0 32G 0% /dev/shm

Looks like things from md1 and md3 "vanished", at least they cannot be seen from the inside of the OS.

How do I get the "lost" partitions mounted again, and how do I ensure that the disk configuration is bootable, both from sda and sdb?

Thanks in advance, and regards from

Zoran