9 Testing

Now let's simulate a hard drive failure. It doesn't matter if you select /dev/hda or /dev/hdb here. In this example I assume that /dev/hdb has failed.

To simulate the hard drive failure, you can either shut down the system and remove /dev/hdb from the system, or you (soft-)remove it like this:

mdadm --manage /dev/md0 --fail /dev/hdb1
mdadm --manage /dev/md1 --fail /dev/hdb5
mdadm --manage /dev/md2 --fail /dev/hdb6

mdadm --manage /dev/md0 --remove /dev/hdb1
mdadm --manage /dev/md1 --remove /dev/hdb5
mdadm --manage /dev/md2 --remove /dev/hdb6

Shut down the system:

shutdown -h now

Then put in a new /dev/hdb drive (if you simulate a failure of /dev/hda, you should now put /dev/hdb in /dev/hda's place and connect the new HDD as /dev/hdb!) and boot the system. It should still start without problems.

Now run

cat /proc/mdstat

and you should see that we have a degraded array:

[root@server1 ~]# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 hda5[0]
      417536 blocks [2/1] [U_]

md0 : active raid1 hda1[0]
      176576 blocks [2/1] [U_]

md2 : active raid1 hda6[0]
      4642688 blocks [2/1] [U_]

unused devices: <none>
[root@server1 ~]#

The output of

fdisk -l

should look as follows:

[root@server1 ~]# fdisk -l

Disk /dev/hda: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/hda1   *           1          22      176683+  fd  Linux raid autodetect
/dev/hda2              23         652     5060475    5  Extended
/dev/hda5              23          74      417658+  fd  Linux raid autodetect
/dev/hda6              75         652     4642753+  fd  Linux raid autodetect

Disk /dev/hdb: 5368 MB, 5368709120 bytes
16 heads, 63 sectors/track, 10402 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Disk identifier: 0x00000000

Disk /dev/hdb doesn't contain a valid partition table

Disk /dev/md2: 4754 MB, 4754112512 bytes
2 heads, 4 sectors/track, 1160672 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk identifier: 0x00000000

Disk /dev/md2 doesn't contain a valid partition table

Disk /dev/md0: 180 MB, 180813824 bytes
2 heads, 4 sectors/track, 44144 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk identifier: 0x00000000

Disk /dev/md0 doesn't contain a valid partition table

Disk /dev/md1: 427 MB, 427556864 bytes
2 heads, 4 sectors/track, 104384 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk identifier: 0x00000000

Disk /dev/md1 doesn't contain a valid partition table
[root@server1 ~]#

Now we copy the partition table of /dev/hda to /dev/hdb:

sfdisk -d /dev/hda | sfdisk --force /dev/hdb

[root@server1 ~]# sfdisk -d /dev/hda | sfdisk --force /dev/hdb
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Checking that no-one is using this disk right now ...
OK

Disk /dev/hdb: 10402 cylinders, 16 heads, 63 sectors/track

sfdisk: ERROR: sector 0 does not have an msdos signature
 /dev/hdb: unrecognized partition table type
Old situation:
No partitions found
New situation:
Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/hdb1   *        63    353429     353367  fd  Linux raid autodetect
/dev/hdb2        353430  10474379   10120950   5  Extended
/dev/hdb3             0         -          0   0  Empty
/dev/hdb4             0         -          0   0  Empty
/dev/hdb5        353493   1188809     835317  fd  Linux raid autodetect
/dev/hdb6       1188873  10474379    9285507  fd  Linux raid autodetect
Warning: partition 1 does not end at a cylinder boundary
Successfully wrote the new partition table

Re-reading the partition table ...

If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
to zero the first 512 bytes:  dd if=/dev/zero of=/dev/foo7 bs=512 count=1
(See fdisk(8).)
[root@server1 ~]#

Afterwards we remove any remains of a previous RAID array from /dev/hdb...

mdadm --zero-superblock /dev/hdb1
mdadm --zero-superblock /dev/hdb5
mdadm --zero-superblock /dev/hdb6

... and add /dev/hdb to the RAID array:

mdadm -a /dev/md0 /dev/hdb1
mdadm -a /dev/md1 /dev/hdb5
mdadm -a /dev/md2 /dev/hdb6

Now take a look at

cat /proc/mdstat

[root@server1 ~]# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 hdb5[2] hda5[0]
      417536 blocks [2/1] [U_]
        resync=DELAYED

md0 : active raid1 hdb1[1] hda1[0]
      176576 blocks [2/2] [UU]

md2 : active raid1 hdb6[2] hda6[0]
      4642688 blocks [2/1] [U_]
      [===========>.........]  recovery = 59.9% (2784512/4642688) finish=7.5min speed=4076K/sec

unused devices: <none>
[root@server1 ~]#

Wait until the synchronization has finished:

[root@server1 ~]# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 hdb5[1] hda5[0]
      417536 blocks [2/2] [UU]

md0 : active raid1 hdb1[1] hda1[0]
      176576 blocks [2/2] [UU]

md2 : active raid1 hdb6[1] hda6[0]
      4642688 blocks [2/2] [UU]

unused devices: <none>
[root@server1 ~]#

Then run

grub

and install the bootloader on both HDDs:

root (hd0,0)
setup (hd0)
root (hd1,0)
setup (hd1)
quit

That's it. You've just replaced a failed hard drive in your RAID1 array.

10 Links

• The Software-RAID Howto: http://tldp.org/HOWTO/Software-RAID-HOWTO.html
• Mandriva: http://www.mandriva.com