How To Set Up Software RAID1 On A Running System (Incl. GRUB Configuration) (Fedora 8)

Version 1.0
Author: Falko Timme
Last edited 01/30/2008

This guide explains how to set up software RAID1 on an already running Fedora 8 system. The GRUB bootloader will be configured in such a way that the system will still be able to boot if one of the hard drives fails (no matter which one).

I do not issue any guarantee that this will work for you!

 

1 Preliminary Note

In this tutorial I'm using a Fedora 8 system with two hard drives, /dev/sda and /dev/sdb which are identical in size. /dev/sdb is currently unused, and /dev/sda has the following partitions:

  • /dev/sda1: /boot partition, ext3;
  • /dev/sda2: swap;
  • /dev/sda3: / partition, ext3

In the end I want to have the following situation:

  • /dev/md0 (made up of /dev/sda1 and /dev/sdb1): /boot partition, ext3;
  • /dev/md1 (made up of /dev/sda2 and /dev/sdb2): swap;
  • /dev/md2 (made up of /dev/sda3 and /dev/sdb3): / partition, ext3

This is the current situation:

df -h

[root@server1 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             4.3G  2.4G  1.8G  58% /
/dev/sda1              99M   12M   83M  13% /boot
tmpfs                 185M     0  185M   0% /dev/shm
[root@server1 ~]#

fdisk -l

[root@server1 ~]# fdisk -l

Disk /dev/sda: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x0007b217

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14          77      514080   82  Linux swap / Solaris
/dev/sda3              78         652     4618687+  83  Linux

Disk /dev/sdb: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Disk /dev/sdb doesn't contain a valid partition table
[root@server1 ~]#

 

2 Installing mdadm

The most important tool for setting up RAID is mdadm. Let's install it like this:

yum install mkinitrd mdadm

Afterwards, we load a few kernel modules (to avoid a reboot):

modprobe linear
modprobe multipath
modprobe raid0
modprobe raid1
modprobe raid5
modprobe raid6
modprobe raid10

Now run

cat /proc/mdstat

The output should look as follows:

[root@server1 ~]# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
unused devices: <none>
[root@server1 ~]#

 

3 Preparing /dev/sdb

To create a RAID1 array on our already running system, we must prepare the /dev/sdb hard drive for RAID1, then copy the contents of our /dev/sda hard drive to it, and finally add /dev/sda to the RAID1 array.

First, we copy the partition table from /dev/sda to /dev/sdb so that both disks have exactly the same layout:

sfdisk -d /dev/sda | sfdisk /dev/sdb

The output should be as follows:

[root@server1 ~]# sfdisk -d /dev/sda | sfdisk /dev/sdb
Checking that no-one is using this disk right now ...
OK

Disk /dev/sdb: 652 cylinders, 255 heads, 63 sectors/track

sfdisk: ERROR: sector 0 does not have an msdos signature
 /dev/sdb: unrecognized partition table type
Old situation:
No partitions found
New situation:
Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/sdb1   *        63    208844     208782  83  Linux
/dev/sdb2        208845   1237004    1028160  82  Linux swap / Solaris
/dev/sdb3       1237005  10474379    9237375  83  Linux
/dev/sdb4             0         -          0   0  Empty
Successfully wrote the new partition table

Re-reading the partition table ...

If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
to zero the first 512 bytes:  dd if=/dev/zero of=/dev/foo7 bs=512 count=1
(See fdisk(8).)
[root@server1 ~]#

The command

fdisk -l

should now show that both HDDs have the same layout:

[root@server1 ~]# fdisk -l

Disk /dev/sda: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x0007b217

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14          77      514080   82  Linux swap / Solaris
/dev/sda3              78         652     4618687+  83  Linux

Disk /dev/sdb: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          13      104391   83  Linux
/dev/sdb2              14          77      514080   82  Linux swap / Solaris
/dev/sdb3              78         652     4618687+  83  Linux
[root@server1 ~]#

Next we must change the partition type of our three partitions on /dev/sdb to Linux raid autodetect:

fdisk /dev/sdb

[root@server1 ~]# fdisk /dev/sdb

Command (m for help): 
<-- m
Command action
   a   toggle a bootable flag
   b   edit bsd disklabel
   c   toggle the dos compatibility flag
   d   delete a partition
   l   list known partition types
   m   print this menu
   n   add a new partition
   o   create a new empty DOS partition table
   p   print the partition table
   q   quit without saving changes
   s   create a new empty Sun disklabel
   t   change a partition's system id
   u   change display/entry units
   v   verify the partition table
   w   write table to disk and exit
   x   extra functionality (experts only)

Command (m for help): 
<-- t
Partition number (1-4): 
<-- 1
Hex code (type L to list codes): 
<-- L

 0  Empty           1e  Hidden W95 FAT1 80  Old Minix       be  Solaris boot
 1  FAT12           24  NEC DOS         81  Minix / old Lin bf  Solaris
 2  XENIX root      39  Plan 9          82  Linux swap / So c1  DRDOS/sec (FAT-
 3  XENIX usr       3c  PartitionMagic  83  Linux           c4  DRDOS/sec (FAT-
 4  FAT16 <32M      40  Venix 80286     84  OS/2 hidden C:  c6  DRDOS/sec (FAT-
 5  Extended        41  PPC PReP Boot   85  Linux extended  c7  Syrinx
 6  FAT16           42  SFS             86  NTFS volume set da  Non-FS data
 7  HPFS/NTFS       4d  QNX4.x          87  NTFS volume set db  CP/M / CTOS / .
 8  AIX             4e  QNX4.x 2nd part 88  Linux plaintext de  Dell Utility
 9  AIX bootable    4f  QNX4.x 3rd part 8e  Linux LVM       df  BootIt
 a  OS/2 Boot Manag 50  OnTrack DM      93  Amoeba          e1  DOS access
 b  W95 FAT32       51  OnTrack DM6 Aux 94  Amoeba BBT      e3  DOS R/O
 c  W95 FAT32 (LBA) 52  CP/M            9f  BSD/OS          e4  SpeedStor
 e  W95 FAT16 (LBA) 53  OnTrack DM6 Aux a0  IBM Thinkpad hi eb  BeOS fs
 f  W95 Ext'd (LBA) 54  OnTrackDM6      a5  FreeBSD         ee  EFI GPT
10  OPUS            55  EZ-Drive        a6  OpenBSD         ef  EFI (FAT-12/16/
11  Hidden FAT12    56  Golden Bow      a7  NeXTSTEP        f0  Linux/PA-RISC b
12  Compaq diagnost 5c  Priam Edisk     a8  Darwin UFS      f1  SpeedStor
14  Hidden FAT16 <3 61  SpeedStor       a9  NetBSD          f4  SpeedStor
16  Hidden FAT16    63  GNU HURD or Sys ab  Darwin boot     f2  DOS secondary
17  Hidden HPFS/NTF 64  Novell Netware  b7  BSDI fs         fd  Linux raid auto
18  AST SmartSleep  65  Novell Netware  b8  BSDI swap       fe  LANstep
1b  Hidden W95 FAT3 70  DiskSecure Mult bb  Boot Wizard hid ff  BBT
1c  Hidden W95 FAT3 75  PC/IX
Hex code (type L to list codes): 
<-- fd
Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help): 
<-- t
Partition number (1-4): 
<-- 2
Hex code (type L to list codes): 
<-- fd
Changed system type of partition 2 to fd (Linux raid autodetect)

Command (m for help): 
<-- t
Partition number (1-4): 
<-- 3
Hex code (type L to list codes): 
<-- fd
Changed system type of partition 3 to fd (Linux raid autodetect)

Command (m for help): 
<-- w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@server1 ~]#

To make sure that there are no remains from previous RAID installations on /dev/sdb, we run the following commands:

mdadm --zero-superblock /dev/sdb1
mdadm --zero-superblock /dev/sdb2
mdadm --zero-superblock /dev/sdb3

If there are no remains from previous RAID installations, each of the above commands will throw an error like this one (which is nothing to worry about):

[root@server1 ~]# mdadm --zero-superblock /dev/sdb1
mdadm: Unrecognised md component device - /dev/sdb1
[root@server1 ~]#

Otherwise the commands will not display anything at all.

Share this page:

5 Comment(s)

Add comment

Comments

From: Anonymous at: 2010-02-09 18:29:32

Great walkthrough overall.  Another forum commented it's a good idea to change fstab/mtab on the RAID, but you have to change them before creating the kernel image.  So change them back on /mnt/sda to ensure the system comes up.

My big gotcha: disable SELinux.  After following the instructions and rebooting, I got "/bin/bash: Permission denied" while trying to ssh into the system.

From: Brandon Checketts at: 2010-01-25 21:16:12

The default speed limits on CentOS (and presumably other distros) have the RAID synchronization run at a relatively slow rate.  It tries to minimize disk I/O and the CPU utilization, but in many cases that will make it take seemingly forever for a modern-sized hard drive.

Use this command to increase the speed at which the drives sync to 100 MB/s

 

[root@host ~]# echo 100000 > /proc/sys/dev/raid/speed_limit_min

From: Sverre at: 2010-08-04 06:19:07

After following the instructions in CentOS 5.2, I'd get a kernel panic.

I logged in with the LiveCD and re-mounted the volumes and temp filesystems:

 

mdadm --examine --scan /dev/sdb1 >> /etc/mdadm.conf
mdadm --examine --scan /dev/sdb2 >> /etc/mdadm.conf

lvm vgchange -ay VolGroup01

mount /dev/VolGroup01/LogVol00 /mnt/sysimage
mount /dev/md0 /mnt/sysimage/boot

mount -t sysfs sys /mnt/sysimage/sys

I had to edit /boot/grub/device.map (I added (hd1) /dev/sdb), then I had to delete the contents of /etc/blkid and re-run blkid, then I had to delete /etc/lvm/cache/.cache and run vgscan, then I had to edit /etc/sysconfig/grub (replace sda with md0).

I also updated /root/anaconda.ks - but I don't think that makes any difference unless you boot the LiveCD into rescue mode?

 THEN I did initrd and grub

From: SmogMonkey at: 2013-05-14 20:15:33

Just in case someone runs across this (I happened to be running CentOS 5), make sure you update bash (yum update bash), according to RedHat the issue is with bash not mkinitrd.  I updated bash and was able to complete the commands above.  Error output:
mkinitrd /boot/initrd-`uname -r`.img `uname -r`
/sbin/mkinitrd: line 489: syntax error in conditional expression: unexpected token `('
/sbin/mkinitrd: line 489: syntax error near `^(d'
/sbin/mkinitrd: line 489: `        if [[ "$device" =~ ^(dm-|mapper/|mpath/) ]]; then' 

So far so good, glad I found this guide 

From: stu at: 2009-08-28 12:44:14

This is a great tutorial for those first dipping into raid on linux. A few comments though....

A short abstract would help. It took me a while to realize that what was going on was this:

Make a new array, copy your existing drive's data to it, make it bootable, boot it, blow away your original drive by making it a part of the new array and let it sync over.

My goal was to make my new drive be the new part of the array and get synced over. I didn't realize that you can't just make a working disk part of an array, you have to start with a new disk and then sync over to it.

The other thing I wanted to add was I had a problem took me a week to figure out.

 I have three disks in the machine, I was only making an array out of one partition.

When the machine was booted normally, grub's perspective of hd0 was sda, hd1 was sdb and hd2 was sdc. Makes sense.

So when I tried to make sda and sdc my array, naturally I made the grub menu.lst say root(hd2,0)

Error 15 file not found.

Then after a week of trying everything I could think of, I finally got the bright idea to boot grub and get to the command line.

After some tab completion experiments I realized that on boot hd1 and hd2 are swapped.

So obviously it couldn't find the kernel or initrd because it was looking at the wrong disk. I set root(hd1,0) and voila. Boot to array.

No idea why it does that, but now I've got it working.

Thanks again for a great tutorial.