How To Set Up Software RAID1 On A Running System (Incl. GRUB2 Configuration) (Ubuntu 10.04) - Page 2

4 Creating Our RAID Arrays

Now let's create our RAID arrays /dev/md0, /dev/md1, and /dev/md2. /dev/sdb1 will be added to /dev/md0, /dev/sdb2 to /dev/md1, and /dev/sdb3 to /dev/md2. /dev/sda1, /dev/sda2, and /dev/sda3 can't be added right now (because the system is currently running on them), therefore we use the placeholder missing in the following three commands:

mdadm --create /dev/md0 --level=1 --raid-disks=2 missing /dev/sdb1
mdadm --create /dev/md1 --level=1 --raid-disks=2 missing /dev/sdb2
mdadm --create /dev/md2 --level=1 --raid-disks=2 missing /dev/sdb3

The command

cat /proc/mdstat

should now show that you have three degraded RAID arrays ([_U] or [U_] means that an array is degraded while [UU] means that the array is ok):

[email protected]:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 sdb3[1]
      4242368 blocks [2/1] [_U]

md1 : active raid1 sdb2[1]
      499648 blocks [2/1] [_U]

md0 : active raid1 sdb1[1]
      498624 blocks [2/1] [_U]

unused devices: <none>
[email protected]:~#

Next we create filesystems on our RAID arrays (ext4 on /dev/md0 and /dev/md2 and swap on /dev/md1):

mkfs.ext4 /dev/md0
mkswap /dev/md1
mkfs.ext4 /dev/md2

Next we must adjust /etc/mdadm/mdadm.conf (which doesn't contain any information about our new RAID arrays yet) to the new situation:

cp /etc/mdadm/mdadm.conf /etc/mdadm/mdadm.conf_orig
mdadm --examine --scan >> /etc/mdadm/mdadm.conf

Display the contents of the file:

cat /etc/mdadm/mdadm.conf

At the bottom of the file you should now see details about our three (degraded) RAID arrays:

# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays

# This file was auto-generated on Mon, 21 Jun 2010 13:21:00 +0200
# by mkconf $Id$
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=68686c40:b924278e:325ecf68:79913751
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=9719181e:3071f655:325ecf68:79913751
ARRAY /dev/md2 level=raid1 num-devices=2 UUID=c3360f0f:7f3d47ec:325ecf68:79913751

 

5 Adjusting The System To RAID1

Now let's mount /dev/md0 and /dev/md2 (we don't need to mount the swap array /dev/md1):

mkdir /mnt/md0
mkdir /mnt/md2
mount /dev/md0 /mnt/md0
mount /dev/md2 /mnt/md2

You should now find both arrays in the output of

mount
[email protected]:~# mount
/dev/sda3 on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
none on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
none on /dev type devtmpfs (rw,mode=0755)
none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
none on /dev/shm type tmpfs (rw,nosuid,nodev)
none on /var/run type tmpfs (rw,nosuid,mode=0755)
none on /var/lock type tmpfs (rw,noexec,nosuid,nodev)
none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
none on /var/lib/ureadahead/debugfs type debugfs (rw,relatime)
/dev/sda1 on /boot type ext4 (rw)
/dev/md0 on /mnt/md0 type ext4 (rw)
/dev/md2 on /mnt/md2 type ext4 (rw)
[email protected]:~#

Next we modify /etc/fstab. Comment out the current /, /boot, and swap partitions and add new lines for them where you replace the UUIDs with /dev/md0 (for the /boot partition), /dev/md1 (for the swap partition) and /dev/md2 (for the / partition) so that the file looks as follows:

vi /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid -o value -s UUID' to print the universally unique identifier
# for a device; this may be used with UUID= as a more robust way to name
# devices that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    nodev,noexec,nosuid 0       0
# / was on /dev/sda3 during installation
#UUID=48d65bba-0f02-44b4-8557-b508309b1963 /               ext4    errors=remount-ro 0       1
/dev/md2 /               ext4    errors=remount-ro 0       1
# /boot was on /dev/sda1 during installation
#UUID=e3a677ee-2db0-4a8a-8d6c-94715c8cd90f /boot           ext4    defaults        0       2
/dev/md0 /boot           ext4    defaults        0       2
# swap was on /dev/sda2 during installation
#UUID=1e27f700-ec54-4de9-9428-c6d47d7921f4 none            swap    sw              0       0
/dev/md1 none            swap    sw              0       0
/dev/fd0        /media/floppy0  auto    rw,user,noauto,exec,utf8 0       0

Next replace /dev/sda1 with /dev/md0 and /dev/sda3 with /dev/md2 in /etc/mtab:

vi /etc/mtab
/dev/md2 / ext4 rw,errors=remount-ro 0 0
proc /proc proc rw,noexec,nosuid,nodev 0 0
none /sys sysfs rw,noexec,nosuid,nodev 0 0
none /sys/fs/fuse/connections fusectl rw 0 0
none /sys/kernel/debug debugfs rw 0 0
none /sys/kernel/security securityfs rw 0 0
none /dev devtmpfs rw,mode=0755 0 0
none /dev/pts devpts rw,noexec,nosuid,gid=5,mode=0620 0 0
none /dev/shm tmpfs rw,nosuid,nodev 0 0
none /var/run tmpfs rw,nosuid,mode=0755 0 0
none /var/lock tmpfs rw,noexec,nosuid,nodev 0 0
none /lib/init/rw tmpfs rw,nosuid,mode=0755 0 0
none /var/lib/ureadahead/debugfs debugfs rw,relatime 0 0
/dev/md0 /boot ext4 rw 0 0
/dev/md0 /mnt/md0 ext4 rw 0 0
/dev/md2 /mnt/md2 ext4 rw 0 0

Now up to the GRUB2 boot loader. Create the file /etc/grub.d/09_swraid1_setup as follows:

cp /etc/grub.d/40_custom /etc/grub.d/09_swraid1_setup
vi /etc/grub.d/09_swraid1_setup
#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
menuentry 'Ubuntu, with Linux 2.6.32-21-server' --class ubuntu --class gnu-linux --class gnu --class os {
        recordfail
        insmod raid
        insmod mdraid
        insmod ext2
        set root='(md0)'
        linux   /vmlinuz-2.6.32-21-server root=/dev/md2 ro   quiet
        initrd  /initrd.img-2.6.32-21-server
}

Make sure you use the correct kernel version in the menuentry stanza (in the linux and initrd lines). You can find it out by running

uname -r

or by taking a look at the current menuentry stanzas in the ### BEGIN /etc/grub.d/10_linux ### section in /boot/grub/grub.cfg. Also make sure that you use root=/dev/md2 in the linux line.

The important part in our new menuentry stanza is the line set root='(md0)' - it makes sure that we boot from our RAID1 array /dev/md0 (which will hold the /boot partition) instead of /dev/sda or /dev/sdb which is important if one of our hard drives fails - the system will still be able to boot.

Run

update-grub

to write our new kernel stanza from /etc/grub.d/09_swraid1_setup to /boot/grub/grub.cfg.

Next, we adjust our ramdisk to the new situation:

update-initramfs -u

Now we copy the contents of /dev/sda1 and /dev/sda3 to /dev/md0 and /dev/md2 (which are mounted on /mnt/md0 and /mnt/md2):

cp -dpRx / /mnt/md2
cd /boot
cp -dpRx . /mnt/md0

 

6 Preparing GRUB2 (Part 1)

Afterwards we must make sure that the GRUB2 bootloader is installed on both hard drives, /dev/sda and /dev/sdb:

grub-install /dev/sda
grub-install /dev/sdb

Now we reboot the system and hope that it boots ok from our RAID arrays:

reboot
Share this page:

8 Comment(s)

Add comment

Please register in our forum first to comment.

Comments

By: Anonymous

Hello,

If problem when reboot, try :

menuentry 'Ubuntu, with Linux 2.6.32-21-server' --class ubuntu --class gnu-linux --class gnu --class os {
recordfail
insmod raid
insmod mdraid
insmod ext2
set root='(md0)'
linux /boot/vmlinuz-2.6.32-21-server root=/dev/md2 ro quiet
initrd /boot/initrd.img-2.6.32-21-server
}

it's better !

By: Steve M

Great article thanks, but I got totally stuck at the end of this page.  My system wouldn't boot from the RAID1 - got grub errors: file not found, no such disk, you need to load the kernel first.

The problem turned out to be that the system was set not to boot a degraded array - the setting is in /etc/initramfs-tools/conf.d/mdadm.

The fix is to run dpkg-reconfigure mdadm and choose Yes when it asks about booting degraded arrays.  Suggest doing this before you run update-grub above.

 

By: ecellingsworth

Let me first say that this is the only "how-to" guide I've been able to find that has up-to-date information for getting grub2 to work with mdraid. Thank you for that. I've been banging my head against the wall trying to get a boot manager installed on an array.

I had both of the problems described by the commentors above. First, after finishing the steps on this page and rebooting, grub threw a few errors: file not found, no such disk, you must load the kernel first. Through some trial and error I was able to determine the cause of each error.

For me, grub was unable to find the mdraid module, so the line "insmod mdraid" was returning the "no such file" error. I was able to remove this line without problems. I'm not sure of the difference between this module and the "raid" module, but it doesn't appear necessary (hopefully I don't find this to be untrue after a drive failure!).

Grub was unable to "set root = '(md0)'". I entered the command console to find it had a device called (md/0) listed instead. After googling around a bit (see links below comment), I've come to the conclusion that this is how grub labels raid devices with metadata 1.x (e.g. 1.2, as opposed to 0.90). After changing the reference to (md/0), grub was able to find the disk. The kernel option stays "/dev/md0".

Finally, I had to fix the location of the kernel and initrd which are found in the /boot directory, as suggested by the second commentor. Once the kernel was found, grub was able to load initrd and no longer complained that the kernel had to be loaded first. Voila! Grub was able to boot linux....or at least try to. At this point I ran into the problem that my mdadm was configured to prohibit booting degraded arrays. I followed the Steve M's advice and reconfigured my mdadm package to permit this (after which I reconfigured and installed grub on the partitions).

 I hope this information is useful. I'm not experienced with raid or grub. One final quick note about testing your array. If you unplug a drive and boot ok, then shutdown and reboot, you have to manually re-add the drive to the array using something like "mdadm /dev/md0 -a /dev/sda1". Beware that doing so requires a complete rebuild. So if it took you 3 hours to sync the two drives the first time (as it did me), expect to spend another 3 hours rebuilding every drive you test by unplugging.

For reference:

http://ubuntuforums.org/showthread.php?t=1681190

http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/maverick/grub2/maverick/view/head:/disk/raid.c#L584

http://www.techrepublic.com/blog/networking/testing-your-software-raid-be-prepared/387

By: Alan

This got me off to a good start but there are two minor differences when on Ubuntu 11.10.  First, there's no "mdraid" in grub in Ubuntu 11.10, you need to use "mdraid1x".  Second, like ecellingsworth pointed out, you need to use "md/0" in the grub config file.

 This worked for my Mythbuntu 11.10 installation, which doesn't support RAID from the installer.  The "alternative" Ubuntu CD does, but Mythbuntu doesn't offer one of those. :)

By: BotoX

If you try to use this on the testing deb (aka. wheezy) and updated grub you need to use insmod mdraid1x instead of mdraid or grub will fail to load the mdraid drivers and wont find your drives.

That just happened to me :/

By: RK

Hi Experts, I followed the setup for software raid, everything went fine but when i tried to copy the contents of /dev/sda1 and /dev/sda3 to /dev/md0 and /dev/md2 (which are mounted on /mnt/md0 and /mnt/md2): using cp -dpRx / /mnt/md2 It gave me error saying cp: cannot stat '/home/user/.gvfs' : permission denied i even tried with sudo, chmod, but didn't work, Plz reply how to fix this error...

By: MrWaloo

Thanks a lot for this tutorial it was really a good basis for me ;-)
I just wanted to show how does a grub2 entry look like for debian testing/wheezy (up to date on 01/18/2013):
menuentry 'Debian GNU/Linux, with Linux 3.2.0-4-amd64' --class debian --class gnu-linux --class gnu --class os {
	insmod gzio
	insmod raid
	insmod mdraid1x
	insmod part_msdos
	insmod part_msdos
	insmod ext2
	set root='(mduuid/e165b8a7ac19f29e8800e7b4f7fb3a5c)'
	search --no-floppy --fs-uuid --set=root 5871e838-3d53-47a6-8ec8-6edeb6998faf
	linux	/boot/vmlinuz-3.2.0-4-amd64 root=UUID=5871e838-3d53-47a6-8ec8-6edeb6998faf ro  quiet
	initrd	/boot/initrd.img-3.2.0-4-amd64
}
The mduuid can be found in "/dev/disk/by-id/md-uuid-*" (in this file delete the ":" of mduuid's)
The UUID can be found in "/dev/disk/by-uuid/*"
 

In order to set up grub2, i used chroot as follow (the copy must allready have been done):
mount -t proc none /mnt/md0/proc
mount -o bind /dev /mnt/md0/dev
mount -o bind /sys /mnt/md0/sys
chroot /mnt/md0
And then the 4 commands in the chroot "update-grub", "update-initramfs -u", "grub-install /dev/sda" and "grub-install /dev/sdb".
With this, grub should be correctly generated.

By: arcasys

In wheezy, the following issues came up for me (the first two have already been reported, I list them for completeness)

  •  mdraid must be replaced with mdraid1x in /etc/grub.d/09_swraid1_setup
  • recordfail must be removed from this file and any other of the files in /etc/grub.d (not supported anymore)
  • /etc/mtab cannot be edited because it is now a symbolic link to /proc/mounts. To revert /etc/mtab to an editable file follow https://www.debian.org/releases/stable/i386/release-notes/ch-information.en.html#mtab
    and change the permissions.
  • grub_install --modules="raid mdraid1x"
    The modules option might be irrelevant (I havn't tested without yet so I cannot say if it made the difference but it doesn't hurt.