Go Back   HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials > Linux Forums > Installation/Configuration

Do you like HowtoForge? Please consider supporting us by becoming a subscriber.
Reply
 
Thread Tools Display Modes
  #1  
Old 24th February 2013, 14:06
zoranpantic zoranpantic is offline
Junior Member
 
Join Date: Feb 2013
Posts: 1
Thanks: 0
Thanked 0 Times in 0 Posts
Default software RAID 1 mixed up after disk fall-out

Hi All,

we have a Debian based LAMP server (Debian-60-squeeze-64), physical, with 2 x 3 TB harddisks configured with 4 partitions in a software RAID 1 mirror.

One day, we get the following warning:
md3 : active raid1 sda4[0](F) sdb4[1]
1822442815 blocks super 1.2 [2/1] [_U]
md2 : active raid1 sda3[0](F) sdb3[1]
1073740664 blocks super 1.2 [2/1] [_U]
md1 : active raid1 sda2[0](F) sdb2[1]
524276 blocks super 1.2 [2/1] [_U]
md0 : active raid1 sda1[0](F) sdb1[1]
33553336 blocks super 1.2 [2/1] [_U]

Seemed like sda is out. The server is then shut down, and a SMART check is run on both harddisks. No errors found.

After the server is booted to the OS again, the sda is visible, but the mirror is degraded, and all partitions are "mixed":
md124 : active (auto-read-only) raid1 sda1[0]
33553336 blocks super 1.2 [2/1] [U_]
md125 : active (auto-read-only) raid1 sda2[0]
524276 blocks super 1.2 [2/1] [U_]
md126 : active (auto-read-only) raid1 sda3[0]
1073740664 blocks super 1.2 [2/1] [U_]
md127 : active raid1 sda4[0]
1822442815 blocks super 1.2 [2/1] [U_]
md3 : active (auto-read-only) raid1 sdb4[1]
1822442815 blocks super 1.2 [2/1] [_U]
md2 : active raid1 sdb3[1]
1073740664 blocks super 1.2 [2/1] [_U]
md1 : active (auto-read-only) raid1 sdb2[1]
524276 blocks super 1.2 [2/1] [_U]
md0 : active (auto-read-only) raid1 sdb1[1]
33553336 blocks super 1.2 [2/1] [_U]
I decide to try to remove the mirror from sda, keeping it only on sdb, but without success:
# mdadm /dev/md127 -r /dev/sda4
mdadm: hot remove failed for /dev/sda4: Device or resource busy
# mdadm /dev/md127 --fail /dev/sda4
mdadm: set /dev/sda4 faulty in /dev/md127
# mdadm /dev/md127 -r /dev/sda4
mdadm: hot remove failed for /dev/sda4: Device or resource busy

What I suppose is that the mirrored partitions were kept on sda, but when it came up and running after the reboot, it could "see" that the partitions are out of sync, and the partitions are kept on both sda og sdb now.

How do I remove the partitions on the sda, and resync them from the sdb again, without loosing any data?

There is also another thing that happened - some partitions were not mounted after the reboot.

Here is the hard disk configuration before mirroring degraded:
Filesystem Size Used Avail Use% Mounted on
---------------------------------------------------------
/dev/md2 1016G 58G 908G 6% /
tmpfs 32G 0 32G 0% /lib/init/rw
udev 32G 172K 32G 1% /dev
tmpfs 32G 0 32G 0% /dev/shm
/dev/md1 496M 38M 434M 8% /boot
/dev/md3 1.7T 554G 1.1T 34% /home

... and hard disk configuration after the degradation & reboot:
Filesystem Size Used Avail Use% Mounted on
---------------------------------------------------------
/dev/md2 1016G 58G 908G 6% /
tmpfs 32G 0 32G 0% /lib/init/rw
udev 32G 188K 32G 1% /dev
tmpfs 32G 0 32G 0% /dev/shm
Looks like things from md1 and md3 "vanished", at least they cannot be seen from the inside of the OS.

How do I get the "lost" partitions mounted again, and how do I ensure that the disk configuration is bootable, both from sda and sdb?

Thanks in advance, and regards from

Zoran
Reply With Quote
Sponsored Links
Reply

Bookmarks

Tags
mirroring, software raid

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
CentOS 6.2 boot problem running software Raid1 bzzik HOWTO-Related Questions 1 16th February 2012 10:56
Intel ESRT2, Matrix RAID or Ubuntu Software Raid? dalitso Installation/Configuration 0 4th October 2010 17:19
Adding non-RAID backup disk to system with RAID 1... BorderAmigos Installation/Configuration 1 20th February 2010 20:50
software raid problem Daisy Installation/Configuration 5 6th January 2010 13:02
SATA Software RAID 1 on Linux mphayesuk Installation/Configuration 2 21st February 2006 11:31


All times are GMT +2. The time now is 10:14.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.