1. Hi friends,


    What & why:
    I'm building this fileserver using Ubuntu Server 12.04.5 with RAID1 to make my files accessible, should the worse thing happen. I succeeded setting up Ubuntu Software RAID1 and it works just great (I think).
    But, I'm not quite satisfied yet! I want to build a system, where a failed diskdrive will not stop the access to the file's as well as making backups. I'm using 2U cases for all my servers. There's only two 5.25" bays in the front of these 2U cases, and I want the disks in this system to be easily replaced in case of a hardware defect. I'll show the case in the end of the post.
    So, what I'm asking all you for, is an idea how to make this FileServer with + BackUp. Which type of disks to use, SSD or ordinary harddrives. How to plcae what and where!?!?

    What do I need?
    1. I need to be able to change faulty diskdrives in case of a crash.
    2. I need to have diskspace for 10 copies/versions of backup. The 11'th backup will overwrite the oldest.

    My latest idea is to build it using:
    2 x ICY BOX Trayless Mobile Rack 3.5" SATA HDD (quick removal of faulty disk)
    2 x 2 SANDisk SSD 240Gb
    2 x 2 Western Digital 2Tb harddrives

    Install the ICY BOX Trayless Mobile Racks in the two available bays for easy front handling.
    Install the two 240Gb SANDisk SSD's in the Mobile Racks above, and setup RAID1 with swapfile and ext4 on the rest of the space.
    Install the two 2Tb harddrives from Western Digital internally, and then configurate these for backup.

    Issues & problems that needs to be solved:
    1. When a disk in Ubuntu Software RAID1 chrashes, you can't see the damage before rebooting the system. Using the known commands to check the array's/md's there's no indication of a problem before the systems is rebooted! That's what I noticed so far.
    2. How to set-up the server to be able to send the warning emails the "RAID Monitor Deamon" creates, when a array gets degraded. (related threads 2)
    3. How to set up the system/e.g. a shell script to make backup on to the two 2Tb harddrives. 10 copies before overwriting the first. I'm thinking that making these two disks run as standard disk/no RAID! But have a script copy the backup from the RAID1 to both these disks. This way we have a backup on 2 normal harddrives, not SSD. Much easier to do recovery on "real" harddrives intead of SSD. And, we've got 2 harddisk with each a copy of the backup.

    So, what am I asking!?!?
    A. Some guidence. Am I totally lost here? Is this the wrong way to do stuff like this??
    B. Suggestions on how to take content from an array, let's call it "MyData", make a backup as a .rar file on to the two "backup disks". A copy on each drive. Daily. A script maybe and CRON!?
    C. Echo to the "screen" (I use SSH/Putty from another computer to remotely control all servers), is a array has gone "degraded".
    D. * I'm currently working on the email warning (RAID Monitor Deamon) thanks to Srijan! But, if you have a suggestion on this matter, please don't hesitate to let us know ;)
    E. A suggestion to another setup. Another kind of RAID and/or another way to backup data.

    I've been working as a datarecovery agent for many years, which is the reason I would like the backups to be on a "normal" harddrive. There's so much more you can do right away having "lost data" from a "normal" harddrive than say a SSD drive. This is why I'm so hooked on getting the backup on to a couple of Western Digital disks. WD disks were the kind of harddrive I saw the fewest times. That's my opinion anyway ;)

    Related threads:
    http://www.howtoforge.com/forums/showthread.php?p=317901#poststop
    http://www.howtoforge.com/forums/showthread.php?p=317900#post317900
    http://www.howtoforge.com/forums/showthread.php?p=317619#poststop

    [​IMG]
     
    Last edited: Aug 25, 2014
  2. sjau

    sjau Local Meanie Moderator

    I have a similar setup, but I run 3 drives in Raid1.

    However I have an IcyBox with 5 bays. So what I do is use a script that

    1) copies parittion layout to newly added disk
    2) since I use 4TB drives and have EFI Mainboard, I need that EFI main parition, so I just DD that one
    3) Then I expand the Raid1 to 4 active disks and add the newly added drive to it

    I do that every month.

    Code:
    #!/bin/bash
    
    
    [[ $2 ]] || { ls -al /dev/disk/by-id; echo "You need to specifiy two arguments. Argument 1 is the origin drive from which partition layout and efi boot partition is copie, Argument two is the new/backup drive. E.g. ./script.sh sda sdd to copy from sda to sdd" >&2; exit 1; }
    
    
    orig="/dev/${1}"
    dest="/dev/${2}"
    
    
    read -p "Clone system from ${orig} to ${dest}. Press Y to continue: "
    echo # Just to move to a new line
    if [[ ! "${REPLY}" =~ ^[Yy]$ ]]
    then
        exit 1
    fi
    
    
    # Make partition layout
    sgdisk -R="${dest}" "${orig}"
    sgdisk -G "${dest}"
    
    
    # Clone the EFI Boot partition
    #dd if="${orig}1" of="${dest}1"
    
    
    # Install GRUB to dest
    #grub-install "${dest}"
    
    
    # Shut down VMs
    #for vm in {100..104}
    #do
    #       qm shutdown "${vm}"
    #done
    
    
    # Grow, Clone, Shrink RAID devices
    declare -A raidInfo
    raidInfo[0]="2"
    raidInfo[1]="3"
    raidInfo[2]="4"
    
    for i in "${!raidInfo[@]}"
    do
    
            mdadm --grow --force "/dev/md${i}" --raid-devices=4
            mdadm --zero-superblock "${dest}${raidInfo[$i]}"
            mdadm --manage "/dev/md${i}" --add --write-mostly "${dest}${raidInfo[$i]}"
            mdadm --wait "/dev/md${i}"
            sync
    done
    
    
    # Clone the EFI Boot partition
    dd if="${orig}1" of="${dest}1"
    
    # Install GRUB to dest
    grub-install "${dest}"
    
    
    for i in "${!raidInfo[@]}"
    do
            mdadm "/dev/md${i}" --fail "${dest}${raidInfo[$i]}"
            mdadm "/dev/md${i}" --remove "${dest}${raidInfo[$i]}"
            mdadm --grow "/dev/md${i}" --raid-devices=3
            echo ""
            echo "Finshed syncing /dev/md${i}"
            echo ""
            echo ""
            echo ""
    done
    
    
    # Restart VM, except the CA
    #for vm in {100..103}
    #do
    #       qm start "${vm}"
    #done
    
    you probably need sfdisk instead of sgdisk... not sure how the syntax is there... sgdisk is required if you have larger than 2TB drives.


    I also do daily backups within each of the VMs (the above just makes a hotcopy of the server). Here's the backup script I use for the ISPC/Mail Server:

    backup.sh
    Code:
    #!/bin/bash
    
    # USER VARIABLES
    backupDir="/backup"                                             # Folder for current and old backups
    doMysqlBackup="y"                                               # Set y/n if you want to make MySQL Backup also
    mysqlUser="root"                                                # MySQL User
    mysqlPasswd="..."                             # MySQL Password
    mysqlHost="localhost"                                   # MySQL Host
    mysqlDataFolder="/var/lib/mysql"                # Folder containing the binary mysql DBs
    mysqlBackupDir="/mysql_backup"                  # Folder for dumping .sql files and binary copies
    excludes="/root/backup_excludes"        # File containing exludes
    days="30"                                                               # After how many days shall the backups be deleted?
    
    
    
    ##############################################################################
    #                                                                            #
    #                            BELOW BE DRAGONS                                #
    #                                                                            #
    ##############################################################################
    
    
    
    # Check if important variables are set
    if [[ -z "${backupDir}" ]]
    then
            echo "The 'backupDir' var cannot be empty. Fix and try again."
            exit 1;
    fi
    if [[ "${doMysqlBackup}" = "y" ]]
    then
            if [[ -z "${mysqlBackupDir}" ]]
            then
                    echo "The 'mysqlBackupDir' var cannot be empty. Fix and try again."
                    exit 1;
            fi
    fi
    
    
    
    # Create necessary folders
    now=$(date '+%Y-%m'-%d_%H-%M)
    mkdir -p "${backupDir}/current"
    mkdir -p "${backupDir}/old/${now}"
    if [[ "${doMysqlBackup}" = "y" ]]
    then
            mkdir -p "${mysqlBackupDir}"
    fi
    
    # Make MySQL Backup
    if [[ "${doMysqlBackup}" = "y" ]]
    then
            # Remove existing mysql backup data
            rm -Rf "${mysqlBackupDir}/*"
    
            # Dump new files
            for i in $(echo 'SHOW DATABASES;' | mysql -u"${mysqlUser}" -p"${mysqlPasswd}" -h"${mysqlHost}" | grep -v '^Database$')
            do
                    mysqldump                                                                                                       \
                    -u"${mysqlUser}" -p"${mysqlPasswd}" -h"${mysqlHost}"            \
                    -Q -c -C --add-drop-table --add-locks --quick --lock-tables     \
                    "${i}" > "${mysqlBackupDir}/${i}.sql";
            done
    
            # Stop and Restart MySQL to make binary copy of the DBs
            /etc/init.dmysql stop
            cp -a "${mysqlDataFolder}/"* "${mysqlBackupDir}/"
            /etc/init.d/mysql start
    fi
    
    
    
    # If it's the first day of the month, make full backup (to have all data read again)
    testMonth=$(date +%d)
    if [[ "${testMonth}" = "01" ]]
    then
            rm -Rf "${backupDir}/current"
            mkdir -p "${backupDir}/current"
    fi
    
    
    
    
    # Run rsync into current
    rsync                                                                                           \
            -avzp --delete --delete-excluded                                \
            --exclude-from="${excludes}"                                    \
            --exclude "${backupDir}/"                                               \
            $mysqlExclude                                                                   \
            /                                                                                               \
            "${backupDir}/current" ;
    
    
    # Update the mtime to reflecht the snapshot time
    touch "${backupDir}/current"
    
    
    # make hardlinke copy  --> this does the magic of creating snapshot-style incremental backups
    cp -al "${backupDir}/current/"* "${backupDir}/old/${now}"
    
    
    
    
    # Remove old backups
    #for i in "$( find ${backupDir}/old/ -maxdepth 1 -type d -mtime +${days} )"
    #do
    #       rm -Rf "${i}"
    #done
    find "${backupDir}/old/" -maxdepth 1 -type d -mtime +"${days}" -exec rm -Rf "{}" \;
    
    exit 0
    

    backup_excludes -> adjust to your own needs
    Code:
    /backup/
    /dev/
    /initrd.img
    /lost+found/
    /media/
    /proc/
    /run/
    /sys/
    /tmp/
    /var/lib/mysql/
    /vmlinuz
    

    Your questions:

    1. When a disk in Ubuntu Software RAID1 chrashes, you can't see the damage before rebooting the system. Using the known commands to check the array's/md's there's no indication of a problem before the systems is rebooted! That's what I noticed so far.

    Because of that I make monthly new hot copies. So the full raid is read from beginning to end. Hot Copies plus daily backups are fine for me.



    2. How to set-up the server to be able to send the warning emails the "RAID Monitor Deamon" creates, when a array gets degraded.

    In the /etc/mdadm/mdadm.conf file just add:
    MAILADDR [email protected]

    And also be sure to enable a mail sending capability. Either setup a full fledge email server or just set it up as smarthost... that should fairly simple with postfix. Smarthost setup means, that you give login credentials for an other mail service, e.g. gmail and it will they send through there.



    3. How to set up the system/e.g. a shell script to make backup on to the two 2Tb harddrives. 10 copies before overwriting the first. I'm thinking that making these two disks run as standard disk/no RAID! But have a script copy the backup from the RAID1 to both these disks. This way we have a backup on 2 normal harddrives, not SSD. Much easier to do recovery on "real" harddrives intead of SSD. And, we've got 2 harddisk with each a copy of the backup.

    In my example I use hardlinks. hardlinks basically doubles the reference in the "Table of Contents" of the filesystem. So two entries point to the same data. That is space efficient and works well together with rsync. Because when rsync notices, that a file has altered, it will unlink it first and create a new file. That way you can have a full backup for everyday yet not use too much space if the data doesn't change often.

    However, bit rot might a problem. That's why I make monthly a hot copy of the system as well as on every first of the month I make a full rewrite of all data.
     

Share This Page