Hybrid RAID 1 (Mirror) of RAM drive & SATA HDD Using LVM with LUKS [and systemd unit file] on Fedora Linux
Submitted by gintonbo (Contact Author) (Forums) on Tue, 2014-07-29 09:33. :: Fedora | Linux | Security | Security
Welcome astute reader.
The IT industry has a continual balance between security and usability. Within this balance, performance usually affects usability. In the realm of protecting "Data at Rest" (i.e. encryption) one may find three factors affecting performance, and therefore usabilty: The harddrive, CPU and RAM. Of these, the harddrive will always prove to be a bottleneck (yes, even with an Solid State Drive, SSD).
Free Software has a rather elegant solution for protecting Data at Rest, called Linux Unified Key Setup (LUKS). In the spirit of "Freedom 0: The freedom to run the program for any purpose." please enjoy my contribution to our collective knowledgebase of a solution to the fascinating problem of "how can we speed up encyption"?
(C) 2014 Justin Mitchell Bennett. Licenced under the GNU FDL.
2. Definitions and Baselines
I used an Acer Aspire V5-171 as the platform for creating a hybrid RAID 1 (RAM drive & SATA HDD). I noted, after about a year of use, that Fedora 19 rarely used over 1 GB of RAM, with the exception of running a Windows 7 virtual machine through Virtual Machine Manager (libvirt/QEMU). As the system allowed for a max of 16GB DDR3, when I upgraded the modules, I factored I could comfortably utilize about 10GB of RAM for a RAM drive to run Fedora in.
I did find limitation however, in how much hard drive space Fedora uses for root /. I tend to partition / (root), /home, and swap on the systems I build. However, with this setup, root (/) tended to run 11-16GB, with squid, ndjbdns, LibreOffice, Android Development Tool, Eclipse, etc. With Fedora, /usr tends to be the largest directory under root. For reasons I won't go into, /usr cannot currently be mounted as a separate partition in Fedora 20; which actually proves useful, as it houses both /bin and /sbin, as well as Xorg. Unless you've already got a somewhat streamlined Fedora installation, creating separate /opt & /var partitions is *highly* recommended.
This guide presumes a prerequisite knowledge of partitions, (i.e. creating them and the benefit of multiple partitions). Additionally, some familiarity with EUFI and GPT, as they affect a Fedora 20 installation, (i.e. creating a /boot/efi & /boot partition). If working with BIOS and/or MBR please adjust your partition strategy accordingly. Additionally, as shrinking/growing partitions can be a huge pain, this guide has been written from the perspective of a fresh Fedora installation or re-installation.
2B. Logical Volume Management (LVM)
This guide also presumes a familiarity with Logical Volume Management. As a minimum, you must have an understanding of Physical Volumes (pv), Volume Groups (vg) & Logical Volumes (lv) as they pertain to LVM. Further reading: RHEL Logical Volume Manager Administration
2C. RAID 1
As used in this guide the reference to RAID 1 (mirror) should be understood as RAID 1 being implemented through LVM2. Development of RAID continues at Red Hat, and of interest specific to this guide, we'll utilize lvchange --writemostly. For reference:
Fedora 20 LVM 2 changelog: 2.02.99-1 Peter Rajnoha
2D. Linux Unified Key Setup (LUKS)
In the simpliest terms, LUKS extends cryptsetup functionality with an aim to conform to TKS1. TKS1 uses a hashed key for performing data de/encryption through an encryption algorithm. In this guide, specifically, AES encryption. If it's good enough to secure US Secret classification of material, it should be secure enough to secure your Data At Rest. Further reading at Cryptsetup
2E. Fedora Live USB image
I utilized two USB sticks for this setup. At a minimum, create a Fedora USB media. As I prefer MATE Desktop to Gnome, I additionally created a net-install image. Additionally, once booted from the Fedora USB media, install gparted.
2F. CPU (AES-NI), RAM, and HDD benchmarks:
(Note, for these tests, I booted off the Fedora USB media)
Newer Intel(R) processors include the AES-NI instruction set. To test whether your processor has such instructions type
cat /proc/cpuinfo | grep aesFor brevity, I've skipped posting the output here.
Next, test the CPU throughput of various encryption algorithms. Type
cryptsetup benchmarkMy output yields:
# Tests are approximate using memory only (no storage IO). PBKDF2-sha1 386073 iterations per second PBKDF2-sha256 209380 iterations per second PBKDF2-sha512 140034 iterations per second PBKDF2-ripemd160 308404 iterations per second PBKDF2-whirlpool 164870 iterations per second # Algorithm | Key | Encryption | Decryption aes-cbc 128b 434.6 MiB/s 1528.2 MiB/s serpent-cbc 128b 63.3 MiB/s 212.4 MiB/s twofish-cbc 128b 133.2 MiB/s 250.6 MiB/s aes-cbc 256b 316.0 MiB/s 1140.8 MiB/s serpent-cbc 256b 63.7 MiB/s 213.1 MiB/s twofish-cbc 256b 132.8 MiB/s 250.2 MiB/s aes-xts 256b 1264.3 MiB/s 1266.8 MiB/s serpent-xts 256b 218.7 MiB/s 207.4 MiB/s twofish-xts 256b 241.3 MiB/s 247.5 MiB/s aes-xts 512b 994.2 MiB/s 986.7 MiB/s serpent-xts 512b 218.8 MiB/s 208.9 MiB/s twofish-xts 512b 243.7 MiB/s 247.7 MiB/s
Of note, AES-XTS with a 256bit key shows the highest throughput. By default, Fedora (through Anaconda) will install an AES-XTS 512 bit key LUKS partition using a SHA1 hash. I chose to reduce the AES key size from 512 to 256 for gaining throughput (~1000MiB/s vs ~1260MiB/s), and increased the key hash to SHA512, SHA1 having documented weakness.
On my system, I have 16GB DDR3L 1600MHz PC3 12800 memory. To display the current memory, type
type sudo dmidecode --type 17 | less
From that, look for the following:
Size: 8192 MB Form Factor: SODIMM Type: DDR3 Speed: 1600 MHz Configured Clock Speed: 1600 MHz
(Sidenote - From the factory, ACER shipped 6GB of RAM, however, the 2GB stick had been DDR3 1333MHz, slowing the other stick, a 4GB 1600MHz to 1333Mhz)
Presuming you have enough RAM, create a RAM disk of 1GB for testing throughput with:
sudo modprobe brd rd_nr=1 rd_size-1048576 max_part=0
Now test its read speed through hdparm:
sudo hdparm -t /dev/ram0
/dev/ram0: Timing buffered disk reads: 5050 MB in 3.00 seconds = 1683.23 MB/sec
After such testing, the RAM disk can be removed with
sudo rmmod brd
For comparison, do a test against the hard drive. In this case, an upgraded Travelstar 0J42251, /dev/sda3 has been fully partitioned as a LVM2 Physical Partition (sda1=/boot/efi @ 256MiB, sda2=/boot @ 512 MiB):
(Sidenote - Though a SATA 3 6GB/s drive, For unknown reasons, Acer released a BIOS update that capped the motherboard at SATA 2, though the chipset specifies SATA 3 capability)
sudo hdparm -t /dev/sda3
/dev/sda3: Timing buffered disk reads: 302 MB in 3.00 seconds = 100.51 MB/sec
In creating a RAID 1 mirror, we're trying to push our encryption throughput for Fedora (OS) from 100.51 MB/sec (HDD speed) to 1260 Mib/~1321 MB/s placing the encryption bottleneck at the CPU.
3. LVM or Creating the PV, VG & LGs
3A. Physical Volume (pv)
I'm not entirely sure why Fedora no longer maintains an LVM GUI. Given some of the added complexity of the underlying LVM, I do understand how the GUI falls very short on harnessing some of the advanced features of LVM, namely on the fly RAID 1 creation.
I chose the following partition scheme: sda1 /boot/efi 256 MiB, sda2 /boot 512 MiB, sda3 LVM2 all remaining space (~930.76 MiB)
3B. Volume Group (vg)
Next I created the Volume Group. Fedora (anaconda) names their vg Fedora_[Hostname]. My hostname being gintonbo (Japanese for silver dragonfly):
vgcreate fedora_gintonbo /dev/sda3
3C. Logical Volumes
The remaining partitions in the system (/, /opt, /var, with a different command for /home):
lvcreate -L 16G -n lv_root fedora_gintonbo
lvcreate -L 16G -n lv_swap fedora_gintonbo
lvcreate -L 16G -n lv_opt fedora_gintonbo
lvcreate -L 16G -n lv_var fedora_gintonbo
Having plenty of space remaining, I used 90% of it for lv_home:
lvcreate -l 90%FREE -n lv_home fedora_gintonbo
From the Cryptsetup FAQ 2.19 "How can I wipe a device with crypto-grade randomness?" next prewipe the drives. *Warning* yes, this will take a while, have a beverage or two of choice handy for this step:
cryptsetup open --type plain -d /dev/urandom /dev/fedora_gintonbo/lv_root to_be_wiped
cat /dev/zero > /dev/mapper/to_be_wiped
cryptsetup close /dev/fedora_gintonbo/lv_root
Repeat for the remaining Logical Volumes create, and it's better to save /dev/fedora_[hostname]/lv_home for last, as it will take the longest!
3D. RAM disk Extents
Either while waiting for cryptsetup to finish or upon completion of filling the partitions with cryptographic randomness, create the RAM disk for RAID. Kernel module brd uses K, so 1048576K=1GiB, and therefore times 10 (for 10GiB) = 10485760. However, the lv_root partition created earlier had been 16GiB. We're going to remove that partition in the next step, and reduce it down to 10GiB, however, math at such powers of 10 versus powers of 2 becomes fuzzy very quickly. Better to oversize your lv_root and shrink it, rather than trying to guess how much more RAM you'll need.
Kernel Module brd create the Ram Drive, 10GB this time:
sudo modprobe brd rd_nr=1 rd_size-10485760 max_part=0
Create a pv with the Ram disk:
Create a temporary VG for getting total extents for /dev/ram0
Next create a Logical Volume utilizing 100% of the RAM drive:
lvcreate 100%FREE -n lv_test testvg
Now we're going to find out how many *extents* are on the drive:
vgs -o +vg_free_count,vg_extent_count
On the RAM disk this gave 2559 total extents.
We can now remove the LV with
Then remove the VG
Next remove the PV
Lastly, unload the RAM disk
sudo rmmod brd
Please note, LVMs do not expect RAM disks to be PVs, and are VERY sensitive to being unable to locate a PV (think powerloss). Always cleanly shutdown your RAM disk, being unable to boot later due to a RAM wipe will not be fun. You have been warned.
3E. lv_root Remove & Recreate Through Extents
From the above, we came up with 2559 extents, with which we'll now match the HDD size to. In similar fashion as above, remove the lv_root:
Then recreate it, specifying extents, rather than MB:
lvcreate -l2559 -n lv_root fedora_[hostname]
4. LUKS AES Encryption & Ext4 File System
At this point, you should have five (5) LVs on the system, lv_root ~10GiB, lv_opt 16GiB, lv_var 16 Gib, lv_swap 16 Gib, and lv_home ~768GiB.
From the cryptsetup benchmark tests, aes-xts-plain had the highest throughput, when using a 256 bit key. Additionally, instead of SHA1 we'll utilize SHA512, and specify spending 5 sec (-i 5000) on hash iteration. When mounted through the mapper, Fedora mounts LUKS encrypted partitions at /dev/mapper/luks-[UUID]. However, as we'll be manually mounting the partition through the mapper, we'll need to utilize cryptsetup luksUUID [dev] for obtaining the UUID.
For each of the five LVs, you'll repeat the following steps. I'll illustrate two of them, to show what changes:
cryptsetup luksFormat -y -c aes-xts-plain -s 256 -h SHA512 -i 5000 --use-urandom /dev/mapper/fedora_gintonbo-lv_root
cryptsetup luksUUID /dev/mapper/fedora_gintonbo-lv_root
cryptsetup luksOpen /dev/mapper/fedora_gintonbo-lv_root luks-[UUID]
mkfs.ext4 -m1 -L gintonbo-root /dev/mapper/luks-[UUID]
cryptsetup luksFormat -y -c aes-xts-plain -s 256 -h SHA512 -i 5000 --use-urandom /dev/mapper/fedora_gintonbo-lv_home
cryptsetup luksUUID /dev/mapper/fedora_gintonbo-lv_home
cryptsetup luksOpen /dev/mapper/fedora_gintonbo-lv_home luks-[UUID]
mkfs.ext4 -m1 -L gintonbo-home /dev/mapper/luks-[UUID]
Again, as I had a new HDD, I'd geared the LVMs, LUKS and EXT4 partions toward a fresh Fedora install. Partitioning the system this way also worked seemlessly with Anaconda, only having to unlock the drives.
5. RAM disk & (LVM) RAID 1
Size matching the root partition to the RAM disk had been the hardest part of this endeavor. With the partitions setup in the last step, I'd then installed Fedora 20 through a Net Install with MATE. Because the root partition had been expressed through extents size (2559 = ~10GiB) the RAM disk as RAID member becomes trivially easy.
5A. To create the RAM disk and add it to the LVM RAID 1:
sudo modprobe brd rd_nr=1 rd_size=10485760 max_part=0
sudo pvcreate /dev/ram0
sudo vgextend fedora_gintonbo /dev/ram0
(Optionally display your LVs at this point) sudo lvs -a -o name,copy_percent,devices fedora_gintonbo
sudo lvconvert --type raid1 -m1 fedora_gintonbo/lv_root
To watch the percentage of mirror building
sudo lvs -a -o name,copy_percent,devices fedora_gintonbo
Lastly, change the HDD to writemostly, so reads occur from RAM disk:
sudo lvchange --writemostly /dev/sda3:y fedora_gintonbo/lv_root
5B. To remove the RAM disk from LVM RAID 1 (prior to system powerloss event!)
sudo lvchange --writemostly /dev/sda3:n fedora_gintonbo/lv_root
sudo lvconvert -m0 fedora_gintonbo/lv_root /dev/ram0
sudo vgreduce fedora_gintonbo /dev/ram0
sudo rmmod /dev/ram0
6. Systemd Unit file
As the finishing touch, have systemd manage RAM disk & RAID 1 creation/removal. We'll create a systemd unit called 2Ro2L (RAMdisk RAID on LVM & LUKS) and two scripts:
sudo pluma /lib/systemd/system/2Ro2L.service
[Unit] Description=Enable RAMdisk RAID 1 on LVM & LUKS After=multi-user.target [Service] RemainAfterExit=yes ExecStart=/usr/sbin/2Ro2Lstart ExecStop=/usr/sbin/2Ro2Lstop Type=oneshot [Install] WantedBy=multi-user.target
sudo chmod 644 /lib/systemd/system/2RoRL.service
Then create the start and stop scripts:
sudo pluma /usr/sbin/2Ro2Lstart
#!/bin/sh # create the RAM disk modprobe brd rd_nr=1 rd_size=10485760 max_part=0 # create an LVM Physical Volume on created RAMdisk: pvcreate /dev/ram0 # Extend Volume Group to include above pv vgextend fedora_gintonbo /dev/ram0 # Convert the Logical Volume to a RAID 1 configuration lvconvert --type raid1 -m1 fedora_gintonbo/lv_root # Change the Logical Volume RAID properties lvchange --writemostly /dev/sda3:y fedora_gintonbo/lv_root
sudo pluma /usr/sbin/2Ro2Lstop
#!/bin/sh # Remove Logical Volume RAID properties lvchange --writemostly /dev/sda3:n fedora_gintonbo/lv_root # Remove the RAMdisk from the RAID lvconvert -m0 fedora_gintonbo/lv_root /dev/ram0 # Remove the RAMdisk from the Volume Group vgreduce fedora_gintonbo /dev/ram0 # Remove the RAMdisk altogether rmmod brd
sudo chmod 755 /usr/sbin/2Ro2Ls*
Start the service, stop the service, and if all goes well, enable the service for multiuser target
sudo systemctl start 2Ro2L.service
sudo systemctl stop 2Ro2L.service
If those ran without error, than enable the service:
sudo systemctl enable 2Ro2L.serviceSpecial thanks to Sergey Davidoff, FedoraZram for the systemd unit information!