Storing Files/Directories In Memory With tmpfs

Want to support HowtoForge? Become a subscriber!
 
Submitted by falko (Contact Author) (Forums) on Sun, 2008-12-07 18:22. :: Linux

Storing Files/Directories In Memory With tmpfs

Version 1.0
Author: Falko Timme <ft [at] falkotimme [dot] com>
Last edited 11/27/2008

You probably know that reading from RAM is a lot of faster than reading files from the hard drive, and reduces your disk I/O. This article shows how you can store files and directories in memory instead of on the hard drive with the help of tmpfs (a file system for creating memory devices). This is ideal for file caches and other temporary data (such as PHP's session files if you are using session.save_handler = files) because the data is lost when you power down or reboot the system.

I do not issue any guarantee that this will work for you!

 

Using tmpfs

There's a standard memory device on each Linux system (except for some virtual machines - depends on the virtualization technology) - /dev/shm.

When you run

mount

you should see something like this:

tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)

By default it's about half the size of the system's memory (you can check it's actual size by running

df -h /dev/shm

) - so if you have 4GB of RAM, its size will be about 2GB.

You can use /dev/shm as if it was a normal hard drive, for example, you can copy a file to it:

cp -af test.tar.gz /dev/shm/

Et voilà, the file is stored in memory:

ls -la /dev/shm/

server1:/# ls -la /dev/shm/
total 316
drwxrwxrwt 18 root root    380 2008-11-27 16:06 .
drwxr-xr-x 12 root root   3780 2008-11-27 15:33 ..
-rw-r--r--  1 root root 311636 2003-04-02 20:00 test.tar.gz
server1:/#

(Please keep in mind that the file will be lost when you power down or reboot the system!)

If you like, you can resize /dev/shm, for example as follows:

mount -o remount,size=3G /dev/shm

(Grow with care - if you make it too big and use all that space, then less RAM will be left for the rest of the system. This could cause all kinds of unwanted behaviours!)

Now let's assume you want to create some kind of file cache for your high-traffic web site in the directory /var/www/www.example.com/cache. Of course, it would be good to have this cache in memory. Here's how:

First create the cache directory:

mkdir -p /var/www/www.example.com/cache

(If needed by your cache, you can change its ownership, e.g. like this:

chown proxy:proxy /var/www/www.example.com/cache

)

Now we mount that directory as a memory device (with a size of 100MB and permissions of 755):

mount -t tmpfs -o size=100M,mode=0755 tmpfs /var/www/www.example.com/cache

Take a look at

mount

... and you should see this:

tmpfs on /var/www/www.example.com/cache type tmpfs (rw,size=100M,mode=0755)

That's it - now you can cache files directly in memory.

If you want to have that directory mounted at boot time, edit /etc/fstab...

vi /etc/fstab

... and add something like this to the file:

[...]
tmpfs /var/www/www.example.com/cache tmpfs size=100M,mode=0755 0 0
[...]

 

Links


Please do not use the comment function to ask for help! If you need help, please use our forum.
Comments will be published after administrator approval.
Submitted by Glen (not registered) on Mon, 2013-09-02 03:07.

I do not issue any guarantee that this will work for you!

It worked quite well. And thanks for the explanation of how to shrink the size of /dev/shm. It's always amazed me that no matter how much RAM I install, Linux finds a way to use all of it, and still requests additional swap space.

Submitted by Anonymous (not registered) on Sun, 2012-05-13 17:17.
The speed is almost the same as working with fylesystem on disk
Submitted by DavidG (not registered) on Thu, 2008-12-11 10:48.

I noticed some common confusion in the comments posted... TmpFS is not necessarily kept in RAM, it uses virtual memory and can thus be swapped out to disk as well...Which is good IMHO. (Not everything you cache may not be used that often.) So the maximum possible size is not limited by the amount of RAM you have, but by the virtual memory size.
Actually, I tent to use the size of my swap as the maximum size for tmpfs on /tmp, since I'm hardly using my swapfile for anything but suspend to disk.
(If you want to keep as much data in memory, you may want to lower the "swappiness" value in /proc/sys/vm/swappiness anyway)

Submitted by Anonymous (not registered) on Tue, 2008-12-09 23:39.

This reminds me of a a hardware add-on for the IBM 360/370 computers in the 1970's.  IBM was beginning to incorporate paging and swapping into it's operating systems.  Hard drives at that time weren't very fast, but then, neither were the computers, so even though paging sped things up, it was still slow.  So this add-on was a lot of RAM, connected to a hard drive interface.  It looked like a hard drive from the OS side, but I/O performance was a lot closer to RAM than to a hard drive.  The price was also closer to the price of RAM, too.  This was back when a single RAM chip still had under 4K bits per chip (yeah, do the math on what 256 kilobytes would require). 

If the temporary file is small enough to fit into a virtual RAM drive, perhaps it would be just as smart to simply ask the developer to keep the information in memory, and not write it out until either the program ends, or the program knows persistent storage is required.

Finally, if temporary space is really required, why not put it on a fast thumb drive?  While not as fast as RAM, it's faster than a hard drive, and leaves more RAM available.

 

Submitted by Anonymous (not registered) on Tue, 2008-12-09 22:38.

Thanks for the article.  TMPFS is a really great tool, and deserves more attention.  Something that your article could be a little clearer on are the advantages of TMPFS over the traditional RAM disks that many of your readers may already be familiar with:

1) A TMPFS system only uses RAM as needed.  The 2GB figure for the  /dev/shm example in the article is just the maximum, it usually uses no memory at all.

2) Infrequently used content in a TMPFS system can be swapped out if the system is under memory pressure.

Submitted by Leszek (registered user) on Mon, 2008-12-08 08:05.
Nice and useful :-)
Submitted by devnull (not registered) on Wed, 2011-08-03 23:15.

please do an equivalent article for FreeBSD.  I want this, but in freebsd. : >

Submitted by MarlboroMan (not registered) on Fri, 2014-12-12 14:54.
Under FreeBSD you probably have a kernel module 'tmpfs.ko'
 
# Load it like this 
/sbin/kldload tmpfs

# Once it loaded you can create a memory mount
mkdir /tmpfs
/sbin/mount -o rw,size=2048000000 -t tmpfs tmpfs /tmpfs

# Now you have a /tmpfs folder sized 2Gb located in RAM (blazing fast)

# Now you can create subdirectories for storing php session files
/bin/mkdir /tmpfs/phpsess
/bin/chmod 777 /tmpfs/phpsess
 
# ...and (for example) for storing nginx static cache
/bin/mkdir /tmpfs/nginx-cache
/bin/chmod 777 /tmpfs/nginx-cache

# If you have heavy loaded MySQL at your server, there is one more MAIN THING you can do:
# locate my.cnf (at FreeBSD you should look in /etc or /usr/local/etc)
# and look at 'tmpdir' variable.
# change it to:
tmpdir = /tmpfs

# You'll see significant performance increase of MySQL :)
# Wish you the best! http://server-kiev.com