Storing Files/Directories In Memory With tmpfs

Do you like HowtoForge? Please consider supporting us by becoming a subscriber.
Submitted by falko (Contact Author) (Forums) on Sun, 2008-12-07 18:22. :: Linux

Storing Files/Directories In Memory With tmpfs

Version 1.0
Author: Falko Timme <ft [at] falkotimme [dot] com>
Last edited 11/27/2008

You probably know that reading from RAM is a lot of faster than reading files from the hard drive, and reduces your disk I/O. This article shows how you can store files and directories in memory instead of on the hard drive with the help of tmpfs (a file system for creating memory devices). This is ideal for file caches and other temporary data (such as PHP's session files if you are using session.save_handler = files) because the data is lost when you power down or reboot the system.

I do not issue any guarantee that this will work for you!

 

Using tmpfs

There's a standard memory device on each Linux system (except for some virtual machines - depends on the virtualization technology) - /dev/shm.

When you run

mount

you should see something like this:

tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)

By default it's about half the size of the system's memory (you can check it's actual size by running

df -h /dev/shm

) - so if you have 4GB of RAM, its size will be about 2GB.

You can use /dev/shm as if it was a normal hard drive, for example, you can copy a file to it:

cp -af test.tar.gz /dev/shm/

Et voilà, the file is stored in memory:

ls -la /dev/shm/

server1:/# ls -la /dev/shm/
total 316
drwxrwxrwt 18 root root    380 2008-11-27 16:06 .
drwxr-xr-x 12 root root   3780 2008-11-27 15:33 ..
-rw-r--r--  1 root root 311636 2003-04-02 20:00 test.tar.gz
server1:/#

(Please keep in mind that the file will be lost when you power down or reboot the system!)

If you like, you can resize /dev/shm, for example as follows:

mount -o remount,size=3G /dev/shm

(Grow with care - if you make it too big and use all that space, then less RAM will be left for the rest of the system. This could cause all kinds of unwanted behaviours!)

Now let's assume you want to create some kind of file cache for your high-traffic web site in the directory /var/www/www.example.com/cache. Of course, it would be good to have this cache in memory. Here's how:

First create the cache directory:

mkdir -p /var/www/www.example.com/cache

(If needed by your cache, you can change its ownership, e.g. lke this:

chown proxy:proxy /var/www/www.example.com/cache

)

Now we mount that directory as a memory device (with a size of 100MB and permissions of 755):

mount -t tmpfs -o size=100M,mode=0755 tmpfs /var/www/www.example.com/cache

Take a look at

mount

... and you should see this:

tmpfs on /var/www/www.example.com/cache type tmpfs (rw,size=100M,mode=0755)

That's it - now you can cache files directly in memory.

If you want to have that directory mounted at boot time, edit /etc/fstab...

vi /etc/fstab

... and add something like this to the file:

[...]
tmpfs /var/www/www.example.com/cache tmpfs size=100M,mode=0755 0 0
[...]

 

Links


Please do not use the comment function to ask for help! If you need help, please use our forum.
Comments will be published after administrator approval.
Submitted by DavidG (not registered) on Thu, 2008-12-11 10:48.

I noticed some common confusion in the comments posted... TmpFS is not necessarily kept in RAM, it uses virtual memory and can thus be swapped out to disk as well...Which is good IMHO. (Not everything you cache may not be used that often.) So the maximum possible size is not limited by the amount of RAM you have, but by the virtual memory size.
Actually, I tent to use the size of my swap as the maximum size for tmpfs on /tmp, since I'm hardly using my swapfile for anything but suspend to disk.
(If you want to keep as much data in memory, you may want to lower the "swappiness" value in /proc/sys/vm/swappiness anyway)

Submitted by Anonymous (not registered) on Tue, 2008-12-09 23:39.

This reminds me of a a hardware add-on for the IBM 360/370 computers in the 1970's.  IBM was beginning to incorporate paging and swapping into it's operating systems.  Hard drives at that time weren't very fast, but then, neither were the computers, so even though paging sped things up, it was still slow.  So this add-on was a lot of RAM, connected to a hard drive interface.  It looked like a hard drive from the OS side, but I/O performance was a lot closer to RAM than to a hard drive.  The price was also closer to the price of RAM, too.  This was back when a single RAM chip still had under 4K bits per chip (yeah, do the math on what 256 kilobytes would require). 

If the temporary file is small enough to fit into a virtual RAM drive, perhaps it would be just as smart to simply ask the developer to keep the information in memory, and not write it out until either the program ends, or the program knows persistent storage is required.

Finally, if temporary space is really required, why not put it on a fast thumb drive?  While not as fast as RAM, it's faster than a hard drive, and leaves more RAM available.

 

Submitted by Anonymous (not registered) on Tue, 2008-12-09 22:38.

Thanks for the article.  TMPFS is a really great tool, and deserves more attention.  Something that your article could be a little clearer on are the advantages of TMPFS over the traditional RAM disks that many of your readers may already be familiar with:

1) A TMPFS system only uses RAM as needed.  The 2GB figure for the  /dev/shm example in the article is just the maximum, it usually uses no memory at all.

2) Infrequently used content in a TMPFS system can be swapped out if the system is under memory pressure.

Submitted by Leszek (registered user) on Mon, 2008-12-08 08:05.
Nice and useful :-)
Sponsored Links: Unified Communications: Thoughts, Strategies and Predictions
Join the discussion.
www.seamlessenterprise.com

IP Convergence
Integrate your wireless and wireline networks.
Learn how from the experts at Sprint.
www.seamlessenterprise.com

Wireless & Wireline Integration
Thoughts, strategies and solutions: join the discussion
www.seamlessenterprise.com

Unified Communications 2009
Join the Discussion. Now.
www.seamlessenterprise.com