Fully Utilizing Your X-Core CPU

Want to support HowtoForge? Become a subscriber!
 
Submitted by gbi (Contact Author) (Forums) on Mon, 2010-03-01 17:43. :: Linux

Fully Utilizing Your X-Core CPU

Almost all systems sold nowadays have at least a dual-core CPU, even triple- or quad-cores are getting cheaper and getting standard in the near future. But how to utilize your shiny x-core to it's full potential, with applications that are only utilizing one core ? With Linux, which has strong multitasking capabilities as all unixoid operating systems, there is an easy possibility to parallelize tasks which are normally only using one core of an x-core CPU.

You should be familiar working in the shell, because all work is done there. No browser, no fancy GUI's, no eyecandy.

 

Disclaimer

The following article describes the way I installed and used the software, I do not issue any guarantee that the same way works for you.

 

1. Parallel Processing Shell Script

This script is hosted on http://code.google.com/p/ppss/. It is released under BSD licensing terms, changing to GPL with version 2.55. It is not packaged in a repository of the popular Linux distributions, so it has to be installed manually.

There is a full, detailed documentation on the hosting site and the script could be downloaded from there in form of an archive. As there is only the script in the archive I unpacked the archive to /usr/local/bin, checked that it's belonging to root and is world read- and executable:

wget http://ppss.googlecode.com/files/ppss-2.50.tgz

tar xvzf ppss-2.50.tgz -C /usr/local/bin

chown root:root /usr/local/bin/ppss.sh && chmod a+rx /usr/local/bin/ppss.sh

ls -l /usr/local/bin/ppss.sh

-rwxr-xr-x 1 root root 54612 2009-12-17 16:40 /usr/local/bin/ppss.sh*

Requirements other than bash is only the mkfifo command. mkfifo is usually in the GNU coreutils package. As we are installing without a package manager, it's obvious that you manually have to take care that all dependencies are resolved. ppss.sh runs not only on Linux, but also on Mac OS X, may be also on *BSD.

When you try to call it without any arguments it should show it's help screen:

$ ppss.sh 

|P|P|S|S| Distributed Parallel Processing Shell Script 2.50

usage: /usr/local/bin/ppss.sh [ -d  | -f  ]  [ -c ' "$ITEM"' ]
                 [ -C  ]  [ -j ] [ -l  ] [ -p <# jobs> ]
                 [ -D  ] [ -h ] [ --help ]

Examples:
                 /usr/local/bin/ppss.sh -d /dir/with/some/files -c 'gzip '
                 /usr/local/bin/ppss.sh -d /dir/with/some/files -c 'gzip "$ITEM"' -D 5
                 /usr/local/bin/ppss.sh -d /dir/with/some/files -c 'cp "$ITEM" /tmp' -p 2

For basic usage of ppss.sh we don't need to configure anything, as the script is smart enough to analyze how much cores the CPU of the machine it's running on, has.

 

2. Examples

A job perfectly suited being parallelized with ppss.sh is, when you want to transform a bunch of .flac files to for instance oggvorbis files, for listening on your portable player. This could easily be done in a for loop, and we are lucky that oggenc is capable of reading .flac files:

for i in *.flac; do oggenc -q3 -o ${i%%flac}ogg $i; done

This is the sequential approach, only one of your x-cores is used. This can be seen by issuing a

top -n1

in another window, while the conversion is running:

Tasks: 112 total,   2 running, 110 sleeping,   0 stopped,   0 zombie
Cpu(s): 56.9%us,  0.6%sy,  0.0%ni, 41.2%id,  1.1%wa,  0.0%hi,  0.2%si, 0.0%st
Mem:   4062968k total,  2303836k used,  1759132k free,    36268k buffers
Swap:   522072k total,        0k used,   522072k free,  1558228k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND          
 7248 root      20   0 19780 2604 1468 R   88  0.1   0:05.02 oggenc  
...

Let's try to parallelize it with ppss.sh:

ppss.sh -d /path/to/your/flacfiles/ -c 'oggenc $ITEM -q3 -o "${ITEM%%.flac}.ogg"'

Instead of the loop-variable i we have to use the variable ITEM, which is preset by ppss.sh as "loop" variable. We get the following output on the screen:

Feb 03 20:16:00:  =========================================================
Feb 03 20:16:00:                         |P|P|S|S|                        
Feb 03 20:16:00:  Distributed Parallel Processing Shell Script version 2.50
Feb 03 20:16:00:  =========================================================
Feb 03 20:16:00:  Hostname:             xxxxx
Feb 03 20:16:00:  ---------------------------------------------------------
Feb 03 20:16:00:  CPU: AMD Athlon(tm) Dual Core Processor 4850e
Feb 03 20:16:00:  Found 2 logic processors.
Feb 03 20:16:00:  Starting 2 parallel workers.
Feb 03 20:16:00:  ---------------------------------------------------------
Feb 03 20:17:00:  Currently 100 percent complete. Processed 7 of 7 items.
Feb 03 20:17:00:  1 job is remaining.      
Feb 03 20:17:34:  Finished. Consult ./ppss/job_log for job output.

And another

top -n1

shows that all cores are utilized:

Tasks: 117 total,   3 running, 114 sleeping,   0 stopped,   0 zombie
Cpu(s): 99.4%us,  0.6%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si, 0.0%st
Mem:   4062968k total,  2070004k used,  1992964k free,    38916k buffers
Swap:   522072k total,        0k used,   522072k free,  1709844k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND          
10933 root      20   0 19856 2600 1468 R   99  0.1   0:06.20 oggenc            
10854 root      20   0 19856 2600 1468 R   87  0.1   0:08.51 oggenc     
...

Another example would be to convert the same .flac files to .mp3. This is a bit more complicated, because we need flac to decode the .flac files and lame for encoding them. This could be done sequential in a for loop, pipeing stdout of flac to stdin of lame:

for file in *.flac; do $(flac -cd "$file" | lame -h - "${file%.flac}.mp3"); done

Another

top -n1

shows that only one core is used:

Tasks: 118 total,   3 running, 115 sleeping,   0 stopped,   0 zombie
Cpu(s): 99.3%us,  0.7%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si, 0.0%st
Mem:   4062968k total,  2080004k used,  1982964k free,    37926k buffers
Swap:   522072k total,        0k used,   522072k free,  1709844k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND          
17837 root      20   0 19856 2600 1468 R   99  0.1   0:04.70 oggenc            
...

The same parallelized by ppss.sh:

ppss.sh -d /path/to/flacfiles -c 'flac -cd "$ITEM" | lame -V 4 -B 160 - "${ITEM%.flac}.mp3"'

Here

top -n1

shows that both cores are utilized nearly to their full potential:

Tasks: 129 total,   3 running, 126 sleeping,   0 stopped,   0 zombie
Cpu(s): 98.2%us,  1.7%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.2%hi,  0.0%si, 0.0%st
Mem:   4062968k total,  2629956k used,  1433012k free,    33996k buffers
Swap:   522072k total,        0k used,   522072k free,  1890244k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
 7287 root      20   0 14788 2480 1124 R   93  0.1   0:13.41 lame               
 7210 root      20   0 14788 2484 1124 R   89  0.1   0:23.59 lame               
 7209 root      20   0 15776 1060  792 S    7  0.0   0:01.67 flac               
 7286 root      20   0 15776 1064  792 S    6  0.0   0:00.95 flac     

A last example (dealing with soundfiles) would be to downsample some .mp3 files encoded with 320kBit to 128kBit VBR, again to save precious space on the portable player. This could be done with lame, without the help of a decoder like in the previous example:

for i in *.mp3; do lame -V 4 -B 160 - "${i%.flac}.mp3"'; done

Parallelizing this with ppss.sh looks like

ppss.sh -d /path/to/320kBit/mp3_files -c 'lame $ITEM -V 4 -B 160 "${ITEM%.mp3}.downmp3"'

and looks like

Tasks: 132 total,   3 running, 129 sleeping,   0 stopped,   0 zombie
Cpu(s): 98.8%us,  1.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.2%si, 
0.0%st
Mem:   4062968k total,  2441644k used,  1621324k free,    37068k buffers
Swap:   522072k total,        0k used,   522072k free,  1671080k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND          
 9073 root      20   0 14792 2724 1248 R   98  0.1   0:10.80 lame              
 8994 root      20   0 14788 2720 1248 R   98  0.1   0:15.57 lame

As we do not like to overwrite the 320kBit files, we simply give the downsampled files the extension .downmp3. After the downsampling is done, you can move the downsampled files out of the way with mmv:

mmv "*.downmp3" "/new/path/#1.mp3"

So you rename the files and move them elsewhere with one command. mmv, by the way, should be in the repositories of the most common distributions.

Another feature of ppss.sh is that you can use a file to provide it with the items to process. Say you have a directory with a lot of pictures, and you first want to view all pictures with feh as image viewer and use feh's filelist feature to pick some files for later processing. The processing should be done with ImageMagick, which has a lot of features for filtering and manipulating images. As manipulation of images could be cpu intensive, this is perfectly suited for ppss.sh.

A filelist of feh looks like

/path/to/images/1.png
/path/to/images/3.png
/path/to/images/17.png
...
/path/to/images/13815.png

It is produced with the -f filelist commandline parameter. You can delete an image from the filelist by pressing the <DELETE> key while in feh. Unfortunately absolute paths cannot be used with ppss.sh, but feh itself could be used to convert it's filelist to one suitable for ppss.sh:

feh *.png -f feh_list -L %n > ppss_list

or awk:

cat feh_list | awk -F/ '{ print $NF }' > ppss_list

Say you want to do some real timeconsuming filtering with all these selected files. With ImageMagick you cannot use filelists of feh, if you want to process a whole directory you can use the above mentioned for loop method. With feh you can also build recursive filelists, which could not be processed within a simple for loop with ImageMagick. But back to the example, now that you have a filelist you do the nifty filtering like

ppss.sh -f ppss_list -d /path/to/images -c 'mogrify -solarize 50 "$ITEM"'

(I only do a solarize, but you get the idea).

On the screen you see ppss working:

Feb 03 15:06:06:  =========================================================
Feb 03 15:06:06:                         |P|P|S|S|                         
Feb 03 15:06:06:  Distributed Parallel Processing Shell Script version 2.50
Feb 03 15:06:06:  =========================================================
Feb 03 15:06:06:  Hostname:		xxxxxxxx
Feb 03 15:06:06:  ---------------------------------------------------------
Feb 03 15:06:06:  CPU: Intel(R) Core(TM)2 Duo CPU     T5470  @ 1.60GHz
Feb 03 15:06:06:  Found 2 logic processors.
Feb 03 15:06:06:  Starting 2 parallel workers.
Feb 03 15:06:06:  ---------------------------------------------------------
Feb 03 15:06:17:  Currently 100 percent complete. Processed 65 of 65 items.
Feb 03 15:06:17:  1 job is remaining.       
Feb 03 15:06:17:  Finished. Consult ./ppss/job_log for job output.

Please do not use the comment function to ask for help! If you need help, please use our forum.
Comments will be published after administrator approval.
Submitted by Ole Tange (not registered) on Mon, 2010-06-07 23:21.

I have here translated your examples to GNU Parallel http://www.gnu.org/software/parallel/ It makes the examples somewhat easier to read:

for i in *.flac; do oggenc -q3 -o ${i%%flac}ogg $i; done

ls *.flac | parallel -j+0 oggenc -q3 -o {.}.ogg {}

for file in *.flac; do $(flac -cd "$file" | lame -h - "${file%.flac}.mp3"); done

ls *.flac | parallel -j+0 'flac -cd {} | lame -h - {.}.mp3'

for i in *.mp3; do lame -V 4 -B 160 "$i" "${i%.mp3}.downmp3"'; done

ls *.mp3 | parallel lame -V 4 -B 160 {} {.}.downmp3

ppss.sh -f ppss_list -d /path/to/images -c 'mogrify -solarize 50 "$ITEM"'

cat feh_list | parallel mogrify -solarize 50

ppss.sh config -C config.cfg -c 'lame $ITEM -V 4 -B 160 "${ITEM%.mp3}.downmp3"' -d /home/ppss/mp -m srvr -u ppss -K ~/.ssh/known_hosts -n /home/ppss/nodes.txt -k /home/ppss/ppss-private.key

ls *.mp3 | parallel -j+0 --sshloginfile nodes.txt --trc {.}.downmp3 lame -V 4 -B 160 {} {.}.downmp3

 

Submitted by Anonymous (not registered) on Thu, 2010-03-04 02:58.

Nice article..

But just to let you know its not recommend to run applications as root for security reasons..