Comments on Processing 10000 Pictures Using Many Computers With Oropo (Debian/Ubuntu)

Processing 10000 Pictures Using Many Computers With Oropo (Debian/Ubuntu) Have you ever had a lot of data to process ? In such a moment after a while of processing we realize that it will take ages to complete. It would be faster if we could use two or three or even more computers. Let's use some computers - you think it is a lot of configuration ? You are wrong. With Oropo it's easy. Let's discuss a problem of processing large number of pictures. First approach for solving this problem is to process pictures sequentially on one computer. Second approach is to process pictures parallelly on many computers.

1 Comment(s)

Add comment

Please register in our forum first to comment.

Comments

By: Ole Tange
Reply  

Using GNU Parallel http://www.gnu.org/software/parallel/ it would be like this:

find imgs | parallel 'djpeg {} | cjpeg -quality 30 > {.}.smaller.jpg'

To run on the local and multiple remote computers with one job per CPU core, transferring data to and from the remote computer do:

find imgs | parallel -j+0 --trc {.}.smaller.jpg -Scomputer1,computer2,: 'djpeg {} | cjpeg -quality 30 > {.}.smaller.jpg'

Watch the intro video for more http://www.youtube.com/watch?v=LlXDtd_pRaY