LITTLEBLACKDOG.COM

Posted: Thu Sep 20, 2007 10:03 am Post subject: Parallel Programming in Python

I just recently was introduced to the Parallel Python module, and I have to say, it rocks!

I have an application that iterates through 18,000 objects, performs a somewhat complex computation on each one and then inserts the results into a postgres database. I add some lines of code to time the process and ran it. It completed in 39 minutes.

I then spent about 20 minutes (and half of that was due to my own ignorance) adapting the application to run in parallel using the pp module. I have a dual core machine, so I set the job_server to 2 workers, and ran the application again. It completed in 23 minutes. Even with the time adapting the code it was faster than the original sequential application. On a dual dual-core machine it would be even faster as I could use 4 workers.

Colour me impressed!