Comments on: MacGyver’s MapReduce in Python! Part 1: Theory

By: MacGyver’s MapReduce in Python! Part 2: Local Security | So Jake Says

MacGyver’s MapReduce in Python! Part 2: Local Security | So Jake Says — Mon, 12 Jan 2009 05:08:58 +0000

[...] For a refresher on the MapReduce process, check out Part 1. [...]

By: Jake

Jake — Mon, 18 Feb 2008 22:19:31 +0000

@rainfay: It looks like Crontab will work. Thanks!

By: kim

kim — Mon, 18 Feb 2008 20:41:40 +0000

Ever consider using BOINC? You can get free CPU time on thousands of machines.

Plan:

1. Setup a BOINC server.
2. use your application (your new hash algorithm) to demonstrate BOINC to other researchers at your school
3. get a job at school administering the BOINC server and helping other researchers write their apps to run on BOINC

http://boinc.berkeley.edu

By: admin

admin — Mon, 18 Feb 2008 15:38:28 +0000

@Bradford,

Unfortunately, students and faculty are able to use the machines via physical access (as well as SSH) all weekend long, so this is not an option. I looked at a few options like this, and unfortunately, all of them weren’t good enough. Unfortunately, I’m stuck with “MacGyver’s MapReduce” because I’m trying to make a distributed computing environment with extreme limitations.

By: rainfay

rainfay — Mon, 18 Feb 2008 08:21:21 +0000

you could see if you can run it as a crontab ^.^ then you would not have to be logged in….

By: Bradford Knowlton

Bradford Knowlton — Mon, 18 Feb 2008 07:00:22 +0000

Have you thought about building a cluster boot cd? If you could have the workstations boot from a CD, even as simple as DSL or knoppix, then download tasks from the server?

This would allow full control of the pc, using the 2 gig of ram as temporary storage, very little concern for problems with the cluster, and you could burn as many CD’s as you want. If you could boot 3 labs on campus on a saturday night and leave them running until monday morning that would probably be enough power for anything you are trying.

For example:
http://bccd.cs.uni.edu/

“The BCCD was created to facilitate instruction of parallel computing aspects and paradigms. Part of the difficulty instructors face is lack of dedicated resources to explore distributed computing aspects lack of time to preconfigure and test the supporting environment. The BCCD image addresses this problem by providing a non-destructive overlay way to run a full-fledged parallel computing environment on just about any workstation-class system…We’re happy to say that this now includes the MAC too!”

This would be very interesting to see your results, and the “super computing rating” you get when the cluster is running.