Intermediate. This is a security discussion, but Python will be used. A basic overview of cryptographic hash functions will also be given.

Note that the lowest version of Python is 2.3 (and I have no power to change this), so I may be restricted from performing currently optimal solutions. Feel free to make suggestions anyways.

For a refresher on the MapReduce process, check out Part 1.

What I’m Up Against Storing Data Locally

Worker computers will be attacked. The people who I share the lab with are my friends, and they will therefore try to do everything they can to mess my day up. Forkbombs and unplugging are probably not out of the question. My friends skills run the gamut from *nix familiarity to mastery of college-level undergrad mathematics and cryptography courses.

Attackers have physical access to the machines. All of the machines are in computer labs. Enough said.

Very small shared drive between computers. The shared drive is large enough to store the final results, but not intermediate calculations.

All computers have different environments, almost none of which I have control over. My personal laptop is an Intel Core 2 Duo with 1.6mHz, and I do have full control of what software it runs. The head machine in the network is a Blade server running Solaris, and it has Python 2.4. The worker machines are Linux machines running Python 2.3.

For the purposes of this exercise, I am considering my own area of the shared network drive secure. Anything that is not taking place within the computer’s memory and within my own shared network drive will be considered “secure”, the truth be damned.

Storing Data on Worker Machines

As I mentioned above, there is not enough room for me to store intermediate data in the shared network drive, so I need to take advantage of the storage capacity of the worker machines. All of the worker machines are sitting right in a computer lab. These labs see mild-to-moderate use every day, so I can’t just use a live CD to start a distributed computing program.

Since the local drives of the machines are open for general use, I need to make a strategy to protect my data. First, let’s look at the naive way to store MapReduce data:

def do_work(self):
    # The only commas are guaranteed to be separators.
    # The work splitting is simplified for this example.
    todo = self.work.split(',')
    temp_file = open(self.tmpfile, 'w')
    for item in todo:
         result = str(mapreduce.map(item))
         temp_file.write(result + '\n')
    temp_file.close()

This gets the job done, but this could give me some serious headaches. What’s to stop someone from potentially altering, inserting, or erasing messages (permissions aside)? We need something called Cryptographic Hash Functions for added security.

Cryptographic Hash Functions

A hash function, in general, takes an arbitrary length message and converts it to a digest of a fixed size. There is no guarantee as to whether or not an attacker can determine the original message, or create a second message with the same digest.

A Cryptographic Hash Function is a hash function with the following properties:

Preimage Resistance: Let hash(message_a) = digest_a. If the attacker is given digest_a, can message_a be determined? If a hash digest is n bits, the preimage resistance is expected to be .

Second Preimage Resistance: If the attacker has a message, message_a, it should be hard to find message_b so that hash(message_a)=hash(message_b). If a hash digest is n bits, the preimage resistance is expected to be .

Collision Resistance: It should be hard for the attacker to make message_a and message_b so that hash(message_a) = hash(message_b). This is expected to have security of .

Message extension resistance: It should be hard for the attacker, when given hash(message_a), to easily calculate any possible extension hash(message_a || message_b), where “||” is concatenation.
A few examples of cryptographic hash functions:

• MD2, MD4, MD5 (Designed by Ron Rivest)
• Whirlpool (Designed by the makers of AES)
• SHA-0, SHA-1, SHA-224/256, SHA-384/512 (Designed by NiST)
• Elliptic Curve Cryptography (Designed by Lenstra, et. al)

It should be noted that the MD series is currently considered broken, it is now considered “practical” to find collisions of SHA-0 with beefy resources. SHA-1 has a theoretical collision attack by a group of Chinese researchers (Wang, Yin, Yu).

Better Data Storage on Worker Machines

I will use Python 2.3′s built-in SHA-0 digest. “But Jake,” I hear you say, “you said that it was insecure! Someone might alter one of your messages!” Let’s look at the tradeoff here: The Wang, Yin, Yu attack takes theoretically operations to complete on average. It’s not out of the realm of possibility to find a collision on an overpowered home computer in a few weeks, or on a cluster within a few days.

However, all of my data results are going to exist on local machines for a few hours, max, before the computation completes and work is sent home. Even if someone wanted to devote the time and energy to find a collision, they would be altering a single data point, when there could potentially be millions. The attackers would either be creating a completely bad data point that can be sanity checked, or they will be spending all of the computation power to throw off my result by a fraction. Needless to say, I’m more worried about losing large sets of data than an attacker adding random messages.

I will be using a method of hashing called “salting” where an extra value is added to the hash to help further disguise the message. This is a common method for storing passwords. As long as the salt isn’t short, the attacker should be unable to create a collision in reasonable time. I will be using a large “nonce”, (or number used once).

Note: I could use encryption for this task, especially with the freely available crypto libraries for Python, but the headache of getting them installed on all of the computers is not what my research is all about. Since it is irrelevant if attackers read the information, I think that it can be shown that encryption and my scheme are both equivalent to finding the encryption key / hash salt of these one-way functions. If you don’t agree, I’d really like to know why.

def do_work(self):
    # The only commas are guaranteed to be separators.
    # The work splitting is simplified for this example.
    todo = self.work.split(',')
    temp_file = open(self.tmpfile, 'w')
    i = 0
    for item in todo:
         result = str(mapreduce.map(item))
         sha_obj = sha.new(self.salt+result+str(i))
         t = str(i)+','+result+','+sha_obj.hexdigest()
         temp_file.write(t + '\n')
         i+=1
    temp_file.close()

The validation code is also simple:

def extract_value(line, salt):
    values = line.split(',')
    if len(values) != 3:
        return 0
    t = salt+values[1].strip()+values[0].strip()
    sha_obj = sha.new(t)
    if sha_obj.hexdigest() != values[2].strip():
        print "Hax"
        return 0
    return values[1]

What’s next?

Next week we will look into network transmissions of the data. Yes, that post is likely to be a lot like this one. Despite my wishes to the contrary, I need to spend more time researching than blogging.

Beginner / Intermediate. Assumes some knowledge of basic computer science. If you’re looking for some serious meat and potatoes, come back next week. If you’re looking for an article that isn’t about MapReduce because you’re sick of hearing about it, come back next month.

Introductory B.S.

TCNJ has about 20 Linux and 30 Unix machines in computer labs, but a HUGE chunk of their time is spent idling. The labs, for the most part, are pretty sparsely used.

I’ve been looking for a good professor-excused reason to do some major cluster computing, and finally my time has come! Another student and I are doing cryptography research to design a new hash algorithm (blog post to follow after Third Quarter 2008!).

Guess what cryptographic hash algorithms need? Lots and lots and lots of statistical testing of millions of separate hashes. It would take my laptop an unfortunate amount of time to process dozens of tests with millions of records even if we could parse 100mBps of data with our design. God help us when we want to change something: all of the testing needs to begin all over again. Yuck!

This situation sounds ideal for MapReduce. Why should I use my own machine when I can make other machines do it? I am a geek, purveyor of the computer lab, master and commander of all that lay before me. Let’s put those things to work!

While I’m on the “Looking For An Excuse” train, I’ve played around with Python extensively in the past and never produced anything substantial. I’ve always been impressed with the speed of which I’ve been able to make things in Python, and in this situation, “Batteries Included” will help me immensely.

So what’s the catch?

I have absolutely no control over what software is on any of the machines. I get to decide between Java 1.5, Python 2.4, and gcc/g++ 3.4.1. I’m not allowed to run as root on any of the machines. Any task needs to be run as the user, so I need to be logged in to the machines when the commands are run (if I’m not right about that, I’d love to hear it, but every way I’ve found so far to run a job in the future requires the user to be logged in).

The consolation prize is that they are all pretty beefy, with decent hard drive space (most empty!), 2GB of memory, and dual-core AMD processors.

System Architecture And Design

The machines are behind a central server (Named “TomHanks” for the sake of discussion… this is not the actual name of the computer) that also doesn’t do much. The machines don’t have any contact through the outside world except through TomHanks. They are not all necessarily in the same room, and they are not necessarily running the same operating system.

All of the computers are on the same architecture, and do have access to the same shared drive. However, my space restrictions on the shared drive are tiny, making any kind of shared storage impractical in the general case. However, this does provide interesting opportunities, as my school-given website is accessible through here, making report-generation a possibility. On the other hand, the important part of my semester is the research I’m doing, not the actual MapReduce client, so I’m going to try to keep it as simple as possible.

Abstract Design of MapReduce

The idea of MapReduce is beautifully simple, and takes simple ideas from functional paradigms and applies them to the world of distributed computing. Let’s say that we have a list of data that we want to hash. Picking an arbitrary list, we’ll use [1, 2, 3, 4, ... n].

First, we apply the map() function to every single value of the array. Assuming our function is named f, our result is [f(1), f(2), f(3), ..., f(n)]

Aside: What is “map”?

When we “map” something, all we are doing is applying the same function to every element of a list, and storing the results in another list. For example, if we have the list [1,2,3,4], and we apply the function f(x)=2x to this list, we get the list [2,4,6,8].

In this example, it’s obvious how individual elements were transformed, but that is not always the case.

What are the constraints on the mappings? For my purpose, it works out best if the map is not always one-to-one, meaning that each time we evaluate the function, we can choose to return nothing, one value, or more values. It *can* be one-to-one, and for some tasks, it is necessary.

Then, we take the resulting array and evaluate a “reduce” function.

Aside: What is “reduce”?

“Reduce” is perhaps the least obvious name to pick. Other names for this idea are “fold” and “accumulate”.

When we “reduce”, we are just combining all of the values in a list to a single value by applying some function.

How do we reduce the list [1,2,3,4] using addition? 1+2+3+4 = 10. We could also have converted it into a CSV string: “1,2,3,4″ or “4,3,2,1″. There are, of course, subtleties that I am skipping. If you would like more of them, please go here.

And there you have it, MapReduce!

That’s It?

As you would expect, we need to make alterations (either above the hood or under the hood) in order to better apply this to the real world.

For our real model, we will borrow from Google’s design. Before I read Google’s paper on MapReduce, I drew out how I thought that the model should work, and what I called PreReduce they called a Combiner, which I like better. It allows us, in certain cases, to do some of the heavy lifting of the Reduce on the satellite machines instead of the client machine.

For example, if we were trying to add the numbers from 1 to 100 trillion, why would we bother passing back the numbers [1, 2, 3, 4, ... n] for the host machine to add together? Addition of integers is commutative and associative, so we can add them on the satellite machine and then send back one large integer instead of an arbitrary number of small integers.

What is our high-level algorithm?

Ideally, what do we want a client to do?

• Get Work
• Map()
• Combine()
• Return Work.
• Reply to “Are you alive?” pings.
• Securely send results (as I’m sure my friends will try to send LoLcat messages to home base).

What do we want a server to do?

• Start all satellite machines.
• Respond to requests for data.
• Perform Reduce
• Occasionally check to see which machines are alive.
• Generate progress reports (optional).
• Track responses and keep a “Shit List” of computers that give a few bad responses (throwing the computer in jail for a few minutes and then trusting it again is probably the most practical).

So what can we do at a high level to get all of this rolling?

The MapReduce Process:

1. Start a central server that will dole out work tasks.
2. The central server starts the satellite machines. Fortunately, all of the machines share a network drive, so this is a relatively trivial task involving a little SSH action.
3. The satellite machines phone home for work. Why do we do it this way instead of letting the central server give work with the start commands? In short, a friend of mine thinks it’s hysterical to fork-bomb a machine every now and then (and sometimes war erupts during class), so I REALLY can’t guarantee that ANY of the machines will be working at any given point. If the machines phone home for work, we are more likely to have work units parsed out sooner.
4. The satellite machines perform map() on every input value. It is obvious that the satellite machines should be doing the heavy lifting of the map() function. There is no law that says that the resulting map has to be stored in one place, and in this case, it will be stored in 50 places for the time being.
5. The satellite machines perform combine() on every input value. As mentioned above, this lets us partition off some of the work of the Reduce function to the satellite machines.
6. The satellite machines return the results to the central machine.
7. The central machine pings satellite machines that are long in responding.
8. Repeat Steps 3-6 until all work units are done.
9. The central server performs Reduce.

What’s next?

Next week we will look into how we will organize the SSH connections, how we will login to the machines, and how we will trust “secure” answers when the satellites don’t have any kind of built in cryptography.