You’re a careful C++ programmer. You use namespaces obsessively, your classes are well-organized, and your files are carefully placed in a nice hierarchy. Name collisions aren’t going to strike your code!

Well, you actually can’t guarantee that: Namespaces are completely ignored by the macro preprocessor. Short of drastic measure (such as using #undef on every symbol you use) it is impossible to tell the C++ macro preprocessor, “You cannot pass!”

C++ Compiler at Work

If a macro is defined before it reaches your class, the preprocessor will try to expand all names in your class.

Does this ever happen? Sure! For example, CLAPACK defines a macro named “max,” which finds the maximum value of two elements. The C++ standard library offers std::numerical_limits<T>::max(), which gives you the maximum value of the T datatype. In fact, CLAPACK and similar libraries can reach through time and hamper std::numerical_limits’ compilation in any project. Surprise!

One name. Two valid uses. One compiler error.

Sure, CLAPACK is a C library, and one has to make allowances for this. In lieu of function overloading, macros are the only tool for approximating generic functions. On the other hand, COME ON! Someone else is going to use the name “max.”

I don’t mean to pick on CLAPACK! It’s a nice library. I like it. In fact, there is are several C++ versions of LAPACK that alleviates this specific issue. The C version exemplifies the problem perfectly.

What’s My So-Called Point?

My point is twofold. As a minor aside, I’m disappointed that this problem isn’t being addressed in the updated C++ standard. Macro namespaces would have been a nice touch (and another 10 pages of the already-huge standard). It’s not, so I won’t dwell on it.

If that were my overall point, you would be justified in leaving me an angry blog comment telling me that this post was a waste of breath. However, this isn’t my overall point! My real overall point is that many programmers don’t take enough advantage of features offered by modern programming languages.

I admit freely that I don’t, especially for one-shot projects. Today I looked at a well-documented (but poorly designed) project I wrote in early 2007. The comments and function names were good enough that I immediately knew what was going on. I was able to use the header files within minutes.

The downside is that I could immediately see that I designed myself into several corners, the worst being a macro named “NAME”.

Why did I do this? Because none of my teachers/professors/programming books ever condemned this kind of usage. I had to figure this out gradually over time.

The Imperfect Macro Workaround

Since macro name collision is a real problem, unofficial standards have risen for dealing with macros. They can usually be summarized by a few rules:

• Use all-caps.
• Use underscores for human-readable spacing.
• Add a likely-unique prefix to your macro names.

There are a few reasons that the conventions aren’t good enough, but most of them are edge cases: You can argue that you might find two libraries that use the same prefix and define the same macros, but this is probably about as common as namespace conflicts.

The real reason that this isn’t good enough is that it’s unenforceable. Any new programmer who just learned how to use macros won’t be using the convention, and they’ll be mucking up the preprocessor with macro definitions, which will cause protons to smash, create several black holes, and destroy the universe. I should know: as I mentioned above, I’ve done it.

Old libraries will also violate the guidelines. So will people with strong opinions who think that the conventions are OBVIOUSLY trash. So will people who code at 3AM on a deadline.

“One Honking Great Idea.”

The Zen of Python is one of the great programming guides available to everyone. How do you get it? Simply type “import this” into the interpreter!

>>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

The last line is reflected in Python’s design: loading code from a different file automatically provides you with namespaces. This is awesome: newbies and professionals alike must use namespaces, and best of all, it’s all done transparent with respect to the programming experience.

This is not unique to Python. Modern popular programming languages offer almost-universal support for namespaces.

• Common Lisp uses packages in order to support namespaces.
• Ruby and Python use modules.
• C++ has explicit namespaces.
• Bash scripts use the uniqueness of the file system (a path points to one file).
• C uses…. name prefixes. When the programmer feels like it.

Why do all of these languages support some form of namespaces? Because name collisions really are that bad. Libraries with major name collisions simply can’t be used together, and that is NOT the aim of programming.

What Can We Do?

I find three things that ultimately help me.

1. Work with people who know a lot more than you do.
2. Learn new languages.
3. Read books written by the greats.

Right now, I have the good fortune of doing all three.

Images used in this post

“The Demon Comes” by Flickr user Islãndßoy.

http://codegolf.com/saving-time I dare you to beat my score!

I go to the site and see that it gives you a problem definition, some examples, and ranks you on Perl, Ruby, Python, and PHP.

Saving Time

Input

A single 24-hour time into STDIN, such as 21:37 or 09:03.

Output

A clock laid out like the following:

        o
    o       o

 o             o

h               o

 o             o

    m       o
        o

This would be the associated output for 21:37. Minutes get rounded down to the nearest 5, and if the ‘h’ and ‘m’ fall on the same element of the clock, an ‘x’ appears instead.

First approach: Calculate each clock position

Here is the end result of my first attempt. This may have been my second or third submission: (Note: I now know that there are optimizations I could have used)

I=raw_input()
T=[0]*12
T[int(I[0:2])%12]+=1
T[int(I[3:5])/5]+=2
L=['o','h','m','x']
F=lambda i:L[T[i]]
print "%9s\n%5s%8s\n\n%2s%14s\n\n%s%16s\n\n%2s%14s\n\n%5s%8s\n%9s"%(F(0),F(11),F(1),F(10),F(2),F(9),F(3),F(8),F(4),F(7),F(5),F(6))

This baby clocked in at 234 bytes.

I coded using the algorithm that I’d use if I had to write the program “for real” (for some value of real.)

We have a list that represents the following clock positions:

T=[12:00, 1:00, 2:00, ... , 11:00]

We then add 1 in the proper position for the hours, and 2 for the minutes. When we’re printing, we print the proper character from L=['o','h','m','x']. Easy, huh?

When I came home from work the next day, I saw that my friends’ scores were significantly lower than mine.

Second Approach: Optimizations and Tricks

Every byte counts. I talked to two of my friends, Rob (who now holds the lead in Python with 143), and Steve (who has 148 in Perl, tied with my current score), and started comparing tricks for small parts of the program. I got hints for newlines and list slices, and found an awesome ternary hack for Python.

I also did some work to combine steps. Python’s good support of functional programming building blocks (lambda, apply, map, etc) played a big part in this. The algorithm is still basically the same, but more happens “on-the-fly.”

My code is now as follows:

I=raw_input()
h=int(I[:2])%12
m=int(I[3:])/5
print'''%9s
%5s%8s
 
%2s%14s
 
%s%16s
 
%2s%14s
 
%5s%8s
 
%9s'''%tuple(map(lambda c:[['o','m'][c==m],['h','x'][h==m]][c==h],(0,11,1,10,2,9,3,8,4,7,5,6)))

This puts me at 194 bytes. I have broken the 200 mark! Note that this program could still be a lot shorter as-is! I just didn’t know this when I submitted it. I was almost certain I was bottoming out for this method.

A conversation I had with Rob gave him the insight to cut his Python program down from a bytecount in the 160s to a bytecount in the 140s. My program is still a bloated cow in comparison.

Third Approach: One-liner

I decided to write a one-liner to see if I could gain any insight into the problem. It didn’t end up helping, but it is at least an example of functional programming in Python:

print"%9s\n%5s%8s\n\n%2s%14s\n\n%s%16s\n\n%2s%14s\n\n%5s%8s\n%9s"%apply(lambda h,m:tuple(map(lambda c:[['o','m'][c==m],['h','x'][h==m]][c==h],(0,11,1,10,2,9,3,8,4,7,5,6))),apply(lambda a,b:(a%12,b/5),map(int,raw_input().split(':'))))

It is the equivalent to this (commented and slightly better spaced) program:

# Reads input, splits on ':', casts to int, and
# processes hour and minute
hour,minute=apply(lambda a,b:(a%12,b/5), map(int, raw_input().split(':')))
 
# Based on hour/minute, returns a list of the proper
# character for each location on the clock face
clockChars=map(lambda currentNum:[['o','m'][currentNum==minute],['h','x'][currentNum==minute]][currentNum==hour],(0,11,1,10,2,9,3,8,4,7,5,6))
 
# Prints out the clock face
print"%9s\n%5s%8s\n\n%2s%14s\n\n%s%16s\n\n%2s%14s\n\n%5s%8s\n%9s"%tuple(clockChars)

Fourth Approach: I’m Not Telling!

In the end, I changed my line of attack completely and developed a version of my program that got me 2nd place for Python (20th overall as of 10:00 Sunday, August 24th, tied with Steve’s Perl program).

I will have to bow down and pledge my allegiance to Rob, as I just can’t get the last 5 bytes.

General Approach

I had a few guidelines that got me down to this program length:

• Lazy coding: Don’t do now what you can save for later.
• Count characters for sections: Count how many characters you use for each particular task. If it seems high, open a new file and try to reduce.
• Try everything: Open up your program reference and look for similar functions that take fewer arguments, or different paradigms.
• Every byte matters: Are you using Unix or Windows newlines? Do you have unnecessary spaces? Can you abuse the language syntax and remove a few bytes?
• Count-as-you-go: If you’re aiming to hit a certain character limit, count the total characters you’ve used as you go along. When you get close to that limit, it’s time to edit down. Is your method going to give you the character count needed? Are you repeating yourself too much?

In my final semester at TCNJ, I researched cryptographic hash algorithm with a math major. His programming experience was extremely limited: he had only programmed using Matlab, so he knew what functions and variables were, but that was the extent of his knowledge. In school I was a math minor, so we had a common ground for communication. However, part of my job was to actually write the programs that our research needed. Not only that, but the math major had to be able to modify what I had written in meaningful ways. I needed to write the clearest code that I had ever written.

So how did I do it? Language, Good Names/Comments, Structure, and Examples. Basically, all of the ordinary tools for making code more readable to programmers.

Language

I had a few criteria for selection:

• To the casual Matlab user, the code written in the language should be readable.
• I needed to quickly reach competency.
• It should run on any OS platform without hassle.

I ended up choosing Python. First of all, downloading it and installing it on any OS is a piece of cake, running a Python file takes exactly 1 step: fire up the interpreter with the file as an argument. Second of all, it’s essentially executable pseudo-code. The syntax necessary to write a Python program looks very clean, and the indentation of Python forces a certain level of readability.

Third, I’ve never explicitly “known” Python, but I’ve always reached the “get things done” level of competency within a day of picking it up for a project.

I ignored Java and C++ because getting the compiler tools installed on new Windows machines is a hassle compared to Python, the installation is in 2 steps, and there is just more syntax for the newbie to read. YMMV, but I’ve always picked Python back up quicker than I’ve picked up Java or C++ after a long hiatus, and I “know” Java and C++.

Good Names/Comments

Extensive comments obviously helped the project. Normally, I describe what a function does at a high level, and then comment any tricky parts. This is enough for programmers, but it turned out to be insufficient for non-programmers. I showed my partner some C++ code in the beginning of the project using this technique and he didn’t understand how it worked.

At the beginning of each file, I added a description of the structure, and pointers to where he could find key parts of the code. At the beginning of each function, I commented the hell out of it, describing what it was supposed to do. At every logical sub-section of the code, I left a comment describing what it accomplished.

The best addition to commenting was using good names: consistent case, minimize abbreviation for the most useful names, and consistent naming across functions.

Structure

Every file that I wrote had to have the same general structure:

• A big honkin’ description of the contents and purpose of the file.
• Functions in increasing order of importance:
  • Small utility functions at top.
  • Important control functions at the bottom.
• Main function with a few examples.

Since everything was always in the same place, my partner never had to look hard to find what he needed.

Examples

The best thing I did to talk to my parter in programming languages was to implement hash algorithms that he already understood. For example, here is my implementation of SHA-1 [.txt file due to hosting limitations]. I sat down with him and made sure that he knew how to run it, and from there, let him on his merry way.

The Result?

“Actually, it was a lot clearer than I thought it was going to be”

The code was a success. He was able to modify and read it successfully, and do everything that needed to be done.

Is This Relevent to Ordinary Programming?

No. I haven’t added anything new to the discussion, only approached an unusual situation with the usual solutions.

Intermediate. This is a security discussion, but Python will be used. A basic overview of cryptographic hash functions will also be given.

Note that the lowest version of Python is 2.3 (and I have no power to change this), so I may be restricted from performing currently optimal solutions. Feel free to make suggestions anyways.

For a refresher on the MapReduce process, check out Part 1.

What I’m Up Against Storing Data Locally

Worker computers will be attacked. The people who I share the lab with are my friends, and they will therefore try to do everything they can to mess my day up. Forkbombs and unplugging are probably not out of the question. My friends skills run the gamut from *nix familiarity to mastery of college-level undergrad mathematics and cryptography courses.

Attackers have physical access to the machines. All of the machines are in computer labs. Enough said.

Very small shared drive between computers. The shared drive is large enough to store the final results, but not intermediate calculations.

All computers have different environments, almost none of which I have control over. My personal laptop is an Intel Core 2 Duo with 1.6mHz, and I do have full control of what software it runs. The head machine in the network is a Blade server running Solaris, and it has Python 2.4. The worker machines are Linux machines running Python 2.3.

For the purposes of this exercise, I am considering my own area of the shared network drive secure. Anything that is not taking place within the computer’s memory and within my own shared network drive will be considered “secure”, the truth be damned.

Storing Data on Worker Machines

As I mentioned above, there is not enough room for me to store intermediate data in the shared network drive, so I need to take advantage of the storage capacity of the worker machines. All of the worker machines are sitting right in a computer lab. These labs see mild-to-moderate use every day, so I can’t just use a live CD to start a distributed computing program.

Since the local drives of the machines are open for general use, I need to make a strategy to protect my data. First, let’s look at the naive way to store MapReduce data:

def do_work(self):
    # The only commas are guaranteed to be separators.
    # The work splitting is simplified for this example.
    todo = self.work.split(',')
    temp_file = open(self.tmpfile, 'w')
    for item in todo:
         result = str(mapreduce.map(item))
         temp_file.write(result + '\n')
    temp_file.close()

This gets the job done, but this could give me some serious headaches. What’s to stop someone from potentially altering, inserting, or erasing messages (permissions aside)? We need something called Cryptographic Hash Functions for added security.

Cryptographic Hash Functions

A hash function, in general, takes an arbitrary length message and converts it to a digest of a fixed size. There is no guarantee as to whether or not an attacker can determine the original message, or create a second message with the same digest.

A Cryptographic Hash Function is a hash function with the following properties:

Preimage Resistance: Let hash(message_a) = digest_a. If the attacker is given digest_a, can message_a be determined? If a hash digest is n bits, the preimage resistance is expected to be .

Second Preimage Resistance: If the attacker has a message, message_a, it should be hard to find message_b so that hash(message_a)=hash(message_b). If a hash digest is n bits, the preimage resistance is expected to be .

Collision Resistance: It should be hard for the attacker to make message_a and message_b so that hash(message_a) = hash(message_b). This is expected to have security of .

Message extension resistance: It should be hard for the attacker, when given hash(message_a), to easily calculate any possible extension hash(message_a || message_b), where “||” is concatenation.
A few examples of cryptographic hash functions:

• MD2, MD4, MD5 (Designed by Ron Rivest)
• Whirlpool (Designed by the makers of AES)
• SHA-0, SHA-1, SHA-224/256, SHA-384/512 (Designed by NiST)
• Elliptic Curve Cryptography (Designed by Lenstra, et. al)

It should be noted that the MD series is currently considered broken, it is now considered “practical” to find collisions of SHA-0 with beefy resources. SHA-1 has a theoretical collision attack by a group of Chinese researchers (Wang, Yin, Yu).

Better Data Storage on Worker Machines

I will use Python 2.3′s built-in SHA-0 digest. “But Jake,” I hear you say, “you said that it was insecure! Someone might alter one of your messages!” Let’s look at the tradeoff here: The Wang, Yin, Yu attack takes theoretically operations to complete on average. It’s not out of the realm of possibility to find a collision on an overpowered home computer in a few weeks, or on a cluster within a few days.

However, all of my data results are going to exist on local machines for a few hours, max, before the computation completes and work is sent home. Even if someone wanted to devote the time and energy to find a collision, they would be altering a single data point, when there could potentially be millions. The attackers would either be creating a completely bad data point that can be sanity checked, or they will be spending all of the computation power to throw off my result by a fraction. Needless to say, I’m more worried about losing large sets of data than an attacker adding random messages.

I will be using a method of hashing called “salting” where an extra value is added to the hash to help further disguise the message. This is a common method for storing passwords. As long as the salt isn’t short, the attacker should be unable to create a collision in reasonable time. I will be using a large “nonce”, (or number used once).

Note: I could use encryption for this task, especially with the freely available crypto libraries for Python, but the headache of getting them installed on all of the computers is not what my research is all about. Since it is irrelevant if attackers read the information, I think that it can be shown that encryption and my scheme are both equivalent to finding the encryption key / hash salt of these one-way functions. If you don’t agree, I’d really like to know why.

def do_work(self):
    # The only commas are guaranteed to be separators.
    # The work splitting is simplified for this example.
    todo = self.work.split(',')
    temp_file = open(self.tmpfile, 'w')
    i = 0
    for item in todo:
         result = str(mapreduce.map(item))
         sha_obj = sha.new(self.salt+result+str(i))
         t = str(i)+','+result+','+sha_obj.hexdigest()
         temp_file.write(t + '\n')
         i+=1
    temp_file.close()

The validation code is also simple:

def extract_value(line, salt):
    values = line.split(',')
    if len(values) != 3:
        return 0
    t = salt+values[1].strip()+values[0].strip()
    sha_obj = sha.new(t)
    if sha_obj.hexdigest() != values[2].strip():
        print "Hax"
        return 0
    return values[1]

What’s next?

Next week we will look into network transmissions of the data. Yes, that post is likely to be a lot like this one. Despite my wishes to the contrary, I need to spend more time researching than blogging.

Beginner / Intermediate. Assumes some knowledge of basic computer science. If you’re looking for some serious meat and potatoes, come back next week. If you’re looking for an article that isn’t about MapReduce because you’re sick of hearing about it, come back next month.

Introductory B.S.

TCNJ has about 20 Linux and 30 Unix machines in computer labs, but a HUGE chunk of their time is spent idling. The labs, for the most part, are pretty sparsely used.

I’ve been looking for a good professor-excused reason to do some major cluster computing, and finally my time has come! Another student and I are doing cryptography research to design a new hash algorithm (blog post to follow after Third Quarter 2008!).

Guess what cryptographic hash algorithms need? Lots and lots and lots of statistical testing of millions of separate hashes. It would take my laptop an unfortunate amount of time to process dozens of tests with millions of records even if we could parse 100mBps of data with our design. God help us when we want to change something: all of the testing needs to begin all over again. Yuck!

This situation sounds ideal for MapReduce. Why should I use my own machine when I can make other machines do it? I am a geek, purveyor of the computer lab, master and commander of all that lay before me. Let’s put those things to work!

While I’m on the “Looking For An Excuse” train, I’ve played around with Python extensively in the past and never produced anything substantial. I’ve always been impressed with the speed of which I’ve been able to make things in Python, and in this situation, “Batteries Included” will help me immensely.

So what’s the catch?

I have absolutely no control over what software is on any of the machines. I get to decide between Java 1.5, Python 2.4, and gcc/g++ 3.4.1. I’m not allowed to run as root on any of the machines. Any task needs to be run as the user, so I need to be logged in to the machines when the commands are run (if I’m not right about that, I’d love to hear it, but every way I’ve found so far to run a job in the future requires the user to be logged in).

The consolation prize is that they are all pretty beefy, with decent hard drive space (most empty!), 2GB of memory, and dual-core AMD processors.

System Architecture And Design

The machines are behind a central server (Named “TomHanks” for the sake of discussion… this is not the actual name of the computer) that also doesn’t do much. The machines don’t have any contact through the outside world except through TomHanks. They are not all necessarily in the same room, and they are not necessarily running the same operating system.

All of the computers are on the same architecture, and do have access to the same shared drive. However, my space restrictions on the shared drive are tiny, making any kind of shared storage impractical in the general case. However, this does provide interesting opportunities, as my school-given website is accessible through here, making report-generation a possibility. On the other hand, the important part of my semester is the research I’m doing, not the actual MapReduce client, so I’m going to try to keep it as simple as possible.

Abstract Design of MapReduce

The idea of MapReduce is beautifully simple, and takes simple ideas from functional paradigms and applies them to the world of distributed computing. Let’s say that we have a list of data that we want to hash. Picking an arbitrary list, we’ll use [1, 2, 3, 4, ... n].

First, we apply the map() function to every single value of the array. Assuming our function is named f, our result is [f(1), f(2), f(3), ..., f(n)]

Aside: What is “map”?

When we “map” something, all we are doing is applying the same function to every element of a list, and storing the results in another list. For example, if we have the list [1,2,3,4], and we apply the function f(x)=2x to this list, we get the list [2,4,6,8].

In this example, it’s obvious how individual elements were transformed, but that is not always the case.

What are the constraints on the mappings? For my purpose, it works out best if the map is not always one-to-one, meaning that each time we evaluate the function, we can choose to return nothing, one value, or more values. It *can* be one-to-one, and for some tasks, it is necessary.

Then, we take the resulting array and evaluate a “reduce” function.

Aside: What is “reduce”?

“Reduce” is perhaps the least obvious name to pick. Other names for this idea are “fold” and “accumulate”.

When we “reduce”, we are just combining all of the values in a list to a single value by applying some function.

How do we reduce the list [1,2,3,4] using addition? 1+2+3+4 = 10. We could also have converted it into a CSV string: “1,2,3,4″ or “4,3,2,1″. There are, of course, subtleties that I am skipping. If you would like more of them, please go here.

And there you have it, MapReduce!

That’s It?

As you would expect, we need to make alterations (either above the hood or under the hood) in order to better apply this to the real world.

For our real model, we will borrow from Google’s design. Before I read Google’s paper on MapReduce, I drew out how I thought that the model should work, and what I called PreReduce they called a Combiner, which I like better. It allows us, in certain cases, to do some of the heavy lifting of the Reduce on the satellite machines instead of the client machine.

For example, if we were trying to add the numbers from 1 to 100 trillion, why would we bother passing back the numbers [1, 2, 3, 4, ... n] for the host machine to add together? Addition of integers is commutative and associative, so we can add them on the satellite machine and then send back one large integer instead of an arbitrary number of small integers.

What is our high-level algorithm?

Ideally, what do we want a client to do?

• Get Work
• Map()
• Combine()
• Return Work.
• Reply to “Are you alive?” pings.
• Securely send results (as I’m sure my friends will try to send LoLcat messages to home base).

What do we want a server to do?

• Start all satellite machines.
• Respond to requests for data.
• Perform Reduce
• Occasionally check to see which machines are alive.
• Generate progress reports (optional).
• Track responses and keep a “Shit List” of computers that give a few bad responses (throwing the computer in jail for a few minutes and then trusting it again is probably the most practical).

So what can we do at a high level to get all of this rolling?

The MapReduce Process:

1. Start a central server that will dole out work tasks.
2. The central server starts the satellite machines. Fortunately, all of the machines share a network drive, so this is a relatively trivial task involving a little SSH action.
3. The satellite machines phone home for work. Why do we do it this way instead of letting the central server give work with the start commands? In short, a friend of mine thinks it’s hysterical to fork-bomb a machine every now and then (and sometimes war erupts during class), so I REALLY can’t guarantee that ANY of the machines will be working at any given point. If the machines phone home for work, we are more likely to have work units parsed out sooner.
4. The satellite machines perform map() on every input value. It is obvious that the satellite machines should be doing the heavy lifting of the map() function. There is no law that says that the resulting map has to be stored in one place, and in this case, it will be stored in 50 places for the time being.
5. The satellite machines perform combine() on every input value. As mentioned above, this lets us partition off some of the work of the Reduce function to the satellite machines.
6. The satellite machines return the results to the central machine.
7. The central machine pings satellite machines that are long in responding.
8. Repeat Steps 3-6 until all work units are done.
9. The central server performs Reduce.

What’s next?

Next week we will look into how we will organize the SSH connections, how we will login to the machines, and how we will trust “secure” answers when the satellites don’t have any kind of built in cryptography.