Late last December, I woke up and it was freezing in my apartment. I checked the thermostat, and it was already less than 60 degrees! I turned it from hot to cold to hot. And again. I turned it all the way up. The furnace never clicked on. COME ON! I checked, and the pilot light was on, so I had no choice but to call maintenance to get it fixed.

Most days this wouldn’t have been a problem, since it would be fixed while I was at work, but I was in my end-of-year vacation-burning crunch. Maintenance always fix minor problems really fast, so I knew it would take all day to come and fix the heater. Sure enough, help didn’t arrive until 6PM.

I tried lying motionless on the floor under some blankets and played some Assassin’s Creed 2. This worked for an hour or so, but if I’m at home all day and not working on something on the computer, I start to feel like a waste. This was in the middle of trying to get a side project of mine off the ground, and I wanted to get a beta version working before returning to work in January.

Could I use my laptop as a heater? It gets pretty warm when the CPU runs at full blast on one core. If I kept both going for hours, would that be enough?

8 minutes later, I had a C++ program that ran 2 CPUs at full blast. I hate picking C++ for a personal project, but I’ve used it for 10 years and I’m just not familiar enough with any other language’s CPU usage patterns to do the work that quickly. I probably didn’t have to worry, but I was cold!

The program, warmup, runs NUM_CPU copies of this thread:

int fn(bool *go)
{
  double a = 0;
  static mt19937 rng(static_cast<unsigned> (std::time(0)));
  normal_distribution<double> norm_dist(1.0, 1.0);
  variate_generator<mt19937&, normal_distribution<double> >
  normal_sampler(rng, norm_dist);
 
  while(*go){
    double val = normal_sampler();
    a += val;
  }
  return 0;
}

I already had Boost, and I’d never used their pRNGs before, so I decided if it was a copy/paste job, I would give them a test run. Sure enough, it plugged right in.

The main function is even simpler:

int main()
{
  bool go = true;
  boost::scoped_ptr<boost::thread> threads[NUM_THREADS];
 
  for(int i=0; i<NUM_THREADS; ++i)
  threads[i].reset(new boost::thread(fn, &go));
 
  std::string asdf;
  cout<<"enter any input and hit <Enter> to kill"<<endl;
  cin>>asdf;
 
  go = false;
 
  for(int i=0; i<NUM_THREADS; ++i)
    threads[i]->join();
 
  return 0;
}

Within 12 minutes of starting, I was nice and toasty!

I posted the code on Github in case I needed it somewhere else: [code]


“Programs must be written for people to read, and only incidentally for machines to execute.”

~Abelson and Sussman, SICP

If 8 years of programming has taught me anything, it’s that programmers need to give themselves every advantage imaginable. I’m not disparaging other programmers.  I say this as someone who has been humbled on the altar of bugs many times.

The fact that my first language (C++) is one of the most complex languages probably didn’t help. C++ isn’t my favorite language, but I always find myself coming home to it, whether for school assignments, or more recently for money. Plus, it ranks higher than Java in my book. Yuck!

Most programming languages have fundamental shortcomings that prevent them from being very human-friendly. Like what? Like nested parentheses and braces and brackets. Fortunately, along with programming languages evolved code editors that do great things with indentation, coloring, and matching brace highlighting. The actual problem isn’t fixed (a colored, indented C++ program is still the exact same thing as a non-colored, non-indented C++ program), but the tools exist to make it a non-issue.

There is a wrong way to do it, and a right way to do it, but none of it is enforced by the compiler.

Assignment Within if(){

When I was 14 and 15 and hadn’t been using the language for very long, I made a whole litany of embarrassing mistakes. For instance, I would spend an hour trying to figure out why the following wasn’t doing what I expected:

 for(int i=0; i<10; ++i);{
     do_something(i);
 }

Fortunately, I stopped falling for it after a few times. However, I couldn’t seem to shake the following error:

 if(my_ptr = NULL){ // Wrong, but no compile error!
     do_something();
 }

To the rookie, this looks correct at a quick glance. Even worse, it compiles without warning. In fact, this was one of the evil no-nos that Java outlawed altogether, even though it has legitimate uses.

No matter how many times I swore I’d look closer, 2 weeks later I’d find this same bug after 30 minutes of frustrated debugging.

Eventually, I eliminated a large subset of this error by writing the equality test differently:

 if(NULL = my_ptr){ // compile error!
     do_something();
 }

This caused a comparatively small boost in my productivity. However, this helped, and it required no effort. Every little bit counts. Interestingly, this kicked me out of my groove: I started making this error a whole lot less frequently because I was thinking more about my equalities. Unless projects have strict style guides, I still code it this way.

For being a simple style change, it comes with a few benefits:

1. Anybody can understand this code.
2. When you screw up, it doesn’t compile.

//TODO: Lists

Some code generators, like Visual Studio 2003, fill generated functions with comments:

 void importantFoo()
 {
    // TODO: Implement this!
 }

For the authors of the code generator, this suits its purpose: informing the programmer that they have some work to do while generating compilable code.

Believe it or not, some code generators improve upon this. For instance, I’ve seen some code generators taking an exceptional approach:

 void importantFoo() {
     throw std::runtime_error("TODO: Implement function importantFoo()!");
 }

This comes with benefits: unless the programmer is a complete ninny, this code will be executed, and the nascent program will be prematurely snuffed from this Earth.

If this doesn’t cause the programmer to get off his lazy rear end and code, then what will?

The // TODO! statement is certainly not unique to code generators: it seems to be pretty popular amongst programmers, too. I am no exception them, too.

For a while, I took the comment approach to the TODO:

 void my_foo() {
     // TODO: Implement me! I am a yummy pizza!
 }

I’ve always felt a little dirty writing these. Realistically, your brain’s stack can only hold 5-9 items, so I can see an argument for temporarily using them. However, all you’re doing is passing the buck to an older, wiser future version of yourself in a brave new world with flying cars and butler monkeys.

It’s obvious that if you had flying cars and butler monkeys, the LAST thing that you would want to do would be to implement a // TODO statement left by a barbaric past self.

It’s my opinion that if you HAVE to write one, you should resolve the issue ASAP. For instance, before you  compile or interpret or “(read)” or whatever your language uses.

But recently, I found an even better TODO method for compiled languages:

 void my_foo() {
     TODO: Implement me! I am a yummy pizza!
 }

In C++, this generates about a trillion compiler errors (more or less). Compiler errors, for whatever reason, put me on a mission to fix a problem, so this is is an easy and helpful motivator.

I call these “Quit your bitchin’” TODO lists: Quit your bitchin’ and implement me! You have to do it sooner or later!

It is important to note that you should NEVER do this in Python. Python is expressive enough that this syntax will likely cause the International Space Station to fall to earth while buttering bread on the wrong side. </bad_joke>

Conclusion

Modern programming languages are designed  with human readability in mind, but there isn’t a “perfect language” in this regard. Best practices develop around programming languages, and is imperative that newbies find out what these are ASAP, because they are only hurting themselves if they do things their own way.

In addition to using best practices, sometimes you can make things easier for yourself with little effort.

Lisp is influencing the way I write C++ in a few different ways.

To Generalize…

Imagine my horror: Lisp caused me to realize that I’ve been refactoring the wrong way! For a few years!

Well, that is not entirely accurate. My code did the same thing after I was done. I broke code down into small logical units. Unless I was careless, the code still worked. My functions were more readable, and one could usually see what they did at a glance.

I asked myself questions along the lines of, “What can I do to break this up logically?”, and therein lies the problem:

Weak questions have weak answers.

Powerful questions are much better suited to refactoring!

• “Can I use an existing solution to solve this?”
• “Could this solve a completely unrelated problem?”
• “Can I rearrange things elsewhere so that this code isn’t even needed?”

Sometimes, the answer is no. Some boilerplate code (like exception catching) is an overhead that can not be avoided, can not usually be generalized, and sticks out like a sore thumb in C++ style languages.

Why did Lisp fundamentally shape my refactoring practices? Paul Graham and Peter Norvig are both to blame for this, and for different reasons.

First up, Graham. I taught myself from “ANSI Common Lisp“. At first, I attacked the problems in the book from a C++ perspective, trying to wedge loops and functions into my solutions. Doing the Lisp examples forced me to think simpler. Within a month I started being able to think of functional solutions instead of C++ solutions (reaching for reduce and lambdas instead of loops and classes).

I worked problems out of this book every night for a month and a half, and it still sits next to me as a quick reference. After 8 years of C++ programming, it took a while for things like “reduce” to become the most obvious solution to a problem.

I haven’t reached that magical enlightening moment that Lisp causes. I don’t have the “ooh, shiny!” awe over macros that seasoned Lisp programmers seem to have, so I’m still hacking away.

Norvig has sent me towards enlightenment with Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp. In these pages, I found something that was missing from Graham: elegance

Some of the solutions that I see in this book give me a sense of awe and enlightenment that I haven’t had since I took Abstract Algebra. He presents them in a very straightforward and honest style that can only come with experience.

This book has given me a whole new level of quality to try to reach. I hope one day to be able to do this.

Tools

A year ago, I would have described my programming process like this:

1. Design
2. Code + document
3. Neaten + prune
4. Return to 1, as needed

Not bad. However, working with Lisp and seeing that there are different tools in the world has led me to fundamentally restructure how I program. I would now describe my process like this:

1. Design
2. Code + document
3. “Would a tool/function/class have made this task easier?”
4. Return to 1, as needed.

For instance, I’m writing some image processing libraries in CL right now, and I’ve written a few tools that I might never have thought of generalizing from a pure-C++ mindset! After all, iterating over an image is really easy in C++, so why bother making it simpler?

However, map-image is invaluable: apply a function at each pixel of an image, and return a new image. Combine this with the lambda construct, and you have a level of flexibility possible only with some of the goofier Boost libraries.

In Short…

While working through the Graham examples, I found that I was playing Code Golf against myself.

I’ve started to see this carry over to my coding at work as well. In the past, I might have done something like the following:

int num_times = SomeFunction();
 
return AnotherFunction(num_times);

This is a reasonable approach. The local variable name reflects the role of the function, and adds a little bit of clarity for the human reader. Using this style is fine if these are your priorities.

However, I’ve started giving functions better names (when I can), and cutting out the middleman:

return AnotherFunction(NumberOfTimes());

This is effectively the same code, and there’s a good chance that it would compile down to the same assembly language. It’s just as readable. The difference is the line count. It’s a difference small enough for a bikeshed issue holy war, but over the course of small projects (all I’ve done so far in Lisp), being able to fit more code on my screen is a big win for me (all else equal).

I am starting to prefer reading and writing compressed code as a result of trying to solve Graham’s examples in as few lines as possible.

Future Directions

I’m still looking for my “Lisp Enlightenment” with respect to macros, which leads me to believe that there’s something significant that I’m still not getting about Common Lisp. I’m not sure where all of these new techniques are going to lead me, but in the meantime, it is starting to fundamentally change how I approach programming problems.

The Algorithm

The algorithm is divided into three stages: precalculation, encryption, and decryption. Precalculation is performed a single time for each person with a public/private key pair, and encryption/decryption is performed for each message.

Precalculation

1. Pick primes: Find two arbitrarily large prime numbers, p and q.

1. Determine the modulus: Multiply p and q to get n. This will be your modulus for all equations.

1. Calculate phi(n): Using Euler’s totient function, calculate . This number is a secret number, so don’t give it away. The strength of the algorithm depends on being hard to calculate when given a sufficiently large n = p * q

1. Determine an encryption exponent: Take any number, e, such that GCD(, e) = 1. This means that they are relatively prime, and share no common factors. This number is considered your public key (when combined with n), and you can give this number to whoever you like.

1. Compute the decryption exponent: Solve the Extended Euclidean Algorithm of GCD(, e) to find . This is your private key.

We have a private decryption key pair: {n, d}, and a public encryption key pair: {n, e}.

Encryption

is the encrypted message, and it can safely be sent publicly.

Decryption

is the decrypted message. The person with the private key for the message will be able to read it, and theoretically, nobody else.

Is the message preserved?

Applying the Corollary to Euler’s Theorem to , we get .

We also notice that

Why is it hard for an attacker to crack?

In a perfect world, the attacker needs to solve some equivalent of the Integer Factorization Problem to factor n, which is suspected to be outside of complexity class P using classical computation. If we have access to quantum computers, we have access to an algorithm, Shor’s Algorithm, to crack integer factorization efficiently, but probabilistically.

The essential problem that the attacker faces is as follows: they have the encryption exponent and the modulus. They know what n is, but they currently have no method of calculating what is without first factoring n. They need to find the decryption exponent, d = ,, but can’t find the exponent without being able to solve the Extended Euclidean Algorithm, where they need to know the value of . So long as n is hard to factor, RSA will remain difficult to break.

We don’t live in a perfect world, and there are plenty of examples of attacks that take advantage of weak implementations of RSA. See RFC 3447 for some best practices.

The Math

Extended Euclidean Algorithm

The Euclidean Algorithm finds the Greatest Common Divisor (GCD) of two integers, a and b. The Extended Euclidean Algorithm finds integers m and n in the following equation:

Euclidean Algorithm

To see how it does this, we will look at an example. Let a = 200, b = 37. Please note that 200 = (101 * 3). Since (200, 37) = 1, this is the equivalent of trying to find 37-1 (mod 200). For previous writings on the material, click here.

1. 
2. 
3. 
4. 

Line 3 is considered the “final” line of the algorithm because it is the last line where the remainder is nonzero. The remainder of that line, 1, is also GCD(200, 37).

Extended Euclidean Algorithm

We start by rewriting every equation so that the remainder is on the RHS.

1. 
2. 
3. 

From here, we start at line 3, and substitute in line 2. Don’t simplify any multiplications, just the additions.

Substitute line 1 into the resulting equation.

[Unparseable or potentially dangerous latex formula. Error 5 : 533x369]

We now have everything that we need for the inverse of 37. It turns out that is equivalent to saying:

so 173 is the multiplicative inverse of 37 (mod 200). Neat, huh?

Fermat’s Little Theorem + Euler’s Theorem

In a letter written in 1640, Fermat (of Pythagorean fame) offhandedly mentioned that he had noticed and proved the following relation:

Fermat’s Little Theorem

For a any integer, and p prime,

As he is occasionally noted for doing, he did not bother to write down the proof. Leibniz is said to have proved it, but didn’t get around to publishing it. He was an inventor of calculus, so we’ll cut him some slack.

Enter Leonhard Euler. Euler was 18 feet tall, shot laser beams from his eyes, and proved Mathematics every second he was awake. In 1736, he took 23 minutes off from a conquest of Mars to prove Fermat’s Little Theorem.

This wasn’t enough, though. He attempted to find a way to generalize it for any number, n, instead of just for primes, p, but couldn’t. The problem tortured him for 24 years, when in 1760* he was finally able to produce his proof.

His proof makes use of a function he defines, phi(x).

Euler’s Totient Function

For any positive integer, n, is equal to the number of positive integers where GCD(a, n) = 1, for a < n.

Most germane to this discussion, for any prime, p, . This makes perfect sense, of course, as a prime is indivisible, and therefore all numbers less than p don’t share factors with p, or else p could be divided!

The formula ends up being:

Euler’s Theorem

For any positive integers a, n:

* Wikipedia says 1736, my number theory book says 1760, tie goes to the book.

Corollary to Euler’s Theorem

Because , we can reduce the multiplication needed for any power of a > n.

Edit: added link to Wikipedia’s article on Integer Factorization, and a better explanation of why the attack is hard

Almost every major program that works with text performs string searches. There are some juicy algorithms in this area, each with their own tradeoffs. Choosing the wrong algorithm for the wrong task produces awful results, and we need to carefully weigh the consequences of each algorithm. I’m going to take a look at just three of them: brute force, Boyer-Moore, and Rabin-Karp.

If you want to cut straight to the chase, feel free to scroll to the bottom to see the results section.

Why these three? Each of these algorithms offers a completely different perspective on the idea of finding strings. There are other algorithms that perform specialized tasks (Knuth-Morris-Pratt for repetitive source strings, for example), but we’re not interested in these today.

Brute Force: Simple, and (Sometimes) Slow

As we see in the image below, the brute force algorithm tries to generate a match from every single starting position.

It should be obvious to the reader that the algorithm can’t possibly miss a match, as every single position is tried. If the pattern exists in the text, brute force search will find it.

Note: Green represents a single character match, and Red represents a single character mismatch.

The string “This is some Test text” isn’t very tough on the algorithm, as there is only one mismatch before we find the correct position of the word “Text”. However, it is very easy to construct pathological cases for this algorithm. It doesn’t perform well for binary data, or any highly-repetitive pattern. If it has to match the pattern of a whole bunch of As with a B, it could very easily perform poorly. If we consider finding AAA..AAB in the string AAA……….AAAB it’s obvious that the algorithm will be taken down a costly blind alley at every single position

Implementation in C++

To check to see if we have a match at any position, all we have to do is check to see if each character of the string is a match. If it is not, we can return false.

bool brute_force_match(const string& text,
                       const string& pattern, int pos)
{
    // If the source text is shorter than the
    // pattern + the position, we can't possibly
    // have a match.
    if(pos + pattern.size() > text.size())
        return false;
 
    int pattern_size = pattern.size();
 
    // Check every single character for a mismatch.
    for(int i=0; i

To search every position, we use the following snippet:

int brute_force(const string& text, const string& pattern)
{
    int search_size = text.size() - pattern.size();
 
    for(int i=0; i <= search_size; ++i)
    {
        if(brute_force_match(text, pattern, i))
        {
            return i;
        }
    }
    return -1;
}

Brutally simple!

Complexity and Potential Problems

In certain document types, brute force is a valid solution for finding a string. Looking for strings in natural language files is still somewhat fast, as the algorithm is likely to into mismatches pretty fast. There are better algorithms (like Boyer-Moore, as written below), but the implementation time of brute force is not to be taken lightly.

When we don’t have a source string with a large number of “false starts”, we can expect the runtime of the algorithm to be roughly linear with the size of the file. It will have a runtime of roughly M+N, for M = text size, and N = pattern size. However, for searching in binary strings, or in strings with many repetitions of close matches to our pattern, the runtime is closer to M*N.

It takes less than 1/10 of a second to find a small string at the end of “Moby Dick” (see below)! I wasn’t joking when I said that brute force is a fast algorithm for certain inputs. Treat this algorithm like Bubblesort: use it for the plainest and smallest inputs.

Boyer-Moore: Right-to-Left Trickery

One of the assumptions that we made for the brute force algorithm is the left-to-right search. The direction of the search isn’t a requirement. In fact, we can search through the text however we please!

Since we are given the whole pattern string, we have a tremendous amount of information at our disposal. We just need to know what to look for. For this algorithm, when we come across a mismatch, there might be information from the mismatch that helps us speed up the search.

Let’s see an example of how we can use right-to-left searching combined with mismatches to search faster:

Note: Green represents a single character match, and Red represents a single character mismatch.

We see that on step 1, the ‘s’ doesn’t match the ‘t’, so we slide the pattern forward 1. Please note that this will line the ‘s’ in “Test” up with the ‘s’ in “This”.

Step 2 demonstrates the magic of the algorithm: there isn’t a <space> at all in “Test”, so we can completely skip over the letter in the source text wholesale. We see this happen again in step 4 with the ‘o’. There is no ‘o’ in “Test”, so why waste time trying to match it?

The Boyer-Moore algorithm takes advantage of these jumps. We search from left to right, looking for the first mismatch. When we find one, we can compute how far we’ll need to slide our search pattern ahead in order to match the character. We can also use a lookup table to speed up the calculation of how far we need to jump.

Implementation in C++

The size of the lookup table for a pattern of N characters is 256 * N bytes for ASCII alphabets, which is cake on modern computers. For larger alphabets, such as Unicode, you need an associative data container in order to be able to implement the algorithm in memory.

The algorithm itself is very easy to follow:

int boyer_moore(const string& text, const string& pattern)
{
  int search_size= text.size() - pattern.size();
  int other_ind, word_start;
  vector > skip_table;
 
  initialize_skip(skip_table, pattern);
 
  for(word_start=0; word_start<=search_size; ++word_start)
  {
    for(int i=pattern.size()-1; i>=0; --i)
    {
      // If there is a mismatch
      if(text[word_start + i] != pattern[i])
      {
        other_ind = (int)(text[word_start +i]);
 
        // Skip forward the proper number of steps.
        // Look this up in the lookup table.
        word_start += skip_table[i][other_ind];
 
        break;
      }
 
      // If we get to the beginning of the string
      // and still haven't found a mismatch,
      // we have the position.
      else if(i == 0)
        return word_start;
    }
  }
 
  return -1;
}

As you see, once we’ve built the skip table, it’s very easy to look up how much the character pointer should skip forward at each step.

But how do we build the skip table? Let’s look at initialize_skip()

Let’s say that we have the string “testing”. The skip table for the letter ‘i’ will start with every single value initialized to the maximum skip, 5.

We then iterate backwards through the string, finding the first instance of each letter. If we find a “t” instead of an “i”, we can only skip 1. If we find “s” or “e”, we can skip 2 and 3 respectively.

Notice that we don’t assume that it might be the “t” that starts the word. If we did that, we might jump the pattern too far and miss the match. We only want the first match going from right to left.

void initialize_skip(vector >& skip_table,
                    const string& pattern)
{
  unsigned int pattern_size = pattern.size();
 
  for(int i=0; i
 to_push(ALPHABET_SIZE, max_skip);
 
    // Have we seen a character yet?
    vector used(ALPHABET_SIZE, false);
 
    int counter = 0;
 
    for(int j=i-1; j>=0; --j)
    {
      // If we haven't used a character
      // yet, add its value in the lookup table
      if(!used[(int)(pattern[j])])
      {
        to_push[(int)(pattern[j])]=counter;
        used[(int)(pattern[j])] = true;
      }
 
      counter++;
    }
    skip_table.push_back(to_push);
  }
}

Complexity and Potential Problems

The worst case runtime of this algorithm is still N*M, although the average case for many different kinds of text (including natural language), is roughly M/N.

Pathological cases are still easy to construct. Try coming up with one yourself, or scroll down to the test section to see a sample string that can be pathological!

Worst case aside, this algorithm is incredibly useful on many different kinds of inputs, especially natural text. Having an average of M/N runtime is awesome, and is heads and shoulders better than brute force.

Rabin-Karp: Clever Use of Hash Values

Extreme Math Warning

The above algorithms both suffer from the same weakness: It is easy to construct pathological cases. If consistency is the goal of your program, these algorithms obviously would fall flat on their face for certain input types. It doesn’t matter if the algorithm runs fast in some cases: as soon as your user has to wait for your program to respond, they’re not happy.

Enter the Rabin-Karp string searching method. It treats the string like it is a large number that has been entered into a hash table. We consider consecutive characters in a string all as the same number.

For example, have the string “abba”, with the alphabet size = 256, the hash value is calculated as follows:

For more on the subject, see my older posts on number theory and hash tables.

The algorithm is as follows: we first generate the hash of the pattern, and the hash of the first N characters of the text. If the two hash values are equal, great! If they’re not equal, we remove the highest term, multiply the value by the alphabet size, and then add the next term. Simple, right?

Actually, I didn’t even understand that. Let’s see it written out with TeX.

The source text string: aabba
The pattern to find: abba

The first four letters of the source, hashed:

This is extraordinarily unlikely to produce a hash value that matches the hash of the text “abba”. We first remove the highest term, as we no longer want it in the hash.

We then multiply the value by the alphabet size, 256:

And add the next character in the sequence, a:

Which obviously DOES hash to the value of the search string “abba”, as it’s the exact same thing we had above. Brilliant!

There is a small chance that there will be a collision for text values that don’t match our source string. We still need to check the string in place in order to make sure that we have the correct string, and not one that coincidentally matches.

Implementation in C++

int rabin_karp(const string& text, const string& pattern)
{
    int search_size= text.size() - pattern.size();
    int pattern_size = pattern.size();
 
    // A large prime such that prime < (2^32)/(ALPHABET_SIZE+1)
    //
    // This is our hash value
    unsigned int prime = 5800079;
 
    unsigned int alphabetM=1;
 
    // Less likely to overflow than powmod()
    for(int i=0; i
<=search_size; ++i)
    {
        if(patternHash == textHash && brute_force_match(text, pattern, i))
            return i;
 
        textHash = (textHash + ALPHABET_SIZE*prime
                        - (int)(text[i])*alphabetM) % prime;
 
        textHash = (textHash*ALPHABET_SIZE
                        + (int)(text[i+pattern_size])) % prime;
    }
    return -1;
}

Complexity and Potential Problems

The worst case of this algorithm is M*N, if you were to come up with a document where every single location hashes to the same value. However, this is extremely unlikely to ever happen, even if you’re trying to craft the worst-case document. My test of “Moby Dick” didn’t even come up with a single collision in the entire text of the novel, and the book is pretty damn big.

Running the Algorithms

Let’s take a look at how the algorithms fare when they compete to find the string “devious-cruising Rachel” in the text of “Moby Dick”. Aside from being a hysterical phrase, “devious-cruising Rachel” appears a single time in the book: in the very last paragraph.

The program was compiled with -O2

Loading Moby Dick
 
Text length is 1201110 characters
 
Position of "devious-cruising Rachel": 1201001
Brute force took 0.007865 seconds
 
Position of "devious-cruising Rachel": 1201001
Boyer-Moore took 0.001762 seconds
 
Position of "devious-cruising Rachel": 1201001
Rabin-Karp took 0.025224 seconds

And Boyer-Moore is the winner by a long shot!

Rabin-Karp is much slower on this run! However, we are going to get consistent performance on that algorithm, so don’t count it out yet.

Let’s see how the various methods deal with the worst-case for the brute force algorithm:

Loading Brute Force torture test
 
Text length is 11015500 characters
 
Position of "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab"
        : 110113
Brute force took 0.010162 seconds
 
Position of "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab"
        : 110113
Boyer-Moore took 0.001673 seconds
 
Position of "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab"
        : 110113
Rabin-Karp took 0.00218 seconds

Rabin-Karp has pulled into second for this particular run. Again, it’s a consistent bugger.

And now let’s look at a worst-case file for the Boyer-Moore algorithm:

Loading Boyer-Moore torture test
 
Text length is 11015556 characters
 
Position of "baaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
        : 110154
Brute force took 0.000656 seconds
 
Position of "baaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
        : 110154
Boyer-Moore took 0.010248 seconds
 
Position of "baaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
        : 110154
Rabin-Karp took 0.00221 seconds

We see the Boyer-Moore algorithm has an awful runtime, from all of the backtracking that it is doing. Again, the Rabin-Karp algorithm is looking pretty consistent!

Where Do I Go Next?

The algorithms described above were written to find a single pattern in a string of text. We still haven’t considered the problem of finding all of the matches for a certain string in a document. There is also the Knuth-Morris-Pratt algorithm, an algorithm that works well for strings that are highly repetitive in nature, and is an improvement upon brute-force otherwise.

I’ll see your sensational headline and raise you a better algorithm!

The Code:

#include "gmpxx.h"
#include 
#include 
 
using namespace std;
 
mpz_class fib(unsigned int num)
{
  mpz_class a(1), b(1);
  mpz_class temp;
 
  while(num-- > 1)
  {
    temp = a;
    a = a + b;
    b = temp;
  }
  return a;
}
 
#define NUM 200000
 
int main()
{
  cout<<"Fibonacci test for Fib("<<<")"<<<"Time: "<<<" seconds"<

Build With:

g++ fib.cpp -O2 -lgmp -lgmpxx

The Result: (On a Toshiba Laptop w/ Core 2 Duo @ 1.6GHz)

  Fibonacci test for Fib(200000)
  Time: 2.44233 seconds

That’s a lot better than what we can get in Haskell using the detailed multicore approach. 32 seconds to calculate fib(45)? Please!

The Lesson:

1. There’s more than one way to get a speedup in your code, and you should carefully weigh your options when writing. This algorithm can be bettered if you have phi calculated to many digits, or if you use a better recurrence, so I’m not claiming “optimal” results here!
2. Throwing more cores at a problem does not always solve it best
3. I found out that writing sensational headlines is actually pretty fun.

Man only likes to count his troubles, but he does not count his joys.
~Fyodor Dostoevsky

What is Combinatorics?

Combinatorics is the math behind counting. All problems that start with the phrase “How many ways..” are most likely combinatorics problem.

Who is this page for?

• Programmers who have a simple background in Mathematics, or need a quick reference for how to implement combinatorial algorithms.
• Those who did not know what the word “combinatorics” meant.

Factorials

Factorials are used to solve the problem, “How many ways can I arrange n objects?” It makes the following assumptions:

• Each object is distinguishable (Can I tell apart the n apples?)
• You can actually distinguish arrangements of objects (Can I line up n apples in a meaningful way?)
• The elements are non-replaceable (Once I use an apple, I can’t use an apple again).

The factorial function is defined as follows for positive integers:
Factorial(1) = 1
Factorial(n) = n * Factorial(n-1)

or, the naieve C implementation:

  // For 32 bit systems, will not return meaningful results
  // for values larger than 12!
  //
  // For 64-bit systems, will not return meaningful results
  // for values larger than 20!
  unsigned int factorial(unsigned int n)
  {
    unsigned int ret = 1;
 
    for(int i=2; i<=n; ++i)
        ret*=i;
 
    return ret;
  }

The C implementation above is fine for the values it can logically compute. However, when we get into arbitrary precision integer calculations, calculating 1 * 2 * 3 * … * n is inefficient! The number of words produced at the end of each step grows linearly. If we divide-and-conquer, we keep down the number of words-per-integer until the very last few multiplications, which produces significant time savings.

Of course, we would NEVER use this in the real world without extra sanity checking of the input. The importance of sanity checks in the real world is demonstrated every day by celebrities. Don’t commit their same blunders.

  mpz_class factorial(mpz_class begin, mpz_class end)
  {
    if(end==begin)
      return end;
 
    if(end-begin == 1)
      return begin*end;
 
    int half = (begin+end)/2;
    return factorial(begin, half) * factorial(half+1, end);
  }

In fact, running the two against each other to find 100,000! produces the following runtimes on my laptop (I was going to do 1 million and got bored waiting for the iterative solution to finish!):

  g++ factorial.cpp -O2 -lgmp -lgmpxx
  ./a.out
  Recursive factorial: 0.2795 seconds
  Iterative factorial: 3.01006 seconds

The most efficient ways to implement factorial algorithms turns out to be from its prime factorization, but the above is far easier than implementing sieving methods to find primes! Maybe another day

Permutations

Permutations are how we solve the problem, “how many different ways can I arrange k objects from a list of n objects? To give an example, assuming the same number could not be used twice in a combination, your “combination” lock from high school was actually a permutation lock. You were given a single permutation that you could use in order to open it. But how many different possible permutations were there, given 45 as a max value and 3 numbers?

• The first number has no restrictions. You can choose from all 45 numbers
• The second number is restricted: it can’t be the same as the first number. You now have 44 to choose from, giving you 45 * 44 different possible 2-permutations of 45
• The third number is also restricted: You can only choose 43 numbers. 45 * 44 * 43 is the number of 3-permutations of 45.

It’s not hard to convince ourselves that the permutation is all of the numbers between and including 45 and 43 multiplied together. Symbolically, we are talking about all of the numbers between and including n and (n – k + 1), inclusive.

The formula we all learned in school (which you can convince yourself is equivalent to the above) is as follows:

  // Picking and ordering k objects from n objects
  mpz_class permutation(mpz_class n, mpz_class k)
  {
      return factorial(1, n) / factorial(1, k);
  }

Never do this! You have to recompute many of the values that you never use, and it is worthless and computationally wasteful to do so. Instead, you can use the following:

  mpz_class permutation(mpz_class n, mpz_class k)
  {
      if(k == 0)
          return 1;
 
      return factorial(n - k + 1, n);
  }

It’s much simpler. It computes much less. Use it. It is calling your name. Use it.

Combinations

Permutations are very closely related to combinations. In fact, there is only a one word difference in the definition. Combinations are unordered, and permutations are ordered. We can derive the formula by what this implies.

Let’s take our combination lock example. We needed to get three numbers in a row, in order. But what happens if we only need to select the three numbers without concern to the order? We know from the factorial section that the number of ways to order 3 numbers is 3 factorial. So let’s say we have the permutations { (1, 2, 3), (1, 3, 2), (2, 1, 3), (2, 3, 1), (3, 1, 2), (3, 2, 1) }. These are all the same combination, because they all pick the same exact elements! So there is clearly 1/6 the number of combinations as permutations. 6 is 3!, which is no coincidence. In general, the formula for combinations is:

  mpz_class combination(mpz_class n, mpz_class k)
  {
      if(k == 0)
          return 1;
 
      return permutation(n, k)/factorial(1, k);
  }

*Hits the Easy Button*

Next week: More complicated number sequences such as the Binomial Coefficients and the Sterling Numbers

We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.
~Donald Knuth

You’ve designed and written your application. You’ve carefully analyzed the algorithm, and it can’t possibly have a lower algorithmic complexity and still do its job. However, your program is still taking 11 seconds to iterate through all data, and that’s absolutely unacceptable. You may now begin to optimize your code.

At this point, inexperienced programmers fumble the ball here: they begin to try to figure out where the bottleneck is in their program. By themselves. By looking at their code.

Always profile your code when you want to improve it.

No exceptions. When optimizing for speed, you always need to know which 20% of the code is taking up 80% of the execution time. A profiler will easily (and always) tell you which functions are being called the most, and how much time your program spends executing them. Experience can only give you a guess as to what should be improved, but profilers always tell you where the code is spending its time being executed.

It’s not magic, it’s common sense.

Understand the Tradeoff of Optimization

• Legibility: What may read as an easy three-liner may be needlessly taking up 40% of the processor time.
• Development effort: Significant rewrites may be necessary in order to gain the performance boost suggested by the profiler.
• Maintenance: Extra development time may needed to be spent to write test cases for all of the changes that you have incorporated into the program. These cases, along with the (potentially) less legible code need to be maintained.
• Memory vs. Speed: Frequently, in order to have a faster program, we need to use more memory. Make sure your new code isn’t consuming unacceptable levels of memory.
• Platform-specific optimizations: Some optimizations are only possible on one computing platform, and others work far better on one platform than they do another. If your goal is truly platform-independent code, you have the burden of showing that your optimizations work on all platforms, and not just the target platform.

How will we profile code?

I will be using gprof and C++ in order to produce examples, but the ideas presented below will be universally applicable between procedural and object-oriented languages. I’ll let those who are experts with functional languages to talk about functional languages, and in my experience, they will.

I am not teaching you how to use gprof (though it is rather easy). I am showing you how to do some of the optimizations I most frequently find myself doing, and the lessons to be learned from them.

Beware of Hidden Complexity!

Assume we are trying to optimize a small code snippet, as follows:

vector a(vect_size),
  b(vect_size),
  result(vect_size);
 
/*
  Snip: Initialize data, perform initial calculations, etc.
*/
 
for(unsigned int i = 0; i < a.size(); ++i)
{
        result[j] = a[i] * b[i];
}

And you might find that the output for gprof would be something like this:

 %   cumulative   self              self     total
 time   seconds   seconds    calls  us/call  us/call  name
15.66      0.06     0.03  4000000     0.00     0.00 __gnu_cxx::__normal_iterator > >::operator+(long const&) const
12.53      0.08     0.02  2000002     0.00     0.00 __gnu_cxx::__normal_iterator > >::base() const
9.40       0.09     0.02  4000000     0.00     0.00  std::vector >::begin()
6.27       0.10     0.01  8000000     0.00     0.00 __gnu_cxx::__normal_iterator > >::__normal_iterator(int* const&)
6.27       0.11     0.01  4000000     0.00     0.00 __gnu_cxx::__normal_iterator > >::operator*() const
6.27       0.12     0.01  4000000     0.00     0.00  std::vector >::operator[](unsigned long)
6.27       0.13     0.01  1000001     0.00     0.00  std::vector >::end() const
6.27       0.14     0.01  1000001     0.00     0.00  std::vector >::size() const
6.27       0.15     0.01  1000001     0.00     0.00  std::vector >::begin() const
6.27       0.16     0.01        1    10.03   160.41  calculate(int)

These charts can be hard to read, but they extremely telling. The type of tasks that are taking up our processor time are: iterator manipulation.

But wait, we’re not using iterators!

Yes we are. Our calls to vector::operator[] are apparently taking up a lot more time than we ever could have imagined, and are using iterators in order to accomplish the task. Instead of using the index operator, let’s write the program the STL way and see what happens (Java, C, etc.. coders, take this as a general lesson instead of a language specific lesson!).

Maximize the Utility of Function/Method Calls

So we now replace all of the loop code with the following:

vector::const_iterator a_iter = a.begin();
vector::const_iterator b_iter = b.begin();
 
for(vector::iterator r_iter = result.begin();
    r_iter != result.end();
    ++r_iter)
{
   *r_iter= (*a_iter) * (*b_iter);
   ++a_iter;
   ++b_iter;
}

and we get the following output from gprof:

20.88      0.06     0.03  1000001     0.00     0.00  std::vector >::end()
16.70      0.08     0.02  1000001     0.00     0.00  bool __gnu_cxx::operator!= > >(__gnu_cxx::__normal_iterator  > > const&, __gnu_cxx::__normal_iterator > > const&)
 8.35      0.09     0.01  3000004     0.00     0.00 __gnu_cxx::__normal_iterator > >::__normal_iterator(int* const&)
 8.35      0.10     0.01  2000000     0.00     0.00 __gnu_cxx::__normal_iterator > >::operator++()
 8.35      0.11     0.01  1000001     0.00     0.00  std::vector >::size() const
 8.35      0.12     0.01  1000000     0.00     0.00  std::vector >::operator[](unsigned long)
 4.18      0.12     0.01  2000004     0.00     0.00 __gnu_cxx::__normal_iterator > >::base() const

For those who are too lazy to scroll across (myself included), the function taking up 20% of the CPU time is… vector::end()! What’s happening is that we’re constantly constructing and returning an iterator to the last element in r_iter, and it’s hurting us, time-wise. Since the end element never changes, it makes no sense to constantly recalculate it. We should store it for future generations of the loop to enjoy, much as one might put the most recent copy of People Magazine in a time capsule for their children to enjoy the tales of celebrity compound-name relationships.

So, we rewrite our loop as follows:

vector::const_iterator a_iter = a.begin();
vector::const_iterator b_iter = b.begin();
 
vector::iterator r_iter = result.begin();
vector::iterator r_end = result.end();
 
for(; r_iter != r_end; ++r_iter)
{
   *r_iter= (*a_iter) * (*b_iter);
   ++a_iter;
   ++b_iter;
}

and we observe another speed increase.

How much of a speed increase? The test program that I wrote in order to get these outputs took 5.11 seconds to run unoptimized. After profiling, it runs in 3.70 seconds. Not bad for a day’s work!

This hints on a broader lesson: reduce the crap in your loops as much as possible. Move it all as far outside as possible, and calculate it as few times as you can.

Be Nice to the Cache

While profiling can give us extraordinary speed increases solely by using innocuous code, it is not perfect. There is a much better way to write the above code that allows it to perform the same calculation with fewer cache misses. This rewrite would never ever be noticed solely through gprof.

This article is an excellent overview of how modern CPUs implement the cache, and it certainly taught this Jake new tricks. I recommend that you at least glance through it in order to truly understand how it works.

So how do we write the above loop to better use the cache?

struct calc_info
{
  int data1, data2, result;
};
 
/*
  snip
*/
 
vector data;
 
vector::iterator d_iter = data.begin();
vector::iterator d_end = data.end();
 
calc_info* a;
 
for(; d_iter != d_end; ++d_iter)
{
   a = &(*d_iter);
   a->result = a->data1 + a->data2;
}

The above code finishes in 2.31 seconds. Why does it finish so fast? In the previous version of the program, whenever it calculated one of the result residuals, it took a value from the first vector and added it to a value from the second vector. For sufficiently large vectors, the values being calculated wouldn’t be anywhere NEAR each other in the memory. They wouldn’t even be in the same galaxy. It might even need to pause and fetch data from the disk. That’s a HUGE speed killer. The same thing could happen when it stores the residual as a result. Cache misses galore!

The rewritten method puts all data that is being calculated next to each other. There is rarely the need for the cache to load new data, because all of the data was already transmitted to the cache, and is sitting in memory next to each other. The profiler never would have turned it up.

So should we avoid profiling?

No. We got some excellent speed increases without having to resort to unsafe arrays, and we did it in next to no time just by using a profiler!

Is profiling a complete substitute for experience?

No. We got a huge speed boost just by knowing how the cache works, and I’m 100% certain that I missed details that would let it perform in even less time.