My girlfriend and I recently went on vacation to Germany. Distant relatives of mine (Wolfgang and Birgit), agreed to host us for a few days, which turned out to be a great break from cheap hostels.

One day, Wolfgang tells us that they are taking us to Öhringen, and town about 16 kilometers away.

He quickly corrects himself for the benefit of the Americans. “That should be about nine miles.”

“It’s closer to 9.6″ I said without skipping a beat.

The speed of my answer surprised him. “Are you a quick multiplier?”

“Well, not exactly. Each kilometer is a little over .6 miles, so I add 1/2 and 1/10 the original number. It’s usually really faster than multiplying or dividing.”

It should be noted that this is an approximate method. The error is around 3.5%, as the conversion factor is closer to .621371. Most of the time, approximation are good enough with respect to distances. Once you have a handle on the distance, you can make all sorts of reasonable estimations, like driving time.

Figuring it out

Last March, I went on a road trip to Canada to compete in the CS Games with my fellow Computer Science majors. Since it’s a long effing way from TCNJ to Sherbrooke in Canada, we needed something to do. My buddy Steve realized that we accidentally printed our return directions in kilometers instead of miles. He read out one of the distances on the map and asked what it was in miles/hour.

We thought for a second, and then I looked at the speedometer and read off the corresponding speed in miles per hour.

“How did you do it so fast?”

“Mental math,” I replied.

“What’s the conversion factor?” somebody asked. I saw that 100km/hr was roughly 60 miles, so I said “about .6.”

Luckily for me, all of the distances Steve read off were under 160km (the limit of my speedometer), so I was able to use the speedometer trick for a while. I figured that charade couldn’t last long, so I broke down the problem (multiplying by ~.6), and saw easy fractions I could use. I continued using this flawlessly.

Going from miles to kilometers is just as easy. Since a mile is ~1.609 kilometers, you can get an even better answer with identical math.

For a distance n, just add together n + n/2 + n/10. This approximation leaves you within .6%!

It’s a piece of cake to develop new methods along these lines. Take the expansion of the multiplicative constant, c, pick the first few digits of c, and break it up however mental arithmetic is easiest. Each digit of c that you use gives you a better approximation.

Determining the relative error is easy: it’s the unused digits of c divided by c.

You probably use this method for calculating tips already. If n is the dollar amount of the tip, it is easy to calculate 10% of the tip: x = .1 * n. The full amount of the tip is x + x/2, giving you 15%. That’s 1/10th + 1/20th, or 10% + 5%. This particular application happens to be exact.

You can even determine a generalization for any affine method, such as temperature conversions, by also adding and subtracting rounded approximations.

Feynman Vs. the Abacus

The above is child’s play! Any kid who has learned division can figure out their own tricks. The real limits of human mental math are accomplished by mental lookup tables. Special numbers are the name of the game; the more special numbers you know, the easier it is to quickly solve a problem.

The best example I have found of this is a story in Surely You’re Joking, Mr. Feynman!, where he is challenged by a Japanese abacus salesman while visiting Portugal. I highly recommend you read the full text of the encounter.

The first challenge is addition. Feynman doesn’t stand a chance against the abacus, even when the numbers are written down and shown to both parties at the same time.

Next comes multiplication. Feynman still loses, but he has started to close the gap.

The next challenge is long division. It is a tie. This bothered the abacus salesman, so he challenges Feynman to cube roots. He writes the number “1729.03″ on a piece of paper and starts working the abacus. Feynman quickly figures out that the first few digits are “12.002″ while the abacus salesman struggles to even determine “12″.

As the book puts it:

How did the customer beat the abacus?

The number was 1729.03. I happened to know that a cubic foot contains 1728 cubic inches, so the answer is a tiny bit more than 12. The excess, 1.03 is only one part in nearly 2000, and I had learned in calculus that for small fractions, the cube root’s excess is one-third of the number’s excess. So all I had to do is find the fraction 1/1728, and multiply by 4 (divide by 3 and multiply by 12). So I was able to pull out a whole lot of digits that way.

Hooray Math!

The Feynman abacus story demonstrates one of the great things about math: every little fact eventually counts for something.

“The use of a program to prove the 4-color theorem will not change mathematics – it merely demonstrates that the theorem, a challenge for a century, is probably not important to mathematics.”~ From “Epigrams in Programming” by Alan J. Perlis

It’s hard to agree or disagree with this quote.

On one hand, results in mathematics often end up in strange places. For example, Fermat’s Little Theorem was little more than a curiosity for centuries. Its generalization is the basis of the RSA system, and it can be used in a pseudoprime generator. In fact, PGP uses it for just that very purpose.

The mere fact that a computer makes the result reachable not only means that other proofs can use the 4-color theorem, it means that computers can be used to prove other theorems. Many proofs are broken down into cases. This one just happens to have a lot of them that couldn’t be checked by hand. The computer helps advance mathematical results as well as humanity.

On the other hand, brute force isn’t a novel technique, so the process of getting the result is itself unimportant. The computer did what 1000 mathematicians with infinite cups of coffee could have done.

If you happen to know where I can witness 1000 mathematicians with infinite cups of coffee, please let me know.

Approximating Constants

We will be calculating Euler’s constant, e, by starting off close to the correct value, and then moving closer and closer until we get to the correct value. This is known as approximation.

There are a few different ways to do this. A few are listed below:

1. Find a function that evaluates to e and approximate the function.
2. Rephrase the problem as a root-finding problem and find solutions via famous root-finding methods (any of Householder’s Methods, Bisection, the Secant Method, etc).
3. Rephrase the problem as a fixed-point problem (I.E., find an such that .

I have opted to write about number 1., find an approximating function using Taylor Series. Why did I pick it? The math is easy, and helps teach basics of approximation. Taylor approximation is generally slow, and is often better used to approximate functions over intervals rather than particular constants. This is just a particular application of Taylor series.

If you have not taken a Calculus course, the material presented here may or may not be lost on you.

Taylor Series

Assume that is continuous and infinitely differentiable. If we want to know values of near , we can use the formula below to determine just how many decimal places we need to use in our approximation.

Formula for a Taylor Series with infinite terms:

If is infinitely differentiable and continuous, a Taylor Series of about is written:

This may look a little confusing, but once you start trying a few test functions, it gets really easy to start approximating the function.

Example: about

The zeroth term:

= f(\alpha) = e^\alpha$[/tex]

The first term:

The result:

Error in Taylor Series

If we add up infinitely many terms, we will have an infinite degree polynomial that exactly represents the function. However, I can’t wait for my computer to calculate infinite terms, so we need to know when to stop.

Remainder of a Taylor Polynomial

In general, the error for an nth degree Taylor Polynomial of a function about can, for some be written as follows:

When we are calculating how many terms we need, we need, in part, to find the maximum value of over the area .

Actually Calculating e

We want to (for some twisted reason) compute e using a Taylor series without the calculation being in terms of e. One way to do this is to calculate the Taylor series for

Taylor Series for about :

We should also note that , so we can go on ahead and substitute 1 for x:

Taylor Series for about :

Set equal to zero. Magically, all of the es will disappear, and we will be able to evaluate e without having to use e.

Taylor Series for about 0:

What Degree Polynomial do we Use?

We can play around with our error formula until we determine how many steps will be needed.

Simplifying Error Formula

We see that we need to determine an upper bound on the value of between 0 and 1. An easy upper bound for us is 2.72, as we know that it is larger than e.

Final error formula

The error in our Taylor Series about 0 is:

Naieve C++ Implementation

Please note that this code is susceptible to a few calculation errors, including mismatched-precision addition.

double calc_e(int degree)
{
    double answer = 0;
    double current_factorial = 1;
    for(int i=0; i
    {
        answer+=1./current_factorial;
        current_factorial*=(i+1);
    }
    return answer;
}

Output for various Taylor polynomials:

Taylor poly of degree 1:2
Taylor poly of degree 2:2.5
Taylor poly of degree 3:2.66666667
Taylor poly of degree 4:2.70833333
Taylor poly of degree 5:2.71666667
Taylor poly of degree 6:2.71805556
Taylor poly of degree 7:2.71825397
Taylor poly of degree 8:2.71827877
Taylor poly of degree 9:2.71828153
Taylor poly of degree 10:2.7182818
Taylor poly of degree 11:2.71828183

This gives us precision of less than with 11 steps. Checking the error formula, we see that is less than , but is greater than , so our error is also correct.

The Infamous Code

x2 = number * 0.5F;
y  = number;
i  = * ( long * ) &y;
i  = 0x5f3759df - ( i >> 1 );
y  = * ( float * ) &i;
y  = y * ( threehalfs - ( x2 * y * y ) );
// y  = y * ( threehalfs - ( x2 * y * y ) );

The Function to Model

How Did the Authors Think of This?

Interestingly, this method is nearly identical to one from a mathematical text called “An Introduction to Numerical Analysis“, where there is an application exercise to compute the square root of a function, taking advantage of the storage of floating point numbers.

My Numerical Methods book for this semester contains the full derivation of the method from “An Introduction to Numerical Analysis” that uses linear interpolation for an initial guess to Newton’s Method that gives the accuracy of the function to provably under 4.7E-14 for four iterations. Chris Lomont’s paper goes into much more detail about the method for choosing a suitable constant that the “Quake 3″ authors likely used. Linear interpolation gives a fairly good guess, but it’s possible to take advantage of the way the constant is stored to give us a much better guess. As you can see below, the guess isn’t linear, but actually fits the curve very well without any iterations of the Newton-Rhapson method.

The initial guess is very good. How good? It nearly overlaps the function. The guess is added in red:

I was going to compare the output of the Quake 3 method with the real output, but it was difficult finding a view where there was any very noticeable difference at all, so suffice it to say that it is very close.

Some of the Math

We are trying to find a quick approximation for the function . This can be rearranged as . We want to find the roots of this function for , which are +/- . Note that .

Newton-Raphson Method

Back in the days of Newton, all math had to be calculated by hand. Since it was often impossible to calculate the exact value of many results, approximations were needed.

The Newton-Raphson method is used to quickly approximate function roots. The basic idea is that we start off with a guess that we think is very close to the value of the root. We then take the tangent line at the function f(x). Provided that f(x) is continuous, we follow the tangent line to the X-axis. We then take the derivative at this point and follow the tangent line to the function to the X-axis. Rinse and repeat until you get the precision you need.

It is important to note that this method doesn’t always work: it is not guaranteed to converge, and in fact, you could continue calculating intersections ad infinitum and never get any closer to having the right answer! Therefore, it is important in this instance to try to optimize the initial guess to have as little error as possible.

As with all good methods, this one has an easy-to-remember formula. For a current approximation, , we find by:

For those interested, the derivation can be found here. For the mildly interested, it is derived by taking the first few terms of the Taylor Series of the function.

So to do the Newton-Raphson approximation on a differentiable function, we need one thing:

1. An initial guess of the root. The closer, the better.

Actual Iterative Derivation

One small nit pick I had with Chris Lomont’s paper was that it skipped the actual derivation of the iterative function, so here it is:

When we substitute “” for “” and ““, (Since the function uses the same variable to store the current and next guess), we find the following:

x = x * (1.5 - (x/2) * x * x)

Which looks awfully familiar.

The Algorithm

The algorithm is divided into three stages: precalculation, encryption, and decryption. Precalculation is performed a single time for each person with a public/private key pair, and encryption/decryption is performed for each message.

Precalculation

1. Pick primes: Find two arbitrarily large prime numbers, p and q.

1. Determine the modulus: Multiply p and q to get n. This will be your modulus for all equations.

1. Calculate phi(n): Using Euler’s totient function, calculate . This number is a secret number, so don’t give it away. The strength of the algorithm depends on being hard to calculate when given a sufficiently large n = p * q

1. Determine an encryption exponent: Take any number, e, such that GCD(, e) = 1. This means that they are relatively prime, and share no common factors. This number is considered your public key (when combined with n), and you can give this number to whoever you like.

1. Compute the decryption exponent: Solve the Extended Euclidean Algorithm of GCD(, e) to find . This is your private key.

We have a private decryption key pair: {n, d}, and a public encryption key pair: {n, e}.

Encryption

is the encrypted message, and it can safely be sent publicly.

Decryption

is the decrypted message. The person with the private key for the message will be able to read it, and theoretically, nobody else.

Is the message preserved?

Applying the Corollary to Euler’s Theorem to , we get .

We also notice that

Why is it hard for an attacker to crack?

In a perfect world, the attacker needs to solve some equivalent of the Integer Factorization Problem to factor n, which is suspected to be outside of complexity class P using classical computation. If we have access to quantum computers, we have access to an algorithm, Shor’s Algorithm, to crack integer factorization efficiently, but probabilistically.

The essential problem that the attacker faces is as follows: they have the encryption exponent and the modulus. They know what n is, but they currently have no method of calculating what is without first factoring n. They need to find the decryption exponent, d = ,, but can’t find the exponent without being able to solve the Extended Euclidean Algorithm, where they need to know the value of . So long as n is hard to factor, RSA will remain difficult to break.

We don’t live in a perfect world, and there are plenty of examples of attacks that take advantage of weak implementations of RSA. See RFC 3447 for some best practices.

The Math

Extended Euclidean Algorithm

The Euclidean Algorithm finds the Greatest Common Divisor (GCD) of two integers, a and b. The Extended Euclidean Algorithm finds integers m and n in the following equation:

Euclidean Algorithm

To see how it does this, we will look at an example. Let a = 200, b = 37. Please note that 200 = (101 * 3). Since (200, 37) = 1, this is the equivalent of trying to find 37-1 (mod 200). For previous writings on the material, click here.

1. 
2. 
3. 
4. 

Line 3 is considered the “final” line of the algorithm because it is the last line where the remainder is nonzero. The remainder of that line, 1, is also GCD(200, 37).

Extended Euclidean Algorithm

We start by rewriting every equation so that the remainder is on the RHS.

1. 
2. 
3. 

From here, we start at line 3, and substitute in line 2. Don’t simplify any multiplications, just the additions.

Substitute line 1 into the resulting equation.

[Unparseable or potentially dangerous latex formula. Error 5 : 533x369]

We now have everything that we need for the inverse of 37. It turns out that is equivalent to saying:

so 173 is the multiplicative inverse of 37 (mod 200). Neat, huh?

Fermat’s Little Theorem + Euler’s Theorem

In a letter written in 1640, Fermat (of Pythagorean fame) offhandedly mentioned that he had noticed and proved the following relation:

Fermat’s Little Theorem

For a any integer, and p prime,

As he is occasionally noted for doing, he did not bother to write down the proof. Leibniz is said to have proved it, but didn’t get around to publishing it. He was an inventor of calculus, so we’ll cut him some slack.

Enter Leonhard Euler. Euler was 18 feet tall, shot laser beams from his eyes, and proved Mathematics every second he was awake. In 1736, he took 23 minutes off from a conquest of Mars to prove Fermat’s Little Theorem.

This wasn’t enough, though. He attempted to find a way to generalize it for any number, n, instead of just for primes, p, but couldn’t. The problem tortured him for 24 years, when in 1760* he was finally able to produce his proof.

His proof makes use of a function he defines, phi(x).

Euler’s Totient Function

For any positive integer, n, is equal to the number of positive integers where GCD(a, n) = 1, for a < n.

Most germane to this discussion, for any prime, p, . This makes perfect sense, of course, as a prime is indivisible, and therefore all numbers less than p don’t share factors with p, or else p could be divided!

The formula ends up being:

Euler’s Theorem

For any positive integers a, n:

* Wikipedia says 1736, my number theory book says 1760, tie goes to the book.

Corollary to Euler’s Theorem

Because , we can reduce the multiplication needed for any power of a > n.

Edit: added link to Wikipedia’s article on Integer Factorization, and a better explanation of why the attack is hard

Man only likes to count his troubles, but he does not count his joys.
~Fyodor Dostoevsky

What is Combinatorics?

Combinatorics is the math behind counting. All problems that start with the phrase “How many ways..” are most likely combinatorics problem.

Who is this page for?

• Programmers who have a simple background in Mathematics, or need a quick reference for how to implement combinatorial algorithms.
• Those who did not know what the word “combinatorics” meant.

Factorials

Factorials are used to solve the problem, “How many ways can I arrange n objects?” It makes the following assumptions:

• Each object is distinguishable (Can I tell apart the n apples?)
• You can actually distinguish arrangements of objects (Can I line up n apples in a meaningful way?)
• The elements are non-replaceable (Once I use an apple, I can’t use an apple again).

The factorial function is defined as follows for positive integers:
Factorial(1) = 1
Factorial(n) = n * Factorial(n-1)

or, the naieve C implementation:

  // For 32 bit systems, will not return meaningful results
  // for values larger than 12!
  //
  // For 64-bit systems, will not return meaningful results
  // for values larger than 20!
  unsigned int factorial(unsigned int n)
  {
    unsigned int ret = 1;
 
    for(int i=2; i<=n; ++i)
        ret*=i;
 
    return ret;
  }

The C implementation above is fine for the values it can logically compute. However, when we get into arbitrary precision integer calculations, calculating 1 * 2 * 3 * … * n is inefficient! The number of words produced at the end of each step grows linearly. If we divide-and-conquer, we keep down the number of words-per-integer until the very last few multiplications, which produces significant time savings.

Of course, we would NEVER use this in the real world without extra sanity checking of the input. The importance of sanity checks in the real world is demonstrated every day by celebrities. Don’t commit their same blunders.

  mpz_class factorial(mpz_class begin, mpz_class end)
  {
    if(end==begin)
      return end;
 
    if(end-begin == 1)
      return begin*end;
 
    int half = (begin+end)/2;
    return factorial(begin, half) * factorial(half+1, end);
  }

In fact, running the two against each other to find 100,000! produces the following runtimes on my laptop (I was going to do 1 million and got bored waiting for the iterative solution to finish!):

  g++ factorial.cpp -O2 -lgmp -lgmpxx
  ./a.out
  Recursive factorial: 0.2795 seconds
  Iterative factorial: 3.01006 seconds

The most efficient ways to implement factorial algorithms turns out to be from its prime factorization, but the above is far easier than implementing sieving methods to find primes! Maybe another day

Permutations

Permutations are how we solve the problem, “how many different ways can I arrange k objects from a list of n objects? To give an example, assuming the same number could not be used twice in a combination, your “combination” lock from high school was actually a permutation lock. You were given a single permutation that you could use in order to open it. But how many different possible permutations were there, given 45 as a max value and 3 numbers?

• The first number has no restrictions. You can choose from all 45 numbers
• The second number is restricted: it can’t be the same as the first number. You now have 44 to choose from, giving you 45 * 44 different possible 2-permutations of 45
• The third number is also restricted: You can only choose 43 numbers. 45 * 44 * 43 is the number of 3-permutations of 45.

It’s not hard to convince ourselves that the permutation is all of the numbers between and including 45 and 43 multiplied together. Symbolically, we are talking about all of the numbers between and including n and (n – k + 1), inclusive.

The formula we all learned in school (which you can convince yourself is equivalent to the above) is as follows:

  // Picking and ordering k objects from n objects
  mpz_class permutation(mpz_class n, mpz_class k)
  {
      return factorial(1, n) / factorial(1, k);
  }

Never do this! You have to recompute many of the values that you never use, and it is worthless and computationally wasteful to do so. Instead, you can use the following:

  mpz_class permutation(mpz_class n, mpz_class k)
  {
      if(k == 0)
          return 1;
 
      return factorial(n - k + 1, n);
  }

It’s much simpler. It computes much less. Use it. It is calling your name. Use it.

Combinations

Permutations are very closely related to combinations. In fact, there is only a one word difference in the definition. Combinations are unordered, and permutations are ordered. We can derive the formula by what this implies.

Let’s take our combination lock example. We needed to get three numbers in a row, in order. But what happens if we only need to select the three numbers without concern to the order? We know from the factorial section that the number of ways to order 3 numbers is 3 factorial. So let’s say we have the permutations { (1, 2, 3), (1, 3, 2), (2, 1, 3), (2, 3, 1), (3, 1, 2), (3, 2, 1) }. These are all the same combination, because they all pick the same exact elements! So there is clearly 1/6 the number of combinations as permutations. 6 is 3!, which is no coincidence. In general, the formula for combinations is:

  mpz_class combination(mpz_class n, mpz_class k)
  {
      if(k == 0)
          return 1;
 
      return permutation(n, k)/factorial(1, k);
  }

*Hits the Easy Button*

Next week: More complicated number sequences such as the Binomial Coefficients and the Sterling Numbers

and Loathing in Los Vegas

What will this article focus on?

This particular article looks at geometric sequences (mod n), and how we can use them instead of linear hashes. A geometric sequence is simply a sequence of powers of some number: 1, , , , … So instead of adding the same number together a bunch of times, we’re multiplying it together a bunch of times. And then you subtract one. More on that below!

First, the math

Euler’s Phi Function

When Euler was attempting to generalize Fermat’s Little Theorem, he defined a function using the Greek symbol (pronounced fee by most people I’ve encountered). It has a simple job: it takes in a natural number, n, and returns the number of positive integers less than n that are relatively prime to n. In this article, we are not concerned with ‘s calculation for anything but prime numbers.

It is easy to show that (p) = p-1 when p is prime: all numbers less than a prime are relatively prime to the prime in question, otherwise it wouldn’t be prime! Easy proof.

Euler’s phi function is of vital to the RSA encryption algorithm, and is the cornerstone of the generalization of Fermat’s Little Theorem, but it makes cameo appearances in many other areas of mathematics.

Examples:

(5) = 4, because gcd(5, 1) = gcd(5, 2) = gcd(5, 3), = gcd(5, 4) = 1.

(6) = 2, because gcd(6, 1) = gcd(6, 5) = 1, but gcd(6, 2) = 2, gcd(6, 3) = 3, and gcd(6, 4) = 2.

Order of a number (mod n)

The order of a number (mod n), where n is an integer, is the smallest positive value of x such that . If it is never equal to 1, it is considered infinite. 6 (mod 10) is an example that never has an answer. Note that this still has a solution under Euler’s generalization of Fermat’s Little Theorem. The laws of the universe won’t let you off that easy.

Example:

The order of 2 (mod 7) is 3, because [Unparseable or potentially dangerous latex formula. Error 1 ](prime) = prime-1, so (p) = p-1. If order(m) (mod p) is p-1, that means that m is a generator for all numbers (mod p) except p itself! Since this will not generate p, and 0 by extension, (since they are in the same congruence class), we must subtract our result by 1. So our generator is m, and our hash function is

It is not true that all numbers have a primitive root, but it WAS proved by Legendre that every prime has at least one generator (mod p). Interestingly, according to my college Number Theory textbook, Euler tried his hand at the proof, but was incorrect. To the uninitiated into the Cult of Euler, this would be akin to a team of Michael Jordan clones failing to score a single point in a basketball game against a team of middle school students.

We need to find one such that the first time this happens is for a power of p-1. Instead of testing every power, we can instead (because of this proof), just test powers where the power divides p-1. If we were looking mod 9, and we knew 3^8 == 1(mod p) (which it has to be because of Fermat’s Little Theorem), then , , , and are the only possible powers that can be equal to one. We will call this the generator test. We can check these particular values quickly through successive squaring. If any of the powers of 3 less than 8 are congruent to 1, then we have a failure, and it is not a generator.

If you do not have access to a good way to factor p-1, the following naive method will work well for small numbers. Please note that the preferable way is to factor p-1 and to find all of the divisors of p-1 that way.

// ***********************************************************************
// Precondition: p is a prime. If it is not, it will return 0 indicating
// failure
//
// This assumes that you are trying to do this for a small p, without being
// able to factorize p-1 quickly.
// ************************************************************************
unsigned int find_generator(int p)
{
  int phi_p(p-1);
  std::vector test_powers; 
 
  int i; 
 
  for(i=1; i
<0; --i)
      {
        if(powmod(test, test_powers[i], p) == 1)
	{
	      found = false;
	      break;
	}
      } 
 
      if(found)
      {
	  return test;
      }
   }
}

So what?

If we have an element a (mod n) who has , and is not equal to 1, we have a generator. The generator is for a set of integers of size (p-1), which is even.

Finding generators is nontrivial

A downside to this method is that there is no free lunch when it comes to finding generators. You have to find one, although fortunately for us, most numbers have generators that are less than 10, so you can find them by linearly searching. There are a few strategies of how we can pick primes that will allow us to (relatively) quickly find a generator (mod p). The one I use is:

One strategy is finding a prime, p, such that q = 2*p + 1 is also prime. The only two numbers that you have to check that violate our generator condition are 2 and p, in which case q is a generator. This helps reduce the complexity of the test. How do we know if our numbers are prime? Probabilistic primality testing, of course . It’s amazing how all of this stuff ties together.

A professor I had for a cryptology course said that the odds of the first generator NOT being less than 10 has been shown to be inordinately small, but I can’t for the life of me find any sort of reference to a figure that states that. As there is no trivial way to find a hash function, it is acceptable to search for the first generator (mod p) linearly, using our generator test, if you are looking for just any generator of p. Likewise, you can also find the largest such generator (mod p) by reverse searching.

This is so complicated. Why would I use this over a linear hash?

• The elements selected are not at a fixed interval, so data is usually less likely to cluster, which results in fewer collisions
• It does better at the avalanche test, which says that when a bit of the input changes, at least half of the bits of the output should change. The linear hash fails miserably at this, and geometric hashes (depending on your generator, of course), perform better than their linear counterparts.

Sometime in the future, (not in the next post, though), I will develop benchmarks to see what is better to deal with various different input scenarios. There’s no sense in developing the mathematics if we don’t actually put it all on the line and see if the “better” method works better in the real world. The real world has an amazing way of yelling “surprise!”, but we can limit that surprise through testing, testing, testing.

Number theory is the study of numbers, their properties, and what can be inferred from their properties. For programmers, it is most practical to focus on the theory of positive integers.

Who should use this guide?

• Those who did not know the answer to the above question
• Those who are interested in the math behind hash functions
• Those who found my last article interesting

What will this article focus on?

This article will focus on using the integers (mod n) as indices of a hash table, and the math behind different choices of hash functions. Our goal is to find a “good” hash function (see below). The mathematical explanation will be done irrespective of Group Theory, and I may write another article to look at a hash table as a group over addition or multiplication of the integers (mod n). For a quick refresher of the (mod n) concept, go here, or for another explanation, please look here.

Useful Tools

Greatest Common Divisor (GCD) of positive integers

Explanation:

Mathematically, the greatest common divisor of two numbers a and b is the product of all common divisors of a and b. For a simple explanation as to why, look here.

The Algorithm:

Naive:

unsigned int gcd(unsigned int a, unsigned int b)
{
    int remaind;
 
    if(!a) { return b; } // gcd(a, 0) = a
    if(!b) {return a; }
 
    if(a < b)
    {
        a ^= b;  // Swap a and b in place
        b ^= a;
        a ^= b;
    }
 
    while((remaind = a % b) > 0)
    {
        a = b;
        b = remaind;
    }
 
    return b;
}

Binary: (It’s always worth it to try to find the algorithms that take advantage of working with bits. If life gives you an integer as the sum of powers of two, make lemonade )

Wikipedia has a page that explains a binary algorithm that takes advantage of the binary format of the data. It reduces the problem by stripping out common multiples of two, and then applying the binary analogy of the GCD algorithm. For more details, follow the above link. I haven’t benchmarked it, but it relies heavily on bit operations, so it should run a little faster on modern popular architectures.

Least Common Multiple (LCM) of positive integers

Explanation:

The least common multiple is as it sounds: the smallest multiple that both a and b share. For example:
LCM(15, 20) = 60.

It appears that for each prime, the LCM of a and b includes the largest power from either a or b. In fact, this is true.

Relation between GCD and LCM

For integers a and b:

This is very powerful, and lets us efficiently calculate the LCM of a and b by dividing out the GCD of a * b. Why does this work? If a and b don’t have any prime factors in common, clearly the only way that we can have a multiple of a equal some multiple of b is by multiplying b by a. If a and b only have one prime factor in common (let’s call it d), if you multiply a by b, we get a*b as an answer. However, (a*b)/d is clearly a multiple of both a and b. We don’t need to multiply a by d, since a already HAS d as a factor. d is uncoincidentally the GCD of a and b, and clearly, GCD(a, b) * LCM(a, b) = a * b. An actual proof is left as an exercise to the reader.

What makes a good hash // hash table?

The short answer is that nobody knows. Hashes that work well for some kinds of inputs can produce intractable results for other kinds of input. For our purposes, we will say that a good hash function minimizes the odds of two different inputs ending up in the same congruence class (mod n). When two different inputs DO end up in the same index, this is called a collision, and is as undesirable in hash tables as it is while driving. Also bad is clustering, which is when collisions are much more likely to happen in certain indices than in other indices.

Ideally, we would like the hash function to be able to place elements at any index in the table. This makes it a generator, namely, it can generate any value in the table.

We will try to find a happy medium of all concerns through experimentation. I will define a few different hash functions in the upcoming articles, and then will show how to compare them. That will be where the “Science” part of Computer Science enters the picture

Linear Hashes

Linear hashes take in some number x, and place the object in the index ax + b (mod n). To make the mathematics easier, we will just use ax(mod n), as it should be obvious that adding b produces the set in the same order, but with a different starting point. In order for us to consider a as a hash function, a must be a generator (mod n). How do we know that it does that? Let’s look at a few different values of a (mod 16).

: {2, 4, 6, 8, 10, 12, 14, 0, 2} (mod 16) (doesn’t generate the integers (mod 16))

= 3: {3, 6, 9, 12, 15, 2, 5, 8, 11, 14, 1, 4, 7, 9, 12, 15, 3} (mod 16) (generates the integers (mod 16))

2*3 = 6: {6, 12, 2, 8, 14, 4, 10, 0, 6} (mod 16) (doesn’t generate the integers (mod 16)).

So what works? It works when gcd(a, n) = 1. This is known as being relatively prime or coprime, meaning they don’t share any common prime factors. and obviously don’t share any prime factors, so 3 is a generator using addition (mod 16).

Why does that work? The largest possible multiple of a that will give us 0 (mod n) is n, because a*n == a * 0 == 0 (mod n). We need to make the LCM of a and n equal to a * n, and since we know that LCM(a, n) = a * n / GCD(a, n), it follows that GCD(a, n) = 1.

Since most hash tables you make will have 2^n elements (this seems to be the standard, for addressing reasons), any odd number a will suffice to be a generator for linear hashes.

Theoretically, which hash value should I use?

Linear hashing is obviously a very simple hash function (the simplest one there is, I believe), and therefore, there is not a single hash fucntion that will work for every input set. In fact, this type of hash will have many input sets that will make it have very poor performance. However, if we have advanced knowledge of the kind of data that will be the input, we can stack the deck in our favor.

If your data is guaranteed to have no collisions (mapping unique integers less than the size of the hash table to some value), you can use any positive integer you want as your hash. I recommend 1 for ease of calculation

If your data is sorted ascending, use hash values close to 1. If you can find the mode of the data in advance, you can yourself by setting the hash value larger than the mode. If the mode is large with respect to the size of the hash table or with respect to the size of the data set, you can make the hash value larger than the average number of repetitions for each input.

If your data is sorted descending, you want to do as above, except make your hash value close to n-1. The reasoning can be derived from the above paragraph.

If your data is either purely random, or of several different varieties, your hash function is not always going to work no matter how hard you try. We should avoid hashes close to 1 and n-1, but other than that, we will need to benchmark to see if there is a better value.

Number theory is the study of numbers, their properties, and what can be inferred from their properties. For programmers, it is most practical to focus on the theory of positive integers.

Who should use this guide?

Those who did not know the answer to the above question.

How do we use modulus?

First, we should bridge the gap between a Programmer’s definition of modulus and a Mathematician’s.

Programmer: a % b is the remainder of a / b. Essentially, the programmer uses the following equation:

That is, the programmer says If we were finding 23 % 5, we would have:

% .

Mathematician: Mathematicians rearrange the above equation into the following:

. They write this as:

All this means is that you can move from a to r just by adding and subtracting b.
23 – 5 – 5 – 5 – 5 = 3, so

Efficient Exponentiation (mod n)

Let’s say that you need to find . The naive way of doing this is to perform the operation just as it’s written:

unsigned int powmod(unsigned int a, unsigned int x, unsigned int n)
{
    return pow(a, x) % n;
}

and it has a few disadvantages.

1. It has a very good chance of overflowing native data types
2. It has an algorithmic complexity of O(n) for the size of the exponent. For large integer types, this becomes O(nm), for integers of (on average) mwords

To show a more efficient way of doing it, we will use a method called “successive squaring”. I will explain it by using an example: Find :

We know that . We need to find :

. This is given.

.

This is where the leap of logic occurs. Since , it follows that

So % 5 = 3, and I was able to do it all in my head! For small numbers, this is usually the case. But it should be obvious that this is a lot easier than ordinary exponentiation, with on the order of O(logn) multiplications.

Code example

The best code example I have found is from Bruce Schneier’s “Applied Cryptography”. The C version using native unsigned integers is as follows:

unsigned int powmod(unsigned int base, unsigned int exp, unsigned int mod)
{
    unsigned int toret=1;
 
    while(exp > 0)
    {
        if(exp & 1)
            toret = (toret * base) % mod;
 
        exp >>= 1;
        base=(base*base) % mod;
    }
    return toret;
}

It’ll still overflow for the wrong values, but it is a quick and dirty example. If you have access to an infinite precision integer, it should be trivial to convert it.

Fermat’s Little Theorem

One of the many things that Fermat conjectured (and supposedly proved) is quite useful to the modern programmer. It says, for any prime number p, and for any integer a

.

Combined with the successive squaring method, this provides us a very powerful tool.

Probabilistic Primality Testing

For any number of applications, we need prime numbers. They are the crack-cocaine of modern mathematics. There are many simple ways to get prime numbers, such as the Sieve of Eratosthenes, but these methods fail when your application needs a 20-digit prime. There are newly developed (but complicated) tests that give a definite yes/no on a number in polynomial time, but they require Abstract Algebra, which is beyond the scope of this entry! For most developers, we don’t need to be 100% sure the numbers we are using are prime. We’re not using RSA in life-or-death (or multi-billion dollar banking) situations! All we want to do is tell whether or not an integer is most likely prime so that we can encrypt our Dawson’s Creek fan fiction and hide it from our father.

Fermat’s Little Theorem is always true if we know that the modulus is prime. The proof, however, doesn’t hold true in the opposite direction: if, for some number a, , we can’t say for sure that n is a prime. However, it is very frequently true, and often enough that we can form a probabilistic test, meaning that the numbers are probably prime. Mathematicians are noted for devastating understatement, so when we say “probably”, we mean “the chance is absurdly close to 100%”. According to pgp.net, PGP uses trial division for primes less than 8191, and the Fermat test for 2,3,5, and 7. I can’t find a reliable source covering the mathematics of why, but an unreliable source gives the chance that a composite is picked as less than 1 in . Yikes!

The Test

Ready?

For some number n, it is probably prime if:

If this makes you uncomfortable by using the first two primes, you can randomly pick two numbers (instead of 2 and 3). The test works just the same. For the ultra paranoid, try it three or four times.

The Carmichael Numbers

There are numbers that cause this test to fail for all test values. They are called Carmichael numbers, named after the first person to find an example. The first three are 561, 1105, and 1729. There are infinitely many Carmichael numbers, though they grow more scarce as the integers approach infinity. For fun, use the method of successive squaring to show that 561 is a Carmichael number.