Quick-Links for the TL;DR Crowd

About Lisp: Describes Lisp as an overview.
High Level Overview: Describes in general what is needed to program with Lisp.
All you need

• SBCL
• Emacs
• SLIME
• asdf-install

Code examples: A few rudimentary Lisp snippets.
Other Lisp Features: Features that I did not feel like explaining
Resources

About Lisp

Note: For a complete introduction, I recommend this e-book. If you only want to learn enough to get started, read on!

Interactive

Most languages take an input file and produce an executable or run a script. Lisp is interactive: all input is entered into the REPL, and the computation’s result is printed to the screen.

REPL: Read Evaluate Print Loop. A C++ coder can think of it like this:

void lisp()
{
    while(true){
        cout<<">"; // prompt
        cout<<evaluate(read())<<endl;
    }
}

If you’ve ever used Python’s command line prompt, you’ve used a REPL. A bash prompt is a REPL. So is Lisp.

Prefix

Most expressions in C-style languages are prefix, but exceptions are made for arithmetic, such as 1/(x+3). All expressions in Lisp are prefix expressions: the first argument of an S-expression (Sexp) is a function name, and the rest of the atoms of the S-expression are the arguments for the function. The following are examples of Lisp function calls:

(/ 1 (+ x 3))         ; 1/(x+3)
(format t "~A~%" x)    ; prints the value of 'x' and a newline.

All Expressions Return Values

All expressions in Lisp return values. For instance, the ‘if‘ statement in Lisp not only chooses which execution branch to follow, but returns the result of the executed expression. ‘nil‘ is the default return value of an expression, which also evaluates to false (‘t‘ is true).

In C-style languages, expressions don’t necessarily return values. The expression ‘1+3‘ will return ‘4‘, but ‘cout<<1+3‘ doesn’t return anything (They return a reference to an ostream, so this is a mistake) void functions don’t return anything; they can’t be used as a sub-expression for a larger statement of code.

Functions in Common Lisp can return multiple values. The ‘multiple-value-bind‘ macro is used to bind multiple return values to multiple variables.

High Level Overview: What You Need To Get Started

1) An implementation. All you need is an implementation that conforms to the standards. All of them have an edge case or two, but odds are it won’t affect you. If you absolutely need to switch later (for CLisp’s fast arbitrary precision arithmetic, for instance), nothing is stopping you; most libraries support all popular implementations.

2) An editor. I personally recommend Emacs with the SLIME extension. You can also use Vim and Limp if you either prefer Vim or you can’t stand Emacs. I left Vim before I started using Emacs, so I’ve never used Limp. Caveat emptor!

You don’t absolutely need to use an editor that interacts with Lisp. It’s POSSIBLE to run a Lisp program like a script, but it usually makes as much sense as opening a bag of chips with a hammer. You will want to use the REPL for most of your Lisp tasks, and this will only be bearable from within a good editor.

3) A Lisp tutorial or reference. I highly recommend “ANSI Common Lisp” by Paul Graham [see my review].

4) An installation system, like asdf-install. Installation systems automatically load Lisp projects into your current image, and are essential for working with a large project. asdf-install allows you to download libraries from cliki.net and load them into your Lisp image from within the interactive Lisp command prompt!

Details: What You Need To Get Started

SBCL

About

Steel Bank Common Lisp (SBCL) was forked from Carnagie-Mellon University Common Lisp (CMUCL) in 1999. It is named after Carnagie and Mellon’s respective industries, steel and banking.

Most Lisps can be interpreted or compiled, depending on the context. However, SBCL compiles all Lisp code it receives. Having used SBCL for 6 months or so, there’s not much of a pause even for large Lisp files. On the other hand, the compiler is very verbose, but this can be tweaked.

Why SBCL: Popular

Naturally, I can’t find good numbers on the popularity of SBCL. However, it is noteworthy that asdf-install and LIMP were initially designed to work with SBCL. This seems to be par for the course for all of the libraries I’ve used.

Why SBCL: Active Development

Development of SBCL is still roaring along, with 1,228 commits to their Sourceforge page at the time of writing, including several in the past few weeks. New releases come at the beginning of every month, and this month was no exception: version 1.0.23 was produced December 1, 2008.

Installing SBCL

Installing SBCL: Linux

SBCL can be installed like any other package:

sudo aptitude install sbcl
# Or your local variant.

Once that’s finished, it’s easy to test the installation:

jake@justalaptop:~/code/genesis$ sbcl
(+ 2 2)
*
4

Installing SBCL: Windows

There is an “experimental” Windows installation binary: [link].

I’ve ran it for brief periods and it seems to work fine, so I’m not sure what’s so experimental about it. Again, Caveat emptor!

After installation, run SBCL. You will get a command-line prompt, and type the following:

> (+ 2 2)
*
4

It works!

Emacs

You can use whatever editor that you want to write Lisp. Emacs has good advantages:

ELisp: Emacs is scripted in its own subset of Common Lisp. Called ‘elisp’, anybody who has been working with Lisp for a few months and can type “ELisp manual” into Google can customize Emacs to their taste.

Irony alert: I typed “elisp manula” into Google on my first try.

Automatic indentation: Yeah, I know, every editor in the history of the world does automatic indentation. However, Emacs’ authors primarily write Lisp code, so they paid close attention to how Emacs interacts with Lisp. I rarely see Emacs do the Wrong Thing. When it does, I’m usually writing awful code.

SLIME: The Superior Lisp Interaction Mode for Emacs. In order to effectively code in Lisp, you need an editor mode that will interact with a Lisp implementation. Typing the code directly into a command line is nice for small experiments, but is painful for anything more than 30 lines of code (for me at least.. your pain tolerance may vary).

SLIME

SLIME combines the editing power of Emacs with the interactive nature of Lisp.

If you’ve ever used Python’s command-line prompt, you’re aware that it’s painful to use for any amount of time. The editing capability is limited to that of your console, and when you exit Python, all of your code disappears!

SBCL (and all other Lisp implementations) is no different. If this were the only option for working with Lisp, nobody would. You must work with the REPL from within a real editor. You COULD write code in an external file and load it into SBCL from the command line, but that removes the interactive nature of Lisp.

Enter SLIME (Superir Lisp Interactive Mode for Emacs). SLIME spawns a new Lisp process and acts as the liaison between you and the process.

To start the whole process, just type “M-x slime“, and a new REPL buffer opens. Any Lisp command can be entered into this buffer, and you get all of the editing commands from Emacs. slime-mode begins in all of your open Lisp buffers, which gives you code completion and function argument hints for all compiled functions.

If slime-mode is enabled in a Lisp buffer, you can compile the file with “C-c C-k“, or compile a single function with “C-c C-c” (it looks to me like dependencies are propagated correctly). SLIME knows what capability your Lisp has, so it can take advantage of implementation-specific capabilities like debugging.

Installing SLIME

SLIME is installed the same way as any other Emacs plugin:

• Download and extract.
• Add the load path to your .emacs file
• Add any other code snippets required.

In this case, you’re going to be adding the following code to your .emacs file (open using “M-x M-f ~/.emacs“):

(add-to-list 'load-path "<em>/the/path/to/slime</em>")
(require 'slime)
(add-hook 'lisp-mode-hook (lambda () (slime-mode t)))
(add-hook 'inferior-lisp-mode-hook (lambda () (inferior-slime-mode t)))
(setq inferior-lisp-program "sbcl")

I haven’t tested it on Windows, so I’m not sure if there are any extra gotchas. If you find that you’re running into undefined function errors, one possible workaround is to install Cygwin and add C:\cygwin\bin and c:\cygwin\usr\bin to your %PATH% environment variable. Some people run away in horror at the idea, so if you don’t like it, you’re on your own.

asdf-install

One thing that you should research is asdf-install, a package manager for Lisp. If you find that you need a package that is found on cliki.net, you can download it with one step:

> (require 'asdf-install) ; Not the one step.
("ASDF-INSTALL")
> (asdf-install:install 'postmodern) ; The one step.
NIL

The above installs Postmodern, a simple Postgres interface for Common Lisp. Once you’ve done this, you only need to load the package into your Lisp image. This is also accomplished using asdf-install:

> (asdf:oos 'asdf:load-op :postmodern)
NIL

Using those 3 lines, you can now start to define database connections and operations on the database, without ever leaving Emacs!

Starting to Code

I highly recommend that you find a real reference or a real tutorial and start using that to write code. However, if you’re looking for things to enter into the REPL, I’ll give you some overviews of the basics of Lisp.

Arithmetic

> (+ 2 2)
4

This adds 2 and 2. Notice that Lisp arithmetic (and all Lisp functions) have prefix notation: the function name always comes first. You can nest values like the following:

> (+ (+ 1 1) 1)
3

Printing

> (format t "Hello, ~A~%" "Jacob")
"Hello, Jacob"
nil

‘format‘ is the function name. This is the Lisp-version of ‘printf‘,

The second argument is the destination stream. You can give this ‘t‘ or ‘nil‘ for true or false, or the name of a stream.

When ‘t‘ is given, the value is printed to Standard Output. When ‘nil‘ is given, the value is returned as a string.

The third argument is the formatting string. “~” is the escape character. “~A” means that it takes an argument that follows (like printf), and ~% is a newline.

There are other functions that you can use for printing, like ‘princ‘. I usually stick with ‘format‘, but there are different print semantics for different functions.

Defining functions

> (defun adding-function (arg1 arg2)
    "Adds arg1 and arg2."
    (+ arg1 arg2))
ADDING-FUNCTION
 
> (adding-function 1 2)
3
 
> (+ (adding-function 1 2) 1)
4

Here we see a function definition and two function calls. Let’s examine the function definition:

> (defun adding-function (arg1 arg2)

This can be considered the prototype line. ‘adding-function’ is the name of the function. It takes two mandatory parameters, ‘arg1′ and ‘arg2′, both with no default values.

"Adds arg1 and arg2."

This is the “documentation string.” A string may optionally be the first atom of a function, and it can be accessed from within Lisp using Lisp’s documentation reading abilities. Note that unlike wimpy strings in most languages, Lisp strings can contain newlines with no extra syntactical work.

(+ arg1 arg2))

This is the executed statement within the function body. A Lisp function returns the return value of the last statement. Since this is the only statement, it returns the addition of arg1 and arg2.

Defining variables

Defining a variable for the first time in SBCL:

(defvar my-var) ; my-var is 'nil'.
(defvar my-var 'a-value) ; my-var is 'a-value'.

Notice the quote in 'a-value. The single quotation mark indicates that the following expression is data, not code. Otherwise, it would try to look up the value of the variable a-value, which we may or may not have defined. Either way, it’s not what we want.

If a variable is already defined, you can use ‘setf‘:

> (setf my-var 3)
3

Conditionals

For statements that only execute when something is true, use ‘when‘. For statements that only execute when something is false, use ‘unless‘.

> (when (= 1 1)
    t)
t
> (unless (= 1 1)
    t)
nil

For statements of the form “if (true), do (x), otherwise do (y)“, use ‘if‘.

> (if expression
    x
    y)

If ‘expression‘ doesn’t evaluate to ‘nil‘, then ‘x‘ is executed and its return value is returned. Otherwise, ‘y‘ is executed and its return value is returned.

For statements involving a lot of if, else-if clauses in other languages, use cond:

(cond ((test1) (expression1))
      ((test2) (expression2))
      ; ...
      (t (default-expression)))

Iteration

The two easiest ways to iterate are the macros ‘dotimes‘ and ‘dolist‘.

> (dotimes (i 3)
    (format t "~A" i)
012
nil

‘dotimes‘ is exactly as it sounds: it executes a statement a number of times. You assign a variable (in our case, ‘i‘) and tell it the number of times it shall execute, and the variable takes on the values of all of the integers in the range [0, n).

> (dolist (i '(1 2 3))
    (format t "~A" i))
123
nil

‘dolist‘ iterates through a list much in the same way ‘dotimes‘ does.

Reduce

The idea of ‘reduce‘ is to iterate through a set, applying a function to each value that acts as an accumulator. For instance, we can do this with addition:

> (reduce #'+ '(1 2 3))
6

Lambdas

If you want to define your own functions for ‘reduce‘ without writing formal functions, you can do them as a ‘lambda‘. That’s just a fancy word for ‘function with no name’. It consists of the name, the argument list, and a function body to execute:

> (lambda (x) (+ x 2))

This defines an anonymous function that adds 2 to the input given. The only difference from ‘reduce‘ is that we need to provide two parameters: the accumulated value and the next value in the list.

You’re going to use this. A lot. Formally defining every function is not worth it. A lot of Lisp code ends up much cleaner with functional approaches.

To define our own addition function for ‘reduce‘:

> (reduce (lambda (x y) (+ x y)) '(1 2 3))
6

Other features of Common Lisp

• Macros: Creating code on-the-fly
• CLOS: Common Lisp Object System. Lisp’s answer to Object Oriented programming.
• Packages: This is how Lisp does namespacing.
• More iteration constructs than you know what to do with.
• ‘mapcar‘: Apply a function to each element of a list
• Hash tables
• Arbitrary precision integers and numbers
• cons: “Construct” function. This is the function that builds lists.car: Returns the first element of a list.
• cdr: Returns the tail of a list (everything but the car).
• Arbitrary binary manipulation
• ‘eval‘: Generate code from data at runtime.
• File streams, input streams, output streams, stream redirections, oh my!
• FFI: Foreign function interface. This lets you interface Lisp with C. You’ll probably need to do this if you want to add functionality to Common Lisp. It’s actually surprisingly easy to use.

Resources

Books

ANSI Common Lisp by Paul Graham. A great first Lisp book. It can be used as a textbook.

Paradigms of Artificial Intelligence Programming by Peter Norvig. The best programming book I have ever read. It does have quite a bit on AI, but it also has extensive sections on advanced Lisp programming.

Websites

SBCL: The Lisp implementation I currently use.

SLIME: A Lisp interaction plugin for Emacs.

Limp: A SBCL plugin for Vim.

cliki.net: A general Lisp resource. The home of all projects remotely installable using asdf-install.

Common Lisp Cookbook: An incomplete, yet still helpful, recipe book for Lisp code.

I’ve been programming a lot recently (instead of blogging about programming!). I caught the genetic algorithm bug along with the rest of the internet, so I’m in the middle of writing a Lisp library to make it easier to develop genetic algorithms: Genesis [Github link].

I’ve made some progress in the past few weeks!

The first version of my program was only a genetic programming library, and barely. It produced a rule set that could be evaluated as a function, but it wasn’t anything different than hill-climbing (a phrase I learned from snobby Reddit commenters). It was inefficient, it was unwieldy, but it produced great results when it ran long enough.

I’ve added a population! An arbitrary number of critters evolve in parallel at the moment. Previously, the individual would just mutate. I also added a basic gene-sharing algorithm: random merging.

It’s faster! In my first version, I used two ‘eval‘s per rule per round. Now I use 1, and I will cut it down to 2 per rule evaluation (instead of per round) soon.

It’s in a package! I finally decided to follow my own advice and namespace everything properly. The package layout still needs work, as I’ve been too busy coding to figure out the ASDF system for installing packages.

The package system in Lisp is a lot more flexible than most module systems I’ve ever used, so I’ve been reading a few other popular Lisp packages to see how they are organized before I jump off the deep end without my swimmies.

I started to add tests! Some of the functions are very fundamental in nature, so I’ve started to add solid tests for them.

I segregated the example! There are currently 2 different files: genesis.lisp and square-root-sample.lisp. square-root-sample shows the code necessary to produce a simple/stupid example: finding an algorithm to calculate square roots. As I’ve mentioned before, the results are very good:

All you need to do in order to run this example is load genesis.lisp and square-root-sample.lisp and call the following:

(square-root-sample <em>generations population-size</em>)

After it has finished (run a small number of generations to get a sense for the time it takes), you can call the following to get the best answer so far:

(funcall-best *CURRENT-POPULATION* #'sample-fitness-function 16)

Work Needed

A recent popular example of genetic programming is the production of the Mona Lisa using random polygons. My program should almost be able to handle this. I don’t allow for a “modify rule” function at the moment, only a “new rule” function. It would probably work, but convergence would be even slower than the example given.

I’m also not allowing for one of the most powerful methods of genetic algorithm development: binary serialization. Imagine: if you can represent your whole algorithm as a binary string, then crossover, reproduction, and mutation are all made trivial. I am instead using this idea for lists, which is convenient in Lisp, but I feel it lacks some of the punch.

On The Horizon

Of course, this is still in the toy phase. I still have a pretty substantial to-do list, but here are some of the high-notes:

• Finish writing tests for core functions.
• Multiple populations, including cross-breeding between populations.
• Add multithreading.
• Give the user much greater control over (genetic-algorithm) by allowing extra key arguments.
• Self-awareness: Gather statistics on which alterations produce better improvements, and make beneficial changes more likely to occur.

What are Genetic Algorithms?

Genetic algorithms are one of the best things that computer science has produced. Rather than figuring out a good algorithm yourself, you let your random number generator find one while you sleep! Well, after an up-front design cost to generate proper random functions, and evaluating them, and determining how you’ll evolve them, and how long they should run…

I’ve long been interested in genetic algorithms, and now that I’ve learned my first language with an eval — Lisp — I’m starting to make a library. I spent the past few days prototyping and testing my code,
and I got some encouraging initial results: I got a great almost-linear approximation to my simple test problem (finding square roots) using only addition, subtraction, multiplication, and division (see below).

I know, square roots. What a scorcher. But real people are making real solutions to real problems using genetic algorithms! NASA engineers produced an antenna. They determined acceptable design parameters for the antenna, figured out how to evaluate new designs, and let randomness determine the rest.

The result is described here.

My Genetic Algorithm Plans

I want to produce a Lisp package over the next few weeks (since I should have a boatload of freetime around Christmas) that makes it easy to define new genetic algorithms. I’m going to do it in a few stages, and blog about intermediate results if and when they are interesting.

This will also give internet denizens plenty of opportunity to criticize my awful Lisp code! I am a nublet when it comes to Lisp, and I have no illusions about the quality of the code I’m writing.

This week: prototyping. I picked an easy problem to work out the basics of genetic algorithms. My problem? Finding square roots between 0 and 99, inclusive, with just one rule set.

Disclaimer: This is potentially the stupidest use of genetic programming known to man. Attempting to regularly solve numeric problems with genetic algorithms can lead to blindness, irrational exuberance towards Republican politics, and death.

To give you a sense of the data it ingests, I present the main interface:

(defun genetic-algorithm (generations starting-rules rule-fun
                          fitness-fun)
  "Runs 'generations' number of generations. The rules are initially
  set to to 'starting-rules', the rule generating function is
  'rule-fun', and the evaluation function is 'fitness-fun'"
  (prep-rules starting-rules)
  (dotimes (gen-num generations)
    (run-generation rule-fun fitness-fun)))

Here is the invocation I used for my square root problem:

(genetic-algorithm 32000 '((/ num 2) (+ num .2))
                   #'sample-rule-fun #'sample-fitness-fun)

The typical work for defining genetic algorithms is split into a few stages:

1) Problem definition: What do I even want to solve?
2) Evaluation function: How far am I from right?
3) Picking components: How can I build my algorithms randomly?

Problem Definition

Find the square root of an integer, n, in the range [0, 99].

Evaluation Function

Sum of the squared error of all of the integers in the problem statement.

Interestingly, we can predict the quality and form of the results: since we are minimizing the sum of the squared errors, the result should mimic a least-squared approach. Since I’m only using basic arithmetic, it probably won’t be a very good one (linear at best..).

Prototype Implementation

Warning: There be dragons in the parens ahead.

The rules are stored as a list of lists:

(defvar *RULES* nil)

We also store the possible operators:

(defvar *sample-nodes* #(leaf + leaf - leaf / leaf * leaf))

Woah, what’s with all of the “leaf”s? Well, a balanced binary tree has about as many nodes as leafs (I’m generalizing…), so I started at leaf/operator parity and moved up until I stopped getting stack overflows when generating new functions, which happened at one .

How do the leafs get translated into values? First, they are picked from an array of random values:

(defvar *sample-values*
    #(-1 1 num num randomnum randomfrac))
 
(defun sample-generate-leaf (list)
  "Picks a random element from 'list' and translates it into the proper atom
  or list."
  (let ((elt (random-array-element list)))
    (case elt
         (randomnum (random 1024))
         (randomfrac
          (let ((val (ignore-errors (/ (random 1024) (random 1024)))))
            (when (null val)
              (setf val (/ 1 2)))
            val))
         (t elt))))

These are pretty self descriptive (num is the input number, and it is included twice because I got better results ). ‘randomnum‘ is a random integer in [0, 1023], and ‘randomfrac‘ is a fraction whose
coefficients are in [0, 1023].

So how is all of this put together?

(defun sample-rule-fun ()
  "Example random rule generator. Generates a random arithmetical
  expression."
  (let ((elt (random-array-element *sample-nodes*)))
    (case elt
      (leaf (sample-generate-leaf *sample-values*))
      (t
       (append (list elt) (at-least-once #'sample-rule-fun))))))
 
(defun sample-generate-leaf (arr)
  "Picks a random element from 'arr' and translates it into the proper atom
  or list."
  (let ((elt (random-array-element arr)))
    (case elt
    (randomnum (random 1024))
    (randomfrac
     (let ((val (ignore-errors (/ (random 1024) (random 1024)))))
       (when (null val)
         (setf val (/ 1 2))) ;; avoid divide by zero
       val))
    (t elt))))

This is also fairly straightforward: if you pick an operator, generate a new list, if you pick a leaf, generate a new value.

Results

Square Root Function Test

I ran it for about 18 hours (overnight and after work), and when I came back, the results were nothing short of amazing:

I included a range of values outside of the initial zone because the results are better than I could have hoped for.

I saved the data, and then was trying to save the rules (there were 18 in all), and then I lost them from my first Emacs freeze. C’est la vie. However, my initial guess that the data would fit as a least-squares appears to be a better guess than I ever could have hoped. I want to add arbitrary polynomials, and I hope to see a higher-order best-squares fit of the data plot, with much less error.

Implementation Time

The first implementation was done in a 4-hour chunk of one evening (during which I played X-Moto for at least an hour…), which I find impressive. I probably would have spent at least 3 times that on the prototype in C++. This problem is made extremely because of several of Lisp’s features:

eval: My method is based on runtime list generation, and eval-ing it as a lambda.

Insanely simple Lisp syntax: The same syntax that defines lists also defines their programs? This reduces random function generation to a random list generation problem, which is almost trivial.

REPL: Lisp is a REPL (read-eval-print loop) language, meaning that it has an interactive prompt. Combining this with Emacs’ multiple buffer capabilities, and it’s super easy to make changes and reload.

TODO

It’s never finished!

Rules

Allow more than one rule set, define the maximum number of rules per set, define rules as arrays for better lookup, etc.

Better use of eval

I eval the rules every time I need them, as opposed to the once I should generate them. I will probably see a significant speedup when I do this.

Define a package

Namespacing everything has obvious benefits.

Cross-breed rules

Allow true genetic properties by copying subsets of rules into other rules.

Better rule trimming

My current program produces some stupid rules: such as f(x)=x. In fact, my “amazing results” had 4 rules that evaluated a number to itself out of a total if 18 rules.

Allowing the user to define rule inputs.

Putting the package in an easy-to-use location

Stay tuned for more!

Lisp is influencing the way I write C++ in a few different ways.

To Generalize…

Imagine my horror: Lisp caused me to realize that I’ve been refactoring the wrong way! For a few years!

Well, that is not entirely accurate. My code did the same thing after I was done. I broke code down into small logical units. Unless I was careless, the code still worked. My functions were more readable, and one could usually see what they did at a glance.

I asked myself questions along the lines of, “What can I do to break this up logically?”, and therein lies the problem:

Weak questions have weak answers.

Powerful questions are much better suited to refactoring!

• “Can I use an existing solution to solve this?”
• “Could this solve a completely unrelated problem?”
• “Can I rearrange things elsewhere so that this code isn’t even needed?”

Sometimes, the answer is no. Some boilerplate code (like exception catching) is an overhead that can not be avoided, can not usually be generalized, and sticks out like a sore thumb in C++ style languages.

Why did Lisp fundamentally shape my refactoring practices? Paul Graham and Peter Norvig are both to blame for this, and for different reasons.

First up, Graham. I taught myself from “ANSI Common Lisp“. At first, I attacked the problems in the book from a C++ perspective, trying to wedge loops and functions into my solutions. Doing the Lisp examples forced me to think simpler. Within a month I started being able to think of functional solutions instead of C++ solutions (reaching for reduce and lambdas instead of loops and classes).

I worked problems out of this book every night for a month and a half, and it still sits next to me as a quick reference. After 8 years of C++ programming, it took a while for things like “reduce” to become the most obvious solution to a problem.

I haven’t reached that magical enlightening moment that Lisp causes. I don’t have the “ooh, shiny!” awe over macros that seasoned Lisp programmers seem to have, so I’m still hacking away.

Norvig has sent me towards enlightenment with Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp. In these pages, I found something that was missing from Graham: elegance

Some of the solutions that I see in this book give me a sense of awe and enlightenment that I haven’t had since I took Abstract Algebra. He presents them in a very straightforward and honest style that can only come with experience.

This book has given me a whole new level of quality to try to reach. I hope one day to be able to do this.

Tools

A year ago, I would have described my programming process like this:

1. Design
2. Code + document
3. Neaten + prune
4. Return to 1, as needed

Not bad. However, working with Lisp and seeing that there are different tools in the world has led me to fundamentally restructure how I program. I would now describe my process like this:

1. Design
2. Code + document
3. “Would a tool/function/class have made this task easier?”
4. Return to 1, as needed.

For instance, I’m writing some image processing libraries in CL right now, and I’ve written a few tools that I might never have thought of generalizing from a pure-C++ mindset! After all, iterating over an image is really easy in C++, so why bother making it simpler?

However, map-image is invaluable: apply a function at each pixel of an image, and return a new image. Combine this with the lambda construct, and you have a level of flexibility possible only with some of the goofier Boost libraries.

In Short…

While working through the Graham examples, I found that I was playing Code Golf against myself.

I’ve started to see this carry over to my coding at work as well. In the past, I might have done something like the following:

int num_times = SomeFunction();
 
return AnotherFunction(num_times);

This is a reasonable approach. The local variable name reflects the role of the function, and adds a little bit of clarity for the human reader. Using this style is fine if these are your priorities.

However, I’ve started giving functions better names (when I can), and cutting out the middleman:

return AnotherFunction(NumberOfTimes());

This is effectively the same code, and there’s a good chance that it would compile down to the same assembly language. It’s just as readable. The difference is the line count. It’s a difference small enough for a bikeshed issue holy war, but over the course of small projects (all I’ve done so far in Lisp), being able to fit more code on my screen is a big win for me (all else equal).

I am starting to prefer reading and writing compressed code as a result of trying to solve Graham’s examples in as few lines as possible.

Future Directions

I’m still looking for my “Lisp Enlightenment” with respect to macros, which leads me to believe that there’s something significant that I’m still not getting about Common Lisp. I’m not sure where all of these new techniques are going to lead me, but in the meantime, it is starting to fundamentally change how I approach programming problems.

Fortunately for me, the internet is best used to tell people they are wrong. I’ll show you my code, and you tell me when I’m doing it wrong. Deal?

For the complete source code listing, click here. I don’t have any public version control set up to date, so it’s just a static file.

If you are a Lisp power user, this will be horribly boring. I’m just looking to see where I make mistakes.

Disclosure: I did not write unit tests for this project, and I’m aware that this constitutes “doing it wrong.” I recently discovered “lisp-unit“, and will be rectifying the situation.

I also make no effort to validate or encode user input strings.

XHTML Syntax

The “XHTML” referred to will be considered Transitional XHTML, since tags like <center> are not explicitly disallowed.

Basically, I wanted the program to convert something like this:

'(html
 (head
  (title "Greetings, Terran orb!"))
 (body
  (img "src" "./img/some_image.jp2")
  (p (** "margin" "1em") "Some paragraph")))

into this:

<html>
 <head>
  <title>
   "Greetings, Terran orb!"
  </title>
 </head>
 <body>
  <img src="./img/some_image.jp2"/>
  <p margin="1em">
   Some paragraph
  </p>
 </body>
</html>

There will be a separate function to generate the DOCTYPE declaration.

The biggest eyesore here is the (**) tags used for attributes. However, I don’t use very many attributes in my coding, so it’s easier for me to specify when I have an attribute than to use the following syntax:

(tag attributes inner-stuff)

and have my code scattered with more nil than you can shake a stick at.

A Summary of my Development

Based on memory and SVN commit messages.

1. I’m going to print some tags!

2. OK, I screwed up the first time, but I’m really going to print some tags this time!

3. Attributes are now supported.

4. Singleton elements are now supported.

5. Holy hell, “FORMAT” is complicated! All I want to do is indent! I am also adding code to specify the output stream.

6. I have realized that it is simpler to redirect *standard-output* than it is to specify the stream.

7. DOCTYPEs are supported.

8. It turns out that XHTML tags are mandated to be lowercase. Who knew?!

Walking Through My Code

Package definitions

(defpackage "XHTML"
  (:use "COMMON-LISP")
  (:export "GENERATE")
  (:export "DOCTYPE")
  (:export *singletons*))
 
(in-package XHTML)
 
(defparameter *singletons* '("IMG" "META" "LINK" "HR" "BR" "INPUT"))

The only interesting thing here are the exports. “GENERATE” is the function that will be generating the XHTML. “DOCTYPE” allows you to generate the DOCTYPE with an optional argument to specify the location of the DTD.

*singletons* is a list of singleton elements (Elements like <img /> or <meta />), and is available to the user as a helper. It will be more useful if/when I alter this to be an XML generator.

Function: GENERATE

(defun generate (lst &optional (depth 0))
  "Recursively outputs a list of pseudo-xhtml as real xhtml to *standard-output*. Does not print doctype."
  (cond ((null lst) nil)
        ((stringp lst) (format t "~VT~A" depth lst))
        ((atom lst) (format t "~A " lst))
        ((singleton? lst) (print-singleton lst (+ depth 1)))
        ((listp lst) (print-tag (car lst) (cdr lst) (+ depth 1)))))

Calling (xhtml-generate my-list) will print the desired output to *standard-output*. The program recursively partitions the work, but all tag calls inevitably go through this function. Atoms are included so that the user is not punished if they slip and forget to quote their sentences (unless they consider caps-lock a punishment).

(p This is a paragraph!)

Will be output as

<p>THIS IS A PARAGRAPH!</p>.

They will still get an admonishment (from SBCL, anyways) if there are commas in the text, but this is a happy medium for compromise.

Function: DOCTYPE

(defun doctype (&optional (dtd-loc "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"))
  (format t "" dtd-loc))

This is included separately so the user can use a different DOCTYPE declaration.

Function: PRINT-TAG

(defun print-tag (name lst &optional (depth 0))
  "Prints a normal html tag. Parameters are the name of the element and a list of elements that are children of this tag. If the first element of the parameter `lst` is a list whose first element is the atom **, then this list is treated as an attribute list."
  (format t "~&~VT<~(~A~)" depth name)
  (let ((cur-lst lst)
        (head (car lst)))
    (cond ((null lst) nil)
          ((and (listp head)
                (equal (car head) '**))
           (progn
             (print-atts (cdr head))
             (setf cur-lst (cdr lst)))))
    (format t ">~&")
    (mapcar #'(lambda(x) (generate x (+ depth 1))) cur-lst)
    (format t "~&amp;~VT<!--~(~A~)-->~&amp;" depth name)))

Most of my time writing and rewriting this function was spent in the “FORMAT” docs, as it is far more complicated than ANSI Common Lisp ever suggested. Not only can it do any type of formatting under the sun, but it can also butter your toast and summon dead relatives.

The logic of this function is almost identical to the simpler “PRINT-SINGLETON” below. The added ugly checks to see if we have attributes to print. The first line and the last 3 lines are all that is really important with respect to the logic of the function.

Function: SINGLETON?

(defun singleton? (inpt)
  "Checks to see if 'inpt' is a singleton (e.g. <img alt="" />)"
  (cond ((null inpt) nil)
        ((not (listp inpt)) nil)
        (t (member (symbol-name (car inpt))
                   *singletons*
                   :test #'equal))))

Function: PRINT-SINGLETON

(defun print-singleton (lst &optional (depth 0))
  "Handles singleton (<img alt="" />, etc.) statements. Parameter: A list whose car
is the singleton in question, and any remaining elements are considered
attributes. Optional: The indentation depth."
  (format t "~&amp;~VT&lt;~(~A~)" depth (car lst))
  (print-atts (cdr lst))
  (format t "/&gt;~&amp;"))

This is the simpler version of “PRINT-TAG” written above. This function is a lot simpler because any extra elements in the list are assumed to be attributes. We don’t have the possibility of attributes that are mixed with tags.

Function: PRINT-ATTS

(defun print-atts (lst)
  "Prints attribute lists. There must be an even number of parameters,
the odd parameters being the names and the even parameters being attributes."
  (if (null lst)
      nil
      (progn
        (format t " ~A=\"~A\"" (car lst) (nth 1 lst))
        (print-atts (cdr (cdr lst))))))

Pretty boring, as things go.

Future Directions

Generating Related Pages

I would like to generate pages that are related. They use the same CSS, refer to the same image directories, and perhaps have the same navigation elements.

Database connections

Does anybody have any suggestions for good database interfaces [such as CLSQL]?

Running as CGI under Apache

The last thing that I will need in order to make simple client-server web applications.

A week or two ago, he threw me a curveball:

“How’s it going? Do you understand it all yet?”

“No. I’m making progress, though. I’m probably not going to have this down for a while. It’s a completely different way of thinking from the programming I normally do.”

“How is it different?”

I froze. He’s “Mr.-Fix-It,” and understands the rock quarry/asphault business inside and out. He’s very analytical, and it’s almost impossible to stump him with my favorite “Devil’s advocate” questions. However, aside from what I’ve told him, he just doesn’t know much about programming.

“Metaprogramming” and “first-class functions” wouldn’t have meant anything. “It lets you write programs faster” is a cop-out; he’s given me detailed explanations of his business. I wanted him to understand.

I was able to stammer out the fact that the language encourages breaking problems into smaller versions of the same problem (recursion), but that other programming languages also supported it.

I eventually told him that I hadn’t even thought about explaining the difference to a non-programmer.

“Well, if you’re ever able to come up with it, let me know.”

In college, I was a Computer Science tutor. I could explain most CS concepts that students saw in early C++ and Java courses to the layman on the street. However, I had nothing for explaining the difference between Lisp and C++ to him, which was a little scary. I spent some time dreaming up examples.

Functions

All models are wrong, but some are useful.

~George Box

Abstract math notions are boring to most people, so I try to relate programming concepts to real-world items.

A fast-food restaurant is a great example of a function: An order and some money is input, and you get cheeseburgers, fries, a soda, and your change. Whether a person likes math or not, they know what a function is: you put something in, you get something out.

The idea is easily extensible to Object Oriented programming: all Burger Kings have orders, and you can place a specific order at an instance of a Burger King. A Burger King “is-a” fast food restaraunt.

I will use this metaphor to explain why Lisp is different from the programming I usually do (C++/Java/C#/Python).

Lambdas

Any restaraunt probably has a few different ways to order food: over the phone, at a drive-through window, at a counter, through a waiter, etc. However, there might be unofficial ways to order food: Bill Gates might write a letter requesting a buffet with a few hundreds slipped in as proof. Someone might tell the owner on his way to work that he’d like him to come back with a pizza.

The owner can easily define new ways to order food through a restaraunt.

Lisp, through “lambdas”, lets a programmer define new functions as easily as a restaraunt owner can devise new ways to order. The inputs and outputs are the same, but making new functions can have very low overhead compared to other programming languages.

Brevity

“Wiz Wit”.

This is my usual order at Pat’s in Philly. With just two words, I get a cheesesteak with Cheez-Wiz and fried onions in less than 5 seconds. The system is designed to maximize the number of cheesesteaks output per hour: orders are short, and you usually get your cheesesteak before you’re done paying [and sometimes, before you're done speaking!]

This lesson can be applied to writing: It’s easier to convey a short thought. Unless you are Charles Dickens or Neal Stephenson, economy of language should be a goal.

Lisp’s syntax is very simple and short, so you can usually convey a thought in Lisp much quicker than you can convey a thought in most other programming languages.

Macros

An order can come in several different forms: it can be written, it can be gestured, it can be be spoken, and it can be implicit.

A lot of programming languages let you generalize the details of what an order can receive. Most orders are the same, so you can handle the contents the same even though the orders are different. In C++, you could even use “templates” to write code that generates code that handles this idea.

Lisp takes this idea all the way to its logical conclusion through “macros.” You can easily write functions that define orders. The difference is that you can easily do this at any scope!

If you notice that all fast food restaraunts are the same except for color scheme, specials, and slogans, it is easy to write functions that define new fast food chains. If you notice that your functions for defining fast food chains are similar to your functions for defining hardware chains, you can write functions that generate these producers.

Lisp lets you easily do this manufacturing on a level that is unimaginable with any other language. It may be possible, but nothing beats Lisp’s ease-of-use.

Self-Modifying Code

The metaphor breaks down here, and I won’t force it. However, the idea is simple when it is explained.

Most programming languages are designed to modify memory. They all store their information in memory, retrieve it, modify it, and write it to memory again.

Some programming languages try to hide the fact that they’re based on a memory-modification model, but they are extremely similar to languages that don’t hide it.

Lisp, however, is written to modify lists. Lisp is also written using lists, so a Lisp program can be programmed to modify itself.

Conclusion

This is just the tip of the iceberg! There are a lot more ideas that lay beneath the surface here, and you can bet that I’ll think about how to explain them to an educated adult. However, this should get someone with passing familiarity to programming to understand what makes Lisp stand out: ease-of-use and brevity.

The Nature of Lisp

First up is “The Nature of Lisp” by Slava Akhmechet (passed along by my friend Steve). His essay is set up as an argument to use Lisp for the beginner. I found it pretty effective, as I decided to learn Lisp instead of Ruby this summer. That’s right, Ruby, you’ll just have to wait.

The essay looks at the design of Ant, Java’s build tool. Ant is an XML-driven build system for the Java system that lets you specify how your Java applications are built. In essence, Slava made the following points:

• Ant is slowly marching (one by one) towards Turing Completeness.
• The commands in Ant are being executed by something. It just happens to be Java.
• Any XML is trivially convertible to Lisp.
• Lisp has all of the same benefits as XML, but is a program and has extra benefits, including automatic data loading and metaprogramming facilities.
• Lisp can be extended to other domain specific languages, and to know how to modify data within the language, you only have to learn a single language: Lisp.

Developing the Ant language as an extension of Lisp is a nice idea, because Lisp solved all problems 154 years ago, back when cavemen wore bowties and were still allowed to ride dirigibles to work. As a result of decades of tweaking, Lisp is stable, fast, and elegant.

Storing your build scripts as Lisp is, in some respects, a very good solution to the problem. Since you are storing the data in what amounts to a Lisp data structure, tree commands can be executed on the data and transform it easily. This gives you the power of XSLT without any of the ugliness. Not only that, but since your data is already a program, the program can load itself into memory with a minimum of hassle. No more complicated parsing!

The Emacs Problem

“The Emacs Problem” by Steve Yegge looks at the problem from the perspective of log parsing. Logs can be written as complicated text strings, or they can be written as XML. Both have their advantages, but in the end, having a standardized way to represent your data as a tree saves you from the trouble of modifying your complex string into a tree in order to do complex manipulations. If your data is in an easy-to-use tree, you can use tree traversing/modifying commands, which gives you more power.

Since your log can easily be written as XML, your log can easily be written as Lisp. Since your log is Lisp, it loads itself into memory and you don’t need to spend any extra time parsing it. Since your data is a program, it can know how to modify itself.

Neat, huh?

Steve takes the idea a step further in his section “Beyond Logs”.

So Web pages have CSS, and JavaScript, and all this other hooey. It’s become so ugly that people don’t really write web pages anymore, not for production stuff. Nowadays people treat the morass of ancient, crufty Web technologies as a sort of assembly language. You write code to assemble your pages piecewise using PHP or XML/XSLT or Perl/Mason or Java/JSP or perhaps all of them in a giant ugly pipeline, which “compiles” down to an unreadable Web page format. Talk about fun!

The solution?

The whole nasty “configuration” problem becomes incredibly more convenient in the Lisp world. No more stanza files, apache-config, .properties files, XML configuration files, Makefiles — all those lame, crappy, half-language creatures that you wish were executable, or at least loaded directly into your program without specialized processing. I know, I know — everyone raves about the power of separating your code and your data. That’s because they’re using languages that simply can’t do a good job of representing data as code. But it’s what you really want, or all the creepy half-languages wouldn’t all evolve towards being Turing-complete, would they?

In fact, if you insist on code/data separation and you’re an advocate of OOP, then you’re talking out of both sides of your mouth. If your gut reaction to having log entries know how to transform or process themselves is “woah, that’s just wrong“, think again: you’re imposing a world-view on the problem that’s not consistent with your notions of data encapsulation and active objects. This world-view dates back to ancient Unix and pre-Unix days. But if you think about it, there’s no reason log entries or config files shouldn’t be executable and subclassable. It might be better.

And what about, oh, web pages? Or word-processor documents? Well, you figure it out.

Your webpage is a Lisp program. Your logfile is a Lisp program. All of your code is a Lisp program. Everything is a Lisp program. Hooray! Steve Yegge saved the internet!

Security in the Small

Everything we’ve discussed here is fine and dandy when you have your Lisp program read and execute Lisp data on your own Lisp machine. You can easily load log files, you can easily write log files, and you can easily manipulate log files. I want to look at another example, far more common than log files: web sites.

Let’s pretend the real world has two people, you and me. I ask you for your personal webpage, and you give me an executable file. It could be a webpage, but it could also be a horrible icky program that changes the contents of all of my Open Office documents to a copy of “Never Gonna Give You Up” by Rick Astley. I don’t trust you, reader.

OK, fine, realistically, we can’t let our web pages just do anything they feel like. After all, web pages come with Javascript, and Javascript can’t just wipe your hard drive the first time someone successfully cross-site scripts CNN. Nobody in their right mind would write a web browser that would just take an executable file that it’s been given and run it without even parsing it for harmful instructions.

… right?

“Of course you shouldn’t just execute a binary someone else gives you!” I hear you shouting. Who in their right mind would do that, right?

OK, so we know that web browsers should be written with a sandbox that prevents the evils of the world from doing whatever they want with your personal computer. But web browsers aren’t the only computer programs that use data. Let’s take Steve Yegge’s log file example. Let’s say that I write a program that uses log files. Since the data is code and knows how to modify itself, I can run it and have it entered into memory. That’s fine.

Let’s say that my program runs your log file. I need to keep it in the same kind of sandbox that web browsers use for very similar reasons. However, there is a new problem here. The set of coders writing “data is code”-style programs isn’t restricted to people with famous names.

Do you remember that guy from college that failed the intro course three times and still got a degree? He’ll be writing programs that execute data without checking it. Do you remember that guy in middle school who was willing to put anything in his mouth for a nickel? He’s now a highly-paid consultant for your company, and he’ll be writing programs that execute data without checking it.

OK, so this is a problem that is easily modularized and distributed. So is parsing, and many who don’t use automated parsing tools (and many who do) screw it up.

Errors in the Large

OK, so we now have a magical browser that won’t Rick-roll my whole hard drive when it requests a bad page. Now we can start sending our executable web pages to each other!

Hm, so you sent me a page that has an error on it. My browser has a few options.

1. It can try to guess what you meant and display the page accordingly.
2. It can refuse to show me the page at all and give me a compilation error.
3. It can let me fix what is wrong with the file.
4. It can remove the error and show me what’s left of the page.

Really, all of the same options that we are left with on current HTML web browsers. Javascript for the web has to warp in order to support the fact that Internet Explorer 6 supports a different flavor of Javascript than everyone else does. This could potentially do things to new executable formats –Lisp included– that you don’t want to see happen in your lifetime.

Imagine that we end up with the IE6 of Lisp browsers. Not only does this browser fail to correctly display correct Lisp, but it also has an extensive quirks mode that is very forgiving: it lets you also use C-style curly brackets for alternating parentheses groups, for instance. You know, just so the programmer can easily see where they end. This is also the web browser that everyone has by default, so it is the one that the unwashed masses are using in order to sink their teeth into Lisp.

The attack could come in any number of flavors, including “embrace and extend” and “ignore the agreed-upon standard”. There even may be types of ignorance that I didn’t think of. I’d bet on it. So we now have what amounts to a defective, entrenched Lisp interpreter. Not what the crowd is looking for.

Conclusion

Storing data as code is a powerful technique. Lisp, for example, has done this successfully since its inception, and it helps give the language some of its power.

Unfortunately, technique is imperfect when brought into the real world. Security is a huge concern, as — no offense– I don’t trust you. We can’t just treat data as Lisp programs and expect that people won’t try to take advantage of the fact that the files will be executed.

Not only that, but if this idea hits the big time for something like a web format, the people who design it have to be very careful about its release. We don’t want another Martian Headsets situation.