12 Steps to Build and Deploy Common Lisp in the Cloud (and Comparing Rails)

Ruby on Rails is becoming a respectable standard for building web applications, for all the good reasons: It is a powerful and fun language, it has a great community and it’s easy to deploy to Heroku. While I was taking SaaS-class, I missed the parentheses and as always, I got curious: how would Common Lisp compare?

A whirlwind course of Ruby on Rails and best-practises of Engineering Long-Lasting Software is what last month’s SaaS-class was all about. Taught by Armando Fox and David Patterson from UC Berkeley (thank you so much, you guys rock!), it was the best online course I completed. Other online classes I completed were Peter Norvig and Sebastian Thrun’s excellent AI-class and Hal Abelson and Gerald Jay Sussman’s canonical SICP class. The SaaS-class will run again this May 18th, make sure you don’t miss out!

The core of the SaaS-class is to build a movie database web application and deploy it in the cloud. This article describes how I built a lite version of the same app using Common Lisp. Both are hosted on Heroku, with the Rails version hosted here (source) and the Common Lisp version here (source).

Note that the Common Lisp version has less features implemented; most notably it is missing filtering by rating, sorting by title or date, a confirmation check when deleting and the fancy flash messages. These are left as an exercise to the reader. Fork here and start playing around!

Before going into the how-to below, here is how I experienced building the app in Rails vs Common Lisp:

• Rails is missing a true REPL; the irb shell is too clunky and not a real REPL. What this means is that the test-debug cycle in Lisp is shorter and more engaging. In Rails, you would change the source code, hit save and refresh the browser to see the results. With Lisp you can — while you run your server — evaluate code on this running image and directly see the results in the REPL or browser. You can also do stuff like retrieve a database entry as a CLOS object and inspect or manipulate it, which I do very often for quick testing of new functions.
• In Rails, convention over configuration is an oft-repeated term. I did not come to appreciate this immediately, since it requires a very steep memorization curve of conventions in order to avoid introducing subtle — hard to get rid of — bugs, such as wrong file/folder/variable names. It also leaves little flexibility in approach. Such well-developed framework is missing in Common Lisp, so there isn’t much convention you need to be aware of; which has a flip-side that much of the code that was abstracted away by means of convention, we will have to write ourselves — wishing we had conventions.
• Lisp lacks a true community, the kind that powerfully drives Rails forward, resulting in great libraries, superb documentation and tutorials aplenty. It is much easier to “get going” with Rails thanks to this, leaving the Lisp community wishing they got the attention they deserve.
• One of the big advantages of using Rails, is that it takes away a lot of little sources of headache that comes with web-app development. Needless to sum them up here, but here are “Four Ways Ruby on Rails Can Help You.”
• Rails is probably the best way to go about a complex web application. However, when it comes to regular programming tasks (which may be part of your core logic behind your web-app), I would still resort to Common Lisp over Ruby. Perhaps to get the best of both worlds, I should investigate calling Lisp code from Rails? Tips are more than welcome!

Now, let’s get our hands dirty

The below describes the steps I took to write a simple database backed web application, served with Hunchentoot, HTML generated with CL-WHO and Postmodern as an interface to PostgreSQL and deployed on Heroku using heroku-buildpack-cl.

You will build a simple movie database called “Rotten Potatoes,” implementing the basic CRUD (Create, Read, Update, Delete) facilities using a MVC approach, applying RESTful URIs and deploy it onto the cloud for whole world to see.

When using Rails, you hardly need to think about these concepts, since the framework is designed to force you to think along these terms; which is a good thing, since these are all great concepts. Rails’ great conventions over configuration are not implemented in any Common Lisp framework that I am aware off, so I will show you how to achieve similar high-level style by writing your own mini-framework.

One more tip before we start: use git commit frequently! This is generally a good practice and at stages when you want to test your app in the cloud, you will need to commit your changes before you can deploy it.

1. Set up your environment

To correctly set up your development environment is probably the highest threshold that prevents one from trying this out. Don’t be disheartened if you don’t succeed immediately, if you can scale step 1, the remaining 11 steps are going to be fun.

Follow the instructions described here to get your Heroku set up correctly. If you are reluctant to click on that link, here’s a short summary.

Get yourself a Heroku account and install the Heroku toolbelt. Then, in your project folder (assuming you have initialized Git for it) run:

$ heroku create -s cedar --buildpack http://github.com/jsmpereira/heroku-buildpack-cl.git

This will tell Heroku to use the Common Lisp buildpack. Additionally, you will need to install a plugin for Heroku to enable config variables during build time. Replace ‘myapp’ below with the name of your app, which if you haven’t changed, defaults to a poetic one the likes of ‘lit-water-9843’ or ‘shielded-stone-0034’.

$ heroku labs:enable user_env_compile -a myapp

Now we can declare that we use SBCL (CCL also supported) and Hunchentoot (Aserve also supported):

$ heroku config:add CL_WEBSERVER=hunchentoot

Also, to avoid trouble with SBCL source encoding use:

$ heroku config:add CL_IMPL=sbcl
$ heroku config:add LANG=en_US.UTF-8

As easy as it sounds, it took me some effort to get going. If you encounter frustration during this step — to the point that you are willing to give up on Heroku — you may proceed the tutorial as a generic web-app building exercise. Either that or ask in the comment section and see if we can help each other out.

Next step is to create a local test environment — deploying in the cloud every time we want to test something would be ridiculous. We will do it once in a while to test a new feature, because it takes about a minute for each push onto Heroku.

As you can see on this project’s github, I have chosen to put static files in a folder called static/ and all the source code of the application in src/. In this folder, I tried to organize things in an MVC way: the code to generate dynamic HTML is in the views folder, controller.lisp holds the dispatch table and handlers that do not require rendering a view, and finally model.lisp attempts to mirror the elegance of Rails’ Active Records. Note that you are free in your organization — we are not cast in a framework after all!

All supporting utility code is put in the util folder. Note that heroku-setup.lisp is necessary to deploy the app onto the cloud, no need to change anything there except to make sure it loads the right package. Furthermore, we have the basic package.lisp and example.asd that package all the code nicely.

Don’t worry about all the components in the example.asd for now. All you’ll need to get started is:

(asdf:defsystem #:example
  :serial t
  :description "Example cl-heroku application"
  :depends-on (#:hunchentoot
	       #:cl-who
	       #:postmodern
	       #:simple-date)
  :components ((:file "package")

A local environment is not complete without a local database. Please refer to the PostgreSQL documentation to learn about how to create a database and add a new user/password.

Once you have created a database and a user that is able to operate on it, you can add your username and password as a list in *local-db-params*, as shown on line 10 in src/util/heroku-utils.lisp. The important function here is db-params, which just returns the parameters that are necessary to connect to the database. If we are working on localhost, it returns your *local-db-params*, otherwise it will fetch the right parameters from Heroku.

Let’s do some testing! Make sure you have the dependencies loaded and that you are in the right package and try this:

EXAMPLE> (with-connection (db-params)
           (query (:select (:+ 1 2)) :single))
3
1

If that worked, congratulations, you have your local environment set up! Now let the fun begin.

2. Creating a table

The great part about Postmodern is that you can use CLOS objects as table entries. You can even use a CLOS definition to create the table in the database. First, let’s define our movie object:

(defclass movie ()
    ((id :col-type serial :reader movie-id)
     (title :col-type string :initarg :title :accessor movie-title)
     (rating :col-type string :initarg :rating :accessor movie-rating)
     (release-date :col-type date :initarg :release-date :accessor movie-release-date))
    (:metaclass dao-class) 
    (:keys id))

We have now defined a Database Access Object. In Rails, the id column is generated automatically, but we will need to specify that explicitly as a column with the type ‘serial’ — which increments automatically.

We have two string slots for the title and the rating and a date slot for the release-date. We use Simple-date for date-fiddling later. The :metaclass declaration is necessary to make it a DAO-class and the :keys attribute defines what the primary key is.

Now let’s try some magic:

EXAMPLE> (with-connection (db-params)
           (execute (dao-table-definition 'movie))))

This should create a table in your local database — use psql in your terminal to check if this really happened (assuming you called your db ‘test’):

$ psql test

test=# select * from movie;
 id | title | rating | release_date 
----+-------+--------+--------------
(0 rows)

Wonderful! Let’s move on and get CRUD!

3. Writing CRUD logic

CRUD is nothing but short for create, read, update and delete — four fundamental database functionalities. Very straight-forward tasks, so here is a short list that will get us by for now:

(defmacro movie-create (&rest args)
   `(with-connection (db-params)
      (make-dao 'movie ,@args)))

(defun movie-get-all ()
   (with-connection (db-params)
     (select-dao 'movie)))

(defun movie-get (id)
   (with-connection (db-params)
     (get-dao 'movie id)))

(defmacro movie-select (sql-test &optional sort)
   `(with-connection (db-params)
      (select-dao 'movie ,sql-test ,sort)))

(defun movie-update (movie)
   (with-connection (db-params)
     (update-dao movie)))

(defun movie-delete (movie)
   (with-connection (db-params)
     (delete-dao movie)))

The create function accepts arguments in the form of keyword-value pairs and calls make-dao. For example to add a movie to your local database:

(movie-create :title "Inception" :rating "PG-13" :release-date (encode-date 2010 7 16))

Note the use of encode-date — when we retrieve this value, we will use decode-date. More on that later.

Use psql to see if the movie has indeed been added to your database. You can also test the reader functions right away in the REPL, once you have populated the table with a few of your favourite movies. Play around, have some fun. Note that update-movie and delete-movie operate on Movie objects, which you may obtain using the reader functions.

These are very general functions that you will most likely need again when you are going to build a database of, say, books. As the founder of Rails probably thought: let’s abstract that away.

4. Abstracting it away using a macro

The most impressive feat of Ruby on Rails — in my humble opinion — is the simplicity and elegance to set the fundamentals straight and start building your application. With only two lines in one file you create tons of generic CRUD functions. You create a database by simply defining the columns. And finally with one more line, you create all the relevant (dispatching) routes with RESTful URIs. Very neat — you are already halfway there.

Unfortunately, there is no Lisp on Rails.

We will not attempt to build a Rails framework for Common Lisp, but the following is to illustrate that it sure is possible with macros. The defmodel macro in model-utils.lisp shows how we can mimic (some of) the power of Rails. With this macro in place, we can define the above CRUD functions and initialize the database table in one go with (model.lisp):

(defmodel movie
  ((title :col-type string :initarg :title :accessor movie-title)
   (rating :col-type string :initarg :rating :accessor movie-rating)
   (release-date :col-type date :initarg :release-date :accessor movie-release-date)))

There are obvious possibilities in extending the macro to be at least as powerful as the Rails framework. Perhaps an idea for a new project? Lisp on Rails? We sure could use one…

Enough logic, let’s move on, get some server running and write some web pages.

Hunchentoot and CL-WHO

Hunchentoot and CL-WHO is to Lisp what WEBrick and HAML is to Rails: A server and a powerful way to write HTML. To take the analogy one step deeper; what a dispatch table is to Hunchentoot is what routes are to Rails: a way of telling incoming requests what to do. In Common Lisp, websites are written like functions — called handlers.

5. The Index Page

Let’s just start with creating a macro for the standard layout that all our pages will be using and call it standard-page (see src/views/layout.lisp). Notice that this macro declares a css stylesheet to be used on line 9, pointing to “/site.css”. This does not point to anything unless we add it to Hunchentoot’s dispatch table (which we will do in a second).

To test our macro and server, we can write our initial index page as follows:

(defun controller-index ()
  (standard-page
    (:h1 "All Movies")
    (:a :href "movies/new/" "Add new movie")))

To make it accessible, we need to add a dispatch rule to the *dispatch-table*, which we’ll define in src/controller.lisp — as an attempt to adhere to the MVC-hype. All we need for now is:

(setq *dispatch-table*
      (list
       (create-regex-dispatcher "^/movies$" 'controller-index)
       (create-static-file-dispatcher-and-handler "/site.css" "static/application.css")))

Once you have this in place and evaluated or loaded, just launch the server by running in the REPL:

EXAMPLE> (defvar *server* (start (make-instance 'easy-acceptor :port 8088)))

This is getting exciting.. hearing the drums? Now go to http://localhost:8088/movies and hopefully you’ll see something! If not, please feel free to ask in the comments.

You might notice that going to http://localhost:8088/ will not lead you to the page that you want, since it is not a recognized URL by one of the rules in the dispatch table. We could add a dispatcher and point to the same handler, but nobody wants to be unRESTful, so let’s do it right and add another handler for it which just redirects:

(defun controller-to-index ()
  (redirect "/movies"))

Don’t forget to add the handler for this. Also note that on our dummy index page, we created a hyperlink to add a new movie, so we might as well add that to the dispatch table, since we’re about to create the handler for it:

(setq *dispatch-table*
      (list
       (create-regex-dispatcher "^/$" 'controller-to-index)
       (create-regex-dispatcher "^/movies$" 'controller-index)
       (create-regex-dispatcher "^/movies/new" 'controller-new)
       (create-static-file-dispatcher-and-handler "/site.css" "static/application.css")))

6. New movie page

This part is where Rails wins big-time. Forms. They are so important and annoying at the same time — a bunch of smart guys made it painless in Rails. If Lisp were to succeed on the web, we’ll need a framework that takes care of forms. Let’s just dive into it and you’ll quickly see what I mean.

Here is the Ruby code that we are attempting to translate. Very concise and powerful, especially the date_select part. If you look at the HTML source of the New page, you’ll see what I mean with powerful.

We do not have a form tool available in CL-WHO that allows us to create selectors or date pickers, unfortunately, but Lisp wouldn’t be Lisp if we couldn’t add that abstraction ourselves.

Before moving on, it is important to note that there is a funny thing about CL-WHO that does not allow us to use macros the way we would like to, as Vsevolod investigated. He proposed some approaches and went as far creating his own version of CL-WHO with the added def-internal-macro function. It worked initially, but once I started deploying it on Heroku, I ran into the CL-WHO version problem; Heroku uses Quicklisp to load the dependant libraries. The CL-WHO version that your app will use in the cloud does not have def-internal-macro.

I also asked it on Stackoverflow, where Inaimathi gave me an elegant solution, by using a wrapper macro:

(defmacro html-to-stout (&body body)
  `(with-html-output (*standard-output* nil :indent t) ,@body))

Now we can go ahead and define a function that will write all the repeating HTML code for us:

(defun selector-form (name options &optional selected)
  (html-to-stout
    (:select :name name
	     (mapcar
	      #'(lambda (x) (if (equal selected x)
				(htm (:option :selected "selected" :value x (fmt "~A" (mkstr x))))
				(htm (:option :value x (fmt "~A" (mkstr x))))))
	      options))))

This function is to be used within CL-WHO’s with-output-to-html macro. It accepts a name, which will become the name of the parameter that is passed to the form, and a list of options. An additional parameter can be provided to pre-select one of the options in your list — this will prove handy once we get to the edit page.

With these utilities in place, we can write the new page as shown here. Note that num-list is just a simple util that lists numbers, to make life even easier.

This is still not as powerful as what you see in Rails, but it’s coming pretty close for some ad-hoc Lisp hacking. Just imagine what a community might achieve…

Wait, we are not done yet. We still need to define what to do with the form’s POST request. Here’s the handler for that (and like any handler, don’t forget to add it to the dispatch table!):

(defun controller-add-movie ()
  (let ((title (parameter "movie-title"))
	(rating (parameter "movie-rating"))
	(year (parse-integer (parameter "year")))
	(month (parse-integer (parameter "month")))
	(day (parse-integer (parameter "day"))))
    (movie-create :title title :rating rating :release-date (encode-date year month day)))
  (redirect "/movies"))

The key here is the ‘parameter’ function, which retrieves the form parameters as named previously.

Notice the sleek usage of the movie-create function you wrote in step 3. Also notice the use of encode-date, which makes storing dates easy and consistent with postmodern. Without it, you would need three separate columns to achieve the same.

Play around and try adding movies to the database through the web-interface. You may check the results with psql; we will build the functionality to display the movies on the index page shortly.

7. Populate DB and show it

Now that we have the ability to add movies, we want to see them. The challenge here is not to create the table, but to populate the rows of the table, each of which should represent a movie that is stored in the database. We’ll create a function to do just that:

(defun all-movie-rows ()
  (dolist (movie (movie-get-all))
    (html-to-stout
     (:tr
      (:td (fmt "~a" (movie-title movie)))
      (:td (fmt "~a" (movie-rating movie)))
      (:td (fmt "~a" (print-date (movie-release-date movie))))
      (:td (:a :href (movies-path movie)
	       (fmt "More info about ~A" (movie-title movie))))))))

Three things to notice here: movie-get-all returns a list of all the movie objects present in the database; print-date is a simple utility to convert the encoded date into a string; movies-path will return the URI that points to the specific movie — as inspired by Rails.

With a function to populate the table, writing the table header is all that remains — we can finish our index page now.

Make sure to test, before continuing! You may even want to deploy it to Heroku to see if everything still works in the cloud.

To do this, first make sure all your changes are committed:

$ git commit -am 'tutorial end of step 7, first attempt'

Now, let’s deploy it! This may take a minute:

$ git push heroku master

To open your app in your browser, just type:

$ heroku open

Any luck? Don’t be afraid to ask for help!

8. Show page

Next, we want to create a show page for when a user clicks on a movie. Now here is a problem, since we want to maintain RESTful URIs, we cannot rely on hidden parameters to point us to what movie we are looking for — we can solely rely on the URI. This means that if someone would enter http://localhost:8088/movies/1 in the URL box, it should bring you to the details page of the first movie directly.

Rails incorporates RESTful URIs very elegantly, by enabling certain parts of the URI to be captured by a variable using regular expressions. This variable is accessible to the controller, which then retrieves the correct movie from the database. If you have no idea what I am talking about, watch some Rails for Zombies.

Perhaps Hunchentoot and CL-PPCRE could use a patch there, Mr. Weitz 😉

Anyway, until that patch arrives, here’s a workaround:

(defun get-id-from-uri ()
  "Returns the ID from the URI request."
  (car (cl-ppcre:all-matches-as-strings "[0-9]+" (request-uri *request*))))

Conveniently, Hunchentoot saves the information about the request, so using the above we can extract the id part of the URI — albeit not robustly, since any other numbers in the URI might screw it up. But it’s just enough to keep us going.

With that, we can define the show page pretty straight-forward and hosted in the cloud it kinda looks like this. Notice the RESTful URI here.

9. The Edit page

The edit page is very similar to the new page, except that the values are already filled in with the movie’s details.

Remember when we mentioned that the selector-form function with an additional optional argument will come in handy?

Other than that, the rest is easy. Notice the learning curve?

We do need to define another handler for updating information on the movie, one that handles the POST:

(defun controller-update ()
  (let ((title (parameter "movie-title"))
	(rating (parameter "movie-rating"))
	(year (parse-integer (parameter "year")))
	(month (parse-integer (parameter "month")))
	(day (parse-integer (parameter "day")))
	(movie (movie-get (get-id-from-uri))))
    (setf (movie-title movie) title
	  (movie-rating movie) rating
	  (movie-release-date movie) (encode-date year month day))
    (movie-update movie))
  (redirect (conc "/movies/" (get-id-from-uri))))

Notice how we update a movie’s attribute using setf and updating it to the database using movie-update. Also, the redirect after updating the movie should be the movie itself.

10. Delete movie

To delete a movie is easy — we already have a link in place on the show page, all we need to do is add a dispatch-rule and a handler:

(defun controller-delete ()
  (movie-delete (movie-get (get-id-from-uri)))
  (redirect "/movies"))

Obviously, it would be better not to use a link, but a button, which then asks for confirmation (like in the Rails version). I got lazy, but it sure is possible using Parenscript to embed some Javascript. But I don’t like Java, so I couldn’t be bothered.

11. Tying it all up

By now, your dispatch-table should look something like this:

(setq *dispatch-table*
      (list
       (create-regex-dispatcher "^/$" 'controller-to-index)
       (create-regex-dispatcher "^/movies$" 'controller-index)
       (create-regex-dispatcher "^/movies/new" 'controller-new)
       (create-regex-dispatcher "^/movies/[0-9]+$" 'controller-show)
       (create-regex-dispatcher "^/movies/[0-9]+/edit" 'controller-edit)
       (create-regex-dispatcher "^/movies/[0-9]+/update" 'controller-update)
       (create-regex-dispatcher "^/movies/[0-9]+/delete" 'controller-delete)
       (create-regex-dispatcher "^/add-movie" 'controller-add-movie)
       (create-static-file-dispatcher-and-handler "/site.css" "static/application.css")))

Depending on how you structure your folders, but here’s my ASD system definition in example.asd. Also, don’t forget your package definition. Lastly, make sure you have changed heroku-setup.lisp appropriately. Are you ready for the grand finally?

12. Sky is the limit!

$ git push heroku master

Now sit back, fingers crossed, it might take a while… it took me at least a few dozen Heroku pushes along the way to get it right. If anything goes wrong, you may want to check the logs:

$ heroku logs

If it finishes without crashing, congratulations, put your hands up in the sky and scream “woohoo!”, (that’s what I did). Now go ahead, get on your jet-pack and admire your app in the cloud:

$ heroku open

Let me know if it succeeds! Also let me know if it fails, I’m happy to help!

Concluding Remarks

This is just the beginning, but the beginning is where it starts. Feel free to copy whatever you see on this post or fork the example app and hack away to your hearts content.

Where is this heading? Mike Travers, who made Lisp on Heroku possible, sure has an optimistic outlook on the possibility of Lisp being everywhere. He has lowered the thresholds with raising Lisp to the clouds — not to discredit Xach who wrote ZS3 back in 2009 to enable Common Lisp to interface with Amazon Web Services.

The cloud computing revolution sure is gaining momentum — hopefully Common Lisp will ride this wave. I hope you have enjoyed doing this tutorial as much as I enjoyed writing it. Any suggestions for improvement are more than welcome!

Discussion on Hacker News and Proggit

---

Sources:

• Lisp for the Web
• Lisp for the Web, Part II
• Engineering Long-Lasting Software