Why Ruby is an acceptable LISP (2005)
Years ago, I looked at Ruby and decided to ignore it. Ruby wasn’t as popular as Python, and it wasn’t as powerful as LISP. So why should I bother?
Of course, we could turn those criteria around. What if Ruby were more popular than LISP, and more powerful than Python? Would that be enough to make Ruby interesting?
Before answering this question, we should decide what makes LISP so powerful. Paul Graham has written eloquently about LISP’s virtues. But, for the sake of argument, I’d like to boil them down to two things:
- LISP is a dense functional language.
- LISP has programmatic macros.
As it turns out, Ruby compares well as a functional language, and it fakes macros better than I’d thought.
Ruby is a denser functional language than LISP
A dense language lets you say things concisely, without obfuscation. You can see more of your program in one glance, and there aren’t as many places for bugs to hide. Beyond a certain point, the only way to make programs denser is to use more powerful abstractions.
One particularly powerful abstraction is lambda
. Using lambda
, you can
create a new function on the fly, pass it to other functions, and even
store it for later use. For example, if you wanted to double each number
in a list, you might write:
(mapcar (lambda (n) (* n 2)) mylist)
mapcar
creates a new list by transforming each element of mylist
. The
transformation, in this case, could be read as “for each value n,
multiply n by two.” In JavaScript, you’d write lambda
as function
,
which is perhaps a bit clearer:
map(function (n) { return n*2 }, mylist)
Of course, this is only a hint of what you can do with lambda
. Languages
which favor this style of programming are called functional languages,
because they work with functions. A dense functional language can be very
concise indeed, and quite clear once you learn to read it.
How does Ruby stack up against LISP for functional programming? Let’s consider Paul Graham’s canonical example, a function which creates an accumulator:
(defun foo (n) (lambda (i) (incf n i)))
This code is marginally shorter in Ruby, and the notation will be more familiar to C hackers:
def foo(n) lambda {|i| n+=i} end
acc = foo 3
acc.call(1) # --> 4
acc.call(10) # --> 14
acc.call(0) # --> 14
But there’s an interesting special case in Ruby which saves us even more
typing. Consider a (very silly) function which takes a lambda
as an
argument:
;; Call 'fn' once for each natural number.
(defun each-natural-number (fn)
(loop for n from 1 do (funcall fn n)))
;; Print 1, 2, 3...
(each-natural-number
(lambda (n) (format t "~D~%" n)))
Now, we could write the same function in Ruby:
def each_natural_number(fn)
n = 0
loop { fn.call(n += 1) }
end
each_natural_number(lambda {|n| puts n })
But we can do better. Let’s get rid of lambda
and fn
using yield
:
def each_natural_number
n = 0
loop { yield n += 1 }
end
each_natural_number {|n| puts n }
Yes, yield
is a special-purpose hack, and yes, it only works for functions
which take a single lambda
. But in heavily functional code, yield
buys us a lot. Compare:
[1,2,3].map {|n| n*n }.reject {|n| n%3==1 }
(remove-if (lambda (n) (= (mod n 3) 1))
(mapcar (lambda (n) (* n n))
'(1 2 3)))
In a large program, the difference adds up. (In LISP’s defense, it’s
possible to write a reader macro which makes lambda
more concise. But
this is rarely done.)
Ruby gives you about 80% of what you want from macros
At this point, the LISP hackers are saying, “A good syntax for lambda
is nice, but what about macros?” And this is a good question. LISP
macros are functions that:
- Run in the compiler, and
- Transform custom syntax into raw LISP.
The most common use of LISP macros is to avoid typing lambda
quite so
much:
(defmacro with-each-natural-number (n expr)
`(each-natural-number (lambda (,n) ,expr)))
(with-each-natural-number n
(format t "~D~%" n))
defmacro
defines a function that takes a list as an argument, and returns
another list. In this example, our macro is called every time the compiler
sees with-each-natural-number
. It uses LISP’s “backquote” syntax to
quickly construct a list from a template, filling in n and expr. The
list is then passed back to the compiler.
Of course, this macro would be useless in Ruby, because it’s working around a problem we don’t have.
The second most common use of LISP macros is to create mini-languages for defining stuff:
;; Generate some bindings to our database
;; using a hypothetical "LISP on Rails."
(defmodel <order> ()
(belongs-to <customer>)
(has-many <item> :dependent? t))
Using Ruby on Rails, we could write:
class Order < ActiveRecord::Base
belongs_to :customer
has_many :items, :dependent => true
end
Here, belongs_to
is a class function. When called, it adds a bunch of
member functions to Order
. The implementation is pretty ugly, but the
interface is excellent.
The real test of any macro-like functionality is how often it gets used to build mini-languages. And Ruby scores well here: In addition to Rails, there’s Rake (for writing Makefiles), Needle (for connecting components), OptionParser (for parsing command-line options), DL (for talking to C APIs), and countless others. Ruby programmers write everything in Ruby.
Of course, there’s lots of advanced LISP macros which can’t be easily ported to Ruby. In particular, macros which actually compile mini-languages haven’t appeared yet, although they might be possible with enough work. (Ryan Davis has done some promising work in this direction with ParseTree and RubyInline, and I’ll be writing about related techniques as I discover them.)
Ruby’s libraries, community, and momentum are good
So if LISP is still more powerful than Ruby, why not use LISP? The typical objections to programming in LISP are:
- There aren’t enough libraries.
- We can’t hire LISP programmers.
- LISP has gone nowhere in the past 20 years.
These aren’t overwhelming objections, but they’re certainly worth considering.
Once upon a time, Common Lisp’s standard library was considered huge. But today, it seems painfully tiny. Java’s manuals fill a wall, and Perl’s CPAN archive has a module for anything you can imagine. Common Lisp, in comparison, doesn’t even have standard way to talk to the network.
Similarly, LISP programmers are scarce. If you’re around Boston, there’s a small pool of grizzled hackers who can very nearly work magic. Elsewhere, there’s a thin scattering of curious young hackers. But LISP has always been a minority language.
Ruby, on the other hand, is growing rapidly in popularity. The big driver seems to be Rails, and the ramp-up started in late 2004. If you’re trying to launch a company, it’s more-or-less a cliché that every potential employee is a Rails nut. Rails will soon trickle back into ordinary web consulting, and from there–eventually–into big business.
Ruby has also been around long enough to develop a good standard library, and a large archive of add-on libraries. If you need to download a web page, parse RSS, generate graphs, or call a SOAP API, you’re all set.
Now, given a choice between a powerful language, and popular language, it may make excellent sense to pick the powerful one. But if the difference in power is minor, being popular has all sorts of nice advantages. In 2005, I’d think long and hard before choosing LISP over Ruby. I’d probably only do it if I needed optimized code, or macros which acted as full-fledged compilers.
(Thank you to Michael Fromberger for reviewing an early draft of this essay.)
Want to contact me about this article? Or if you're looking for something else to read, here's a list of popular posts.
if
statement really is an expression; the parser just refuses to nest it. In general, the Python Language Services are a pretty promising framework for implementing a macro system; somebody just needs to roll up their sleeves and make it happen. Bill, you're absolutely right about LISP and compile-time versus runtime performance. Exhibit A: Ruby on Rails takes a couple of seconds to launch a new server process. With a good compiler, LISP could do the same thing as fast as the OS could read pages into memory. The long-term trends probably favor the LISP (or even Scheme) approach, mostly for reasons of IDE support and better compile-time checking. The Ruby approach is actually dangerous if your team doesn't maintain unit tests.iter
macro lives in the compiler. The same goes for the IDv3 parser: It would look fine in Ruby, but get absolutely clobbered on performance. A year from now, this could actually be a pretty good sales pitch for Common Lisp. "It's like Ruby, but it runs at full hardware speed!"IO
, which makes Haskell statements execute sequentially, so you can talk to the outside world. Other monads implement assignment, list comprehensions, continuations, and domain-specific languages (like the parser combinators above). Haskell can be a challenging language—some common programming idioms can only be expressed with monads, which are at least as hard as macros—but, like Prolog, Haskell makes certain programs look gorgeous. And of course, there's no reason why you couldn't apply many Haskell ideas in Lisp, if that's what you want. If you've mastered macros, and you're looking for cool programming styles, Haskell's not a bad place to start.Here's an example of a simple arithmetic parser and evaluator using the Parsec library. And below is a little snippet of a non-deterministic program in action. Say you want find a pair of numbers which multiply out to 8633. Well, this isn't the most efficient way to do it, but it looks nice and declarative...
...nothing new there, something that could also be done in lisp without macros (especially if your lisp comes with an "amb" operator). But it is nice to note that the list monad which makes this work is implemented in 7 lines of code.if
,defun
, etc.) are technically "special forms," not functions, because they don't necessarily evaluate all their arguments. Other than that, your point is good. Bill, here's some Ruby code (based on ParseTree and an unpublished library of my own), which isn't _entirely_ unreasonable:assert_body
compares to the block's AST. This is a small step towards recreating Paul's example. Jamie, speaking as a minor contributor to several Lisp compilers, I doubt that Ruby will ever be as fast as Common Lisp. There's two big obstacles: (1) Ruby metaprogramming happens at runtime, but Lisp macros run in the compiler. It's hard to beat zero runtime cost. :-) (2) Ruby supports a variety of dynamic features, such as duck typing andmethod_missing
, which are _much_ harder to optimize than the equivalent features in Lisp. Of course, the best SmallTalk VMs are very good, and a hypothetical Ruby VM could certainly be adequate for almost all applications.lambda
andmap
in Python, and why they're not quite as helpful as they might be, see the earlier discussion.(mapcar (lambda (n) (* n n))
'(1 2 3))) BUT. More can be said in Lisp's defence than you say. Macros can be used not just to make lambdas more consize. If you consider the above expression hard to read, and prefer to read in the order of computation, as in your Ruby example, take a look at this: (stream-2nd-arg '(1 2 3)
(mapcar (lambda (n) (* n n)))
(remove-if (lambda (n) (= (mod n 3) 1)))) Note that mapcar and remove-if have no 2nd argument. Where do they get it from? From the previous computation. The first arg to 'stream-2nd-arg' macro is evaluated first. Other args are function calls evaluated after that in turn, and each gets their 2nd argument from the previous call. Simple, concise and readable too. The implementation of stream-2nd-arg macro is left as an excersize to the reader :)