Seeking Closures

So, what’s a closure and why should I care?


As Bob Dylan said, “…the times they are a changin’.”   The strange world of functional programming, once restricted to more esoteric languages such as LISP, Erlang and Haskell, is now invading the mainstream.  Languages such as C# and Java are now incorporating concepts once unique to functional programming languages.  These changes should bring simplicity to these more mainstream languages, but I find they are often misunderstood (by myself for certain) and thereby misused or not taken advantage of.  I’ve had a fair understanding of closures for a while, but wanted to develop a more solid understanding, ergo this post.


One aspect of functional languages, called closures, have been talked about since .Net 2.0 came on the scene.  Of course, what many presented as closures in 2.0 really weren’t — for a number of reasons that I don’t care to get into at this time.  More recently with the 3.0 framework (and the same will hopefully be true with Java 7), we see the introduction of lambda expressions; and finally have true support for closures (remember, not all anonymous functions are closures).  You may ask, however, what the heck are closures anyway?  Well, I’ll do my best to give an accurate illustration.


First off, what are they.  As the Wikipedia link above stated, “a closure is a function that is evaluated in an environment containing one or more bound variables.”  Neal Gafter, during a Google Talk given in January 2007, put it another way by saying that a closure is “a function that refers to free variables in its lexical context.”  We all should know what functions are…they are constructs that take a number of parameters and return a result (this includes more than just returning void).  A free variable (or bound variable) is defined as a name (or symbol) that has a definition outside of the closure itself.  The “lexical context” can be thought of as the context of the caller.  I’ll give an example later.


One thing that I found particularly interesting about Gafter’s presentation was how he helped define closure by describing what they should support.  Among other things, he mentioned that closures should have access to private members of the “lexical context” in which they are used.  In C# and Java, they can also access the “this” instance of the caller.  They should also support return values and support throwing exceptions (the latter is more of a Java concern).  He also pointed out that although they are executed within the context of the caller, they can only return control to the caller (and not force the caller to return). 


Gafter also pointed out that closures — as proposed in Java at least — are methods that act as new statements in the language and can be used to extend the native API (he calls these “control abstraction”).  He described them as “concise function literals” that should behave as built-in statement, such as constructs such as for, while, lock, using, etc. in C# but without the problems associated with anonymous delegates and the like.  Again, they are not anonymous class instances (such as delegates, which are not pointers to methods, but rather classes used to box methods).


Okay, so we have at least a rudimentary definition of closures; now let’s see how they can be useful.  To again steal from Gafter’s presentation, I’ll use a couple of his examples that I found useful. 


First off, what’s one of the problems closures can address?  Gafter was describing a bit of code that was used repeatedly throughout an application at Google to gather run-time performance metrics on various operations.  He argued that closures could be used to extend the native API and offer a more elegant syntax.  Here’s what the code looked like:


image


Granted this is only 8 lines of code, but only one of the 8 lines relates to the business logic of the application.  The additional code (the other 7 lines) ends up cluttering the code and thereby obscures the surrounding business code.  It would be nice if this code could be replaced by something like this (this would probably be a static method from some util class):


image


Which with the Java’s proposed new method invocation syntax would mean the same thing as:


image


Neat, huh?  With this example, the time() function is used in a manner similar to “for”, “using”, and other such operations native to the API.  This is an instance where using a closure could be used to wrap common functionality around business logic in a more sensible way.  Since the time method is a closure, we could also access doSomething() is it was a private method, such as this:


image


Pretty groovy huh? (pun intended)


 


From the horse’s mouth…


*WARNING: If you have a burning lack of curiosity or can’t fathom looking at code other than code written in your favorite language DO NOT continue reading.  Reading LISP code without an open mind can lead to a general sense of disaffection and in more extreme cases unreasonable zealotry for your preferred language. 



As stated recently I’ve been digging deeper into LISP lately.  In LISP such features are native (and natural) to the language.  For some reason I tend to think that closures look for magical (as well as cleaner) in LISP programs.  I’ll give another example of closures, but this time with one of the parent languages to all this closure goodness.


So here’s the scenario…which, by the way, is an adaptation of the practical exercise in Chapter 3 of Practical Common Lisp.  In the example I’m writing a simple little application to help break down the living things in my yard into a truncated taxonomy.  Once finished, I’ll be able to look up groups of living things by either kingdom or phylum.  Alternately, I can look up a living thing by name and see to which kingdom and phylum they belong.  Bear in mind I probably would never bother with this in real life…


Setting things up:


First off, I’ll create a global symbol for my collection of living things, a function that creates a new living thing, and a function to add a new living thing to my global collection.  Now I know that I should be using the CLOS, but I don’t want to confuse anyone with slots and whatnot.


image


That was simple enough.  We see that *living-things* is initialized as an empty list (or null set).  The function “make-living-thing” uses the “list” operator to create a special kind of list called a “property list”…think of it as a hashmap.  The “add-living-thing” function simply pushes whatever is passed to it onto the *living-things* collection (which I’m using as a stack in this case).


Now I’ll walk around my yard and add the living things that I find…


image


Okay, so I left Bermuda Grass out…most of the grass in my yard is dead anyway.  I can print out the contents of *living-things* and see the raw data:


image


Now to make things look a little prettier (only a little, mind you), I’ll add a few functions for displaying the data in a more readable format:


image


And now I have something like this (output is truncated):


image


And now for something completely different…


Okay, now we are nearing where closures enter the example.  Next I want be able to pull data from *living-things* using a SQL-like syntax.  Don’t ask why, I blame it on too much HQL. 


Since I have only one entity type, I’ll just copy the list passed to the from clause and return it (nothing to it…copy-list came free with Common LISP).  The select statement takes a data source (ultimately using the from function) and n comparison predicates.  The function “remove-if-not” is a Common LISP function that removes items from the list (passed as the second parameter) whenever the predicate evaluates to false/nil (the first parameter).


image


If you run the code from the comments you get the following:


image


Here you see the first use of a lambda expression to make an anonymous function (a function not associated with a symbol).  The inner function, getf, is just a getter function that gets the value associated with the symbol :kingdom.  The variable x in the getf is the current list item.  For instance, the following gets the value associated with :kingdom for the first item in *living-things*:


image


The equal function returns true (T) if the value for :kingdom at x is equal to ‘PLANTAE.  That’s all that there is to the lambda expression…it simply serves as the comparison predicate for “remove-if-not”.  There’s still no closure here…any symbol could be used instead of :kingdom and the expression would still be valid:


image


See? return NIL, no problem. 


End game…


Now we’ll work on the where clause.  Passing lambda expressions is kind of ugly, though, so we’ll want to create an operator that will handle the where clause.  Once finished, we should be able to make something like the following call and get all the matching entities back:


(select (from *living-things*) (where :kingdom ‘ANIMALIA :phylum ‘ARTHOPODA))


This part gets a little funky because we’re going to use macros.  Macros in LISP are similar to macros in C, but pumped up on steroids.  Macros in LISP are used for run-time code generation (but much more elegant than IL generation in .Net).  I’ll skip over some of the details on macros but will attempt to get at the spirit of this whole thing:


image


This part’s a little hard to follow.  We are focusing on the “where” macro, but I think we’ll start with the simpler predicate functions. 


Birds eye view: First the “where” macro utilizes the “make-comparisons-list” function to generate a comparison predicate from a set of criteria.  The “make-comparisons-list” function utilizes “make-comparison” for create a part of the overall comparison predicate using a field/value pair (such as :kingdom ‘ANIMALIA).


The “make-comparison” function works just like the lambda function in the example above.  First it gets the value associated with the specified symbol (such as :kingdom) and then checks that value against value that was passed as a parameter.  Here’s where we see a closure, where “living-thing” appears out of nowhere.  First off, the backquote before the list containing “equal” is used to evaluate the list as code, the commas latter in the list mark variables.  Without the backquote and commas we’d have an error because “living-thing” is not a bound variable (and that’s bad).  This backquote facility is very important in that it allows the coder to easily define code that can be evaluated later.  So, if you execute “make-comparison” you get a piece of the code that the macro will ultimately execute.  Again, I’m sure you noticed that “living-thing” isn’t a bound variable…yet.  We’ll get back to that.


Since we want to support more than one comparison predicate, we’ve also created the “make-comparison-list” function with iterates over a list of fields and values (or properties and their associated values).  The “loop while x collecting y” construct is an iterator with an accumulator.  Basically it is iterating over all the field/value pairs (x), popping the field then the value and passing them to “make-comparison”, and finally appending the resulting code (y) to a new list (this is a Common LISP feature).  The result is a list of comparison predicate expressions. 


Here the macro takes everything following the “where” operator and constructs the appropriate of comparison predicates.  The where function will iterate over the list returned from the “from” function and pass each list item to the lambda expression. The variable “living-thing” is introduced as a parameter in the lambda expression, which will be bound by the time the code resulting from “make-comparison” is evaluated (where the closure is).  The “and” function is used for the conjunction between all the comparisons resulting from calling “make-comparisons-list”.  Ideally, I’d use a more generic name for the bound variable used in “make-comparison”, so that I could create predicates for any other property list.


To illustrate the code that’s generated I’ll make a few calls to the function macroexpand-1 (it’s kind of like reflecting over the macro):


image


So, now you’ve seen the final lambda expression generated by the macro when using both multiple field/value paris and when using only one.  Now we’ll put together the entire select statement and test it…I’ll get all animals from *living-things*.


image


…and now we’ll select all plants and use the “friendlier” display:


image


So, as far as using closures, the closure on “make-comparison” was handy for supporting the lambda expression in the “where” function.  I think that LISP is nice to clearly illustrate how closures work.  One doesn’t NEED to use closures…but at times they are quite useful for making code a bit simpler and easier to read.  I hope this post was helpful in elucidating the nature of closures and how they might be used.  I also hope it helps show how fresh a 50+ year old language can be.


For more information on closures:


This article seemed like a decent illustration of their use in C#.


If you can make it through the near 2 hour presentation, Neal Gafter’s discussion of Closures in Java about a year and a half ago was an interesting illustration of what they might look like in Java.  I also used some of his discussion as an example in this article.  (It’s also fun to watch the original crowd of near 60 people dwindle down to about 10 by the end of the presentation)


Then of course there are the scores of books and sites about LISP, Erlang, Haskell, Ruby, JavaScript and other languages that feature closures as a central feature of the language.


Happy Coding!


Joshua

Related Articles:

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

About Joshua Lockwood

I code stuff and people pay me for it.
This entry was posted in c#, closures, Functional Languages, Java, LISP, Programming. Bookmark the permalink. Follow any comments here with the RSS feed for this post.

20 Responses to Seeking Closures

  1. Ramarren says:

    Lisp was not called LISP since the seventies or so. So when someone who knows that sees LISP, to mind comes an image of original LISP 1.5 or so, which didn’t have lexical scoping, and so no real closures anyway, which makes this post cause some cognitive dissonance. Minor one, but still…

  2. foo says:

    there are no closures in your second part

  3. foo says:

    actually in your Lisp examples are no closures at all. I see only ordinary functions.

  4. Justin Paston-Cooper says:

    I like how you feel the need to pander to the average programmer’s thoughts of “OH NO ITS NOT A C BASED LANGUAGE IT LOOKS DIFFERENT SO IT MUST BE BAD AAAAAH”, in order to make these people associate with you. If it’s these people whom you’re writing for, maybe you should stop writing these articles altogether. What you are doing is analogous to Bill O’Reilly’s added comments after a news story, which are only there to agree with what the average American might be thinking while watching TV. Clearly he’s not trying to disseminate truth and give education.

  5. jlockwood says:

    LOL, looks like I’ve rubbed a couple of people the wrong way.

  6. David Alpert says:

    Another interesting discussion of closures is taking place in the JavaScript space (e.g. http://blog.morrisjohns.com/javascript_closures_for_dummies).

    If not identical to the way closures operate or are implemented in modern LISP, Java, or .NET, then at least JavaScript closures might be another way into the concept.

  7. foo says:

    Joshua, but it is clear to you why your Lisp code does not use any closure?

    First: an anonymous function is not a closure.

    (defun foo () (lambda () ‘bar))

    Calling foo does not create a closure. A closure is a function that references a variable from the enclosing lexical scope. None of the function you define does that.

    Second: the closure is a function and its bindings:

    See this:

    (defun make-counter (start increment) (lambda () (let ((value start)) (incf start increment) value)))

    MAKE-COUNTER returns a closure, since the LAMBDA expression references START and INCREMENT, which are defined outside the lambda expression. The closure then is the anonymous function plus the bindings for START and INCREMENT.

    If you look at your code, none of the functions references variables that are introduced by its lexical surroundings.

    You simply don’t explain anything about closures here.

  8. jlockwood says:

    @Ramarren

    I’m afraid I must repectfully diagree with you, but I do admit that the name came after the language. Here’s a little history…to avoid argument I’ve provided references.

    We can use numerous references, but I’ll start with wikipedia (1). First off, I agree with the article when it asserts that LISP is actually a family of languages…so Scheme is LISP, just a different dialect. Given that, the term LISP is a symbol referencing all languages in the family allowing us to reference implementations that pre-date the term itself.

    The wikipedia article credits John McCarthy as the language designer (in 1958), which I guess is technically true, but it was originally designed strictly as a mathematical notation for describing computer programs (based on lambda calculus). McCarthy first set out to create a programming language for AI (based on list processing) in 1956 (2). The first actual implementation of LISP was written later that year by S. R. Russell (2). Because Russell first implemented LISP on an IBM 704, we now have the obiquitous and funky functions car and cdr (since they were machine instructions native to the platform).

    I did find a site that has an image from the LISP 1.5 Programmers Manual which was written by McCarthy et al. in 1962, so it’s evident that the language was called LISP much earlier than the 70s (3). McCarthy regarded his new creation as a list processing language from its inception, so I think it’s fair to say that the language was indeed LISP as early as 1958.

    Thanks for the post and I hope that clears things up a bit.
    Cheers!

    (1) Lisp (programming language). (2008, September 1). In Wikipedia, The Free Encyclopedia. Retrieved 14:05, September 4, 2008, from http://en.wikipedia.org/w/index.php?title=Lisp_(programming_language)&oldid=235653687

    (2) McCarthy, John (1996). The implementation of LISP. The History of Lisp. Retrieved 14:08, September 4, 2008, from http://www-formal.stanford.edu/jmc/history/lisp/node3.html

    (3) McJones Paul, Ed. (2008). History of Lisp. Software Preservation Group. Retrieved 14:10, September 4, 2008,
    from http://www.softwarepreservation.org/projects/LISP/index.html#LISP_I_and_LISP_1.5_for_IBM_704,_709,_7090_

  9. jlockwood says:

    @Foo

    I’m afraid you are incorrect. Perhaps you misread something? The closure is in “make-comparison” and it references the variable (its the parameter) in the lambda expression.
    (see the paragraph starting with “The “make-comparison” function works just…”)
    The article clearly states that “make-comparison” is where the closure is (which is restated in the paragraph describing the marco). One objective of the article is to illustrate that labmda expressions are not closures – which is apparently not understood by same (based on articles that I’ve read).

    The lexical context of the closure is the lambda expression in the macro (which is shown in the example using macroexpand-1). The function “make-comparison” can not be evaluated by itself, since “living-thing” is not a bound variable (hence the closure).
    For example:
    (eval (make-comparison :test ‘me)
    would cause the following error:
    “attempt to take the value of the unbound variable ‘LIVING-THING’.”

  10. foo says:

    Joshua, unfortunately that’s not correct. There is no closure.

    Look at the source of MAKE-COMPARISON.

    It returns a list. The variable LIVING-THING is just a symbol in that list. The macro takes those lists and creates a new list which then gets used as code.

    The backquote expression you have there:

    `(equal (getf living-thing ,field) ,value)

    living-thing is just a symbol in a list. The list is simply like:

    (list ‘equal (list ‘getf ‘living-thing field) value)

    Evaluate that and you get the result list.

    Of course you can’t evaluate the result list, because living-thing is nowhere defined. To form a closure you need two things: living-thing needs to be a variable and it needs to reference a definition somewhere. But there is no definition of living-thing around MAKE-COMPARISON.

    But yor code later is:

    (lambda (lliving-thing) (and (equal (getf living-thing :test) :me)))

    Then living-thing is a variable in the lambda-expression. Again no free variable.

    You CAN repair your example but then you need to get rid of your macro. Can you imagine how a closure-based version would look like? If not, I’ll post a version for you.

    Think about it… ;-)

  11. jlockwood says:

    @Justin Paston-Cooper

    Come now Justin, you must know that I joke with tounge in cheek! I don’t feel that I need to pander to anyone…nor do I try associate with many people for that matter (ask one of my handful of friends…LOL).

    In truth, I know of a number of people who are adept at several languages yet still find LISP to be cryptic and even intimidating. I wouldn’t doubt that the way some universities present the language contribute to this paradigm.

    > I like how you feel the need to pander to the average
    > programmer’s thoughts of “OH NO ITS NOT A C BASED
    > LANGUAGE IT LOOKS DIFFERENT SO IT MUST BE BAD
    > AAAAAH”…
    The funny things about this is that I’ve found such attitudes to be more common among some Ruby developers that I’ve met…they seem to think that there is no other language but theirs. Of course, there is the VB crowd too…

    The bottom line is that LISP is not a common language and is often forgotten outside of the university. In spite of this, its influence is becomming increasingly apparent in the mainstream languages that get the most attention. I believe that studying the language will help improve overall programming ability as well as offer clear illustrations of concepts introduced into the languages used daily in the workplace.

    If I appeared to pander to one group or another I apologize. I was simply joking about the way LISP is commonly perceived. Its a great language and you don’t have to look like the Unibomber to enjoy it!

  12. foo says:

    Joshua: what Ramarren says: Lisp is nowadays called ‘Lisp’. In earlier years it was called LISP. Years ago it was all upcase. Even the programs were upcase. Common Lisp has the symbols also internally in upcase format. Nowadays when somebody sees LISP then he thinks of the older Lisp dialects from before the 70s.

  13. jlockwood says:

    @Dave

    Nice article! I was considering getting into JavaScript, but decided against it because — although I love JavaScript as a language — I thought the language syntax might have made things a little confusing. After getting deeper into JavaScript, I began to view functions as class definitions. Closures are central to JavaScript as they are the only means in supporting object-oriented programming in the language.

    It’s interesting, I think if one views JavaScript through the eyes of a LISPer the language kind of makes more sense. I say this because in LISP the line between code and data is kind of blurred. JavaScript uses the same syntax to define functions as it does data constructs (similar to lisp).

    I certainly agree that looking at JavaScript’s implementation of closures is quite useful in developing an overall understanding of the concept.

  14. jlockwood says:

    @foo

    Ah, if that’s the case then I misunderstood. I still write it as LISP, even when referencing CL. I didn’t even consider case, as the language is not case sensitive (at least with the implementation I’m using). As you stated, everything is evaluated in upcase.

    @Ramarren
    My apologies if I misunderstood you.

  15. Ramarren says:

    Yes, I meant the case. I probably should have said that directly. But really, I wasn’t even born in the seventies, so I’m just nitpicking. So, while I’m doing that, Common Lisp is case sensitive, and required to be so by the standard, it is just that it is by default in :upcase mode for compatibility with older LISPs. See: http://www.lispworks.com/documentation/HyperSpec/Body/23_ab.htm

  16. foo says:

    Here is a version that really uses closures:

    http://lispm.dyndns.org/source/closures-at-work.lisp

    You can see that the result of calling (where …) is a closure – the Lisp systems (LispWorks) also tells us that. There are multiple closures involved. MAKE-COMPARISON-FUNCTION returns a closure. MAKE-COMPARISON-FUNCTIONS returns a list of closures. WHERE also returns a closure.

  17. @foo

    Why not use your real name here? You have good input. Just curious. Anonymous commenters drive me nuts.

  18. jlockwood says:

    @Ramarren

    Okay, I see your point. Point taken. :)

    @foo

    Ah, I see what you are saying…my mistake. Thanks for clearing things up. As you say, “make-comparison” is currently used to create a chunk of code and is not truly a closure.

    I’ve quickly read over your example and found it useful. I’d never seen the “every” function before and couldn’t find it in the common lisp command reference that I’ve been using. I imagine its part of common lisp…ran fine in Allegro CL. With your permission I’d like to update the blog entry to reflect what you’ve provided.

    I appreciate the feedback. I’m but a LISP neophyte, so I’m still figuring a number of things out. I see now that I must always use “let” to bind a variable locally and then I can use it in a closure. I’ll have to play with this a bit.

  19. Peter says:

    If you’re wondering where all the anonymous commenter(s) came from, this article got linked on Reddit: http://www.reddit.com/r/programming/comments/6zkav/seeking_closures_so_whats_a_closure_and_why/

  20. foo says:

    joshua:

    EVERY is a function in Common Lisp, see here:
    http://www.lispworks.com/documentation/HyperSpec/Body/f_everyc.htm

    You actually don’t need the variable to be introduced by LET. Any (lexical) variable will do.

    Also:

    (let ((a 13) (b 14)) (+ a b))

    is conceptually the same as

    ((lambda (a b) (+ a b)) 13 14)

    So you get a closure with:

    (defun foo (a) (lambda (b) (+ a b)))

    and (foo 13) then is a closure that you can call and will always add 13 to a number: (funcall (foo 13) 22)

    Also note that you get a closure from this:

    (let ((a 10)) (flet ((add (b) (+ a b))) (function add)))

    Any function will do. Local named functions, too.

    Above you see that a closure is returned.

    You can also pass a closure to another function:

    (let ((a 10)) (flet ((add (b) (+ a b))) (mapcar (function add) ‘(1 2 3 4))))

    Above will add 10 to every element of the list and return the result list.
    MAPCAR gets passed a closure and calls it with every element of the list as argument.

    Sure, go ahead and use the examples…

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>