hackinghat.com

Recently I have been using Common Lisp’s eval function a bit. Since it’s eval that put’s the E in REPL it fair to say that it is a fairly fundamental part of Lisp. However, no code that I have seen appears to use it directly. I think I know why. To make (eval …) always work in the way you’d expect doesn’t appear to be that intuitive.

Paul Graham in On Lisp, has this to say about using eval in your own code:

Generally it is not a good idea to call eval at runtime, for two reasons:

1. It’s inefficient: eval is handed a raw list, and either has to compile it on the spot, or evaluate it in an intepreter. Either way is slower than compiling the code beforehand, and just calling it.
2. It’s less powerful, because the expression is evaluated with no lexical context. Among other things, this means that you can’t refer to ordinary variables outside the expression being evaluated

And so when I discovered a need in my project to persist and reload closures I decided that my needs would not violate either of Paul’s points. Firstly, because I don’t know what the code I’m going to persist is going to be, and secondly because no lexical context is needed to create my closures. Therefore, I would store them as strings and then I would use read, and eval to restore them. This worked fine so I put the code into the package and declared my work done.

It turned out that once the code was in a package it didn’t really work as I’d intended. When I tried to run it I got unknown symbol conditions raised when I tried to restore the closures. Qualifying all the symbols with the correct package name worked but it made my shiny new DSL all messy by requiring me to always prefix all my symbols. It turns out then that eval doesn’t work this way by design. The reason is because of this statement in the HyperSpec page of eval.

…
Description:

Evaluates form in the current dynamic environment and the null lexical environment .
…

I was expecting that since my eval was inside a package it would be able to see function symbols in that package. Not so, eval works in the dynamic environment, which implies then that the current package is a special variable and hence part of the dynamic environment.

This means my code could only ever work when the current package is the library package. Any other package and the code fails because eval is checking the dynamic environment to determine which symbols are visible without package qualifiers. Indeed it seems that, in SBCL to make my code work in the way that I should expect I need to wrap it in the following:

   (let ((*package* (find-package "MY-PACKAGE")))
      (eval ...))

And this works just fine. The most pleasing thing about this outcome is that it illuminated a point that I’d heard before but never been able to substantiate: Lisp, It Doesn’t Get In Your Way. eval has to work how it does otherwise Common Lisp would probably not work properly. However, because the package system is an accessible part of the language to the programmer it seems as if I can adapt any part of that system to suit my purpose.

You’d be right in thinking too much of this sort of thinking is bad for maintainability, but this single line hack allows me to safely persist executable-code at run time. Since there’s few languages that have closures to begin with, making a minor hack to make them easily persisted too (with the help of macro) seems a small price to pay.

“Lisp. It Doesn’t Get In Your Way. Much.” - I like the phrase so much I think I’m going to trade-mark it.

Here’s an interesting quote from Jeff Attwood (emphasis is mine):

“I use implicit variable typing whenever and wherever it makes my code more concise. Anything that removes redundancy from our code should be aggressively pursued — up to and including switching languages.”

It sounded like something I’d heard before. Here’s another quote from Paul Graham:

“The kind of dirtiness Arc seeks to avoid is verbose, repetitive source code. The way you avoid that is not by forbidding programmers to write it, but by making it easy to write code that’s compact.”

I’m not in anyway claiming that Jeff is lacking originality I’m suggesting that when two influential people with a large audience express the same thought … well something ought to happen right? Perhaps programming languages are going to become more expressive and concise. This certainly seems to be one of Paul’s aims at least.

Programming is changing, that’s for sure. It seems like only yesterday that Python was new and I was bending my head around it and liking being released from static types and embracing dynamic languages. Indeed if you watch this video (and I highly recommend you do) you can get a feel for how far Python itself has come in about 5 minutes.

The point though is our industry is still evolving. Natural selection finds the best parts of all those programming languages and software products and begats them into new ones. 10 years ago I believed that the future had arrived and that we would all use C++ where performance mattered and Java everywhere else. It seems that I was hopelessly naive to believe that evolution had ended. It had scarcely even begun.

I’m also eternally grateful to the innovators and early adopters that I don’t have to write much of either C++ or Java anymore. But that’s another story.

The Heathrow Terminal 5 story is not all bad. Yes it was a bit of a shambles, yes senior management was fired. But today I found some joy in BA’s misery.

Now don’t get me wrong this isn’t just me deriving pleasure from others misfortune. Although admittedly as a Brit I am innately very good at that. So good in fact that it’s a constant surprise that the Germans managed to invent a word for what is typically a British malaise: schadenfreude.

No, the silver lining of BA’s lead balloon is that T5 has become a common intellectual currency. It’s failure has so clearly underlined the pitfalls of not doing enough testing that I heard T5 being used as an analogy in a recent implementation meeting. A.N.Other said:

“I would not be happy committing to that deadline if we had to cut testing. The last thing I want is for this to become another T5 …”

Nothing gives a better fuzzy feeling than completing a long testing phase. However if testing is getting squeezed out then you have to get management to agree to extending the deadline before you’ve actually reached that deadline. Indeed cutting testing is to invite what Steve Connell has coined as “Wishful Thinking”, and is the 13th classic mistake of software project management:

Wishful thinking isn’t just optimism. It’s closing your eyes and hoping something works when you have no reasonable basis for thinking it will. … It undermines meaningful planning and may be at the root of more software problems than all other causes combined.

Amen.

One recurring theme I have noticed with users of systems I’ve worked on, is that they aren’t nearly as stupid as I think. They often try to make sense of the system thrust in-front of them. This is mostly out of necessity since it stands between them and them doing their job, and so they need to make sense of it. Having written some truly awful systems myself I wish all of them the very best of luck.

Another, seemingly unrelated observation, is that business people are truly astounded, and often suspicious, of how long it takes to provide a solution to a particular problem. Writing software is simply hard so that partly explains it. Sometimes however some of the solutions that are asked for can come quickly. This might happen if the system was expressly designed to handle new cases of the particular solution being requested or if producing the solution requires little more than a configuration or script change. Or it might just be dumb luck that the release cycle has worked in their favour.

Disconnect

This disconnect between implementation times, with no apparent reason to the business user can cause problems. Sometimes it feeds the suspicion that they are being ‘had’ in some elaborate con:

“If change ‘x’ takes a week then surely change ‘y’ should take half as long. How could it not? It only takes half as many words to say out loud. Those guys in IT need firing.”

When that business person is a manager it can lead to awkward situations for developers:

“Change ‘x’ took a week, change ‘y’ will take half as long. How can it not? It’s the only thing that stands between us and product success. If it doesn’t I’ll fire those IT guys”.

The simple truth is that unless a business person has also the developer’s view of the system they will not be able to make sound judgements about it. Hell, I have a developers view and not even my judgements are particularly sound.

However humans are pretty adaptable creatures, and rather than telling them the answer we should explain the answer in a way that they can understand. If they want to listen then educating them has a few potential benefits. For one thing it might make you look like you care about your users, rather than being that IT jerk who steals everyone’s food from the refrigerator. However, if you get your point across without sounding (to them) like a lunatic then you might improve their mental model of how the system actually works.

Breed

There is a breed of programmer out their in the world today that has either evolved or engineered themselves into a situation where they are the only one who ‘knows’. Yes, you know who you are. Sometimes they do this as a survival instinct to make themselves indispensable, sometimes because they’re not great communicators or educators. These are the people that need to be fired because their value is way-less than they think it is. They actually harm the productivity of the company by being obstructive or uncommunicative, plus they’re a real pain-in-the-ass to work with.

Yes you will need to keep a watchful eye on your newly educated fledglings. Especially the managers, but there’s nothing knew about that.

A minor thought for today. Part of becoming an expert in any programming language is learning the idioms. Through experience, research and your peers you discover what makes your programming language hum. How to get the best performance and how to trade that against the best expressiveness. Since spoken language affects the way that what we think it’s reasonable to expect that programming languages affect the way we think about programming problems.

I have on a number of occassions wanted to write an expression in Lisp that will return true if all the elements are T and nil if any are not T. A sort of ‘and’ for a list. It seemed like a classic case of a ‘reduce’ or fold operation to me and so I went about doing that as a starting point.

CL-USER> (reduce #'(lambda (x y) (and x y)) '(t t t t t t t))
T

Hmm, because AND is a macro it can’t be an argument to reduce therefore I have to wrap it in a lambda. Which is ok but it removes the short-circuit feature from the AND.

So, I wasn’t terribly excited by this result but it served the purpose and life continued as normal. That was until today when I discovered the LOOP tutorial which inspired me with this trivial alternative …

CL-USER> (loop for x in '(t t t t t t) always (eq t x))
T

Yes it’s a little bit less functional but it it’s very clear what’s going on here and that was my original problem with the second reduce form. I was interested to see how the LOOP expanded too, so:

CL-USER> (macroexpand-1 '(loop for x in '(t t t t t t) always (eq t x)))
(BLOCK NIL
  (LET ((X NIL) (#:LOOP-LIST-1613 '(T T T T T T)))
    (DECLARE (TYPE LIST #:LOOP-LIST-1613))
    (SB-LOOP::LOOP-BODY NIL
                        (NIL
                         (SB-LOOP::LOOP-REALLY-DESETQ X (CAR #:LOOP-LIST-1613))
                         NIL
                         (SB-LOOP::LOOP-REALLY-DESETQ #:LOOP-LIST-1613
                                                      (CDR #:LOOP-LIST-1613)))
                        ((UNLESS (EQ T X) (RETURN-FROM NIL NIL)))
                        ((WHEN (ENDP #:LOOP-LIST-1613) (GO SB-LOOP::END-LOOP))
                         (SB-LOOP::LOOP-REALLY-DESETQ X (CAR #:LOOP-LIST-1613))
                         NIL
                         (SB-LOOP::LOOP-REALLY-DESETQ #:LOOP-LIST-1613
                                                      (CDR #:LOOP-LIST-1613)))
                        ((RETURN-FROM NIL T)))))
T
CL-USER> 
… Jeeeeezus.    No way that can be efficient surely?  So I ran both arguments with a temporary list of 1,000,000 elements all equal to true:
CL-USER> (time (reduce #'(lambda (x y) (and x y)) (loop for i below 1e6 collect t)))
Evaluation took:
  0.149 seconds of real time
  0.100006 seconds of user run time
  0.040002 seconds of system run time
  [Run times include 0.068 seconds GC run time.]
  0 calls to %EVAL
  0 page faults and
  7,999,488 bytes consed.
T
CL-USER> (time (loop for x in (loop for i below 1e6 collect t) always (eq t x)))
Evaluation took:
  0.059 seconds of real time
  0.052003 seconds of user run time
  0.0 seconds of system run time
  0 calls to %EVAL
  0 page faults and
  7,995,392 bytes consed.
T
Guess again brother.   The more I thought about it the more other solutions I could find to this problem.   For instance I could just count all the nulls and if I find one then the answer is false.
(eq 0 (count-if #'null
           (loop for i below 1e6 collect t)))


This was marginally faster than the ‘loop’ on my implementation, but then I reasoned that on a list of a million items  ‘AND’ should really retain that short-circuit ability to make it stop on the first false.   So perhaps the correct thing to do is return the first non-true argument:  
(not (find-if #'null
          (loop for i below 1e6 collect t)))
This solution is less imperative in style and less complicated than all the other solutions.  Now that I’ve thought of it I can’t imagine how I didn’t think of it sooner.   A winner!
Lisp is such an old language that there seems literally dozens of ways to attack most problems and that really is the point.   Since I don’t know what I don’t know there’s probably dozens more ways I could try to solve this trivial problem that only experience, research and interaction with my peers will get me.      
This is where books like the O’Reilly cookbook series should come in right?  They should have all that hard work done for me so I can stop making mistakes and start making systems.    Books, however, are not very easily searchable when they are on the shelf.   Therefore on-line resources are best for searching for optimal solutions.  So I was pleased to find that Common Lisp has an online cookbook.   However you still have to KNOW where to look in a resource like this because I might not have found the ‘find-if’ solution if I was stuck in a rut banging on the ‘reduce’ solution.    Not only that, every situation is different, ‘reduce’ definitely has its uses but perhaps not in this case.  
This further suggests to me that it’s only your peers and your own research (i.e. trial and error) that can show you the way.  Just make sure you give yourself enough time for the research and pick your peers carefully