Artful Code

newLISP receives an unexpected level of hostility from lispers. Languages like Arc and newLISP share an enmity that stems from the assumption that these languages are in some way attempting to replace Common Lisp. This is not the case.

It should be pointed out that newLISP is an interpreted language with primary emphases on low memory usage, short start-up time, and efficient implementation. In that, newLISP is exceptionally successful; the amount of functionality that has been packed into an executable of just over 250 Kb is impressive. Automatic memory management is fast and simple and requires little overhead. In terms of speed, newLISP compares favorably to other interpreted languages (e.g. Perl and Python).

It also needs to be mentioned that newLISP is not Common Lisp. The name newLISP seems to indicate that newLISP modernizes and replaces Common Lisp. The FAQ on newLISP.org does take a decidedly populist tone with regard to other lisps. This is unfortunate, because newLISP is a very different language than CL or Scheme and is not truly in competition with them.

I’d like to discuss of a few of the more significant sticking points for potential users.

Macros

newLISP is a purely interpreted language. This has some important implications in its implementation. Macros are disimilar to compiled lisps. Most importantly, there is no compile-time expansion. All expansion is performed at runtime and with the overhead of a function all. In fact, newLISP macros are more like functions with lazy evaluation.

However, this does not completely diminish the power of newLISP macros. newLISP’s operator, letex, may be used to both expand and evaluate a block of code. Using the args function and implicit indexing, variable capture may be avoided in macros. Here is an extremely basic macro to iterate over the lines of a file using both letex and args:

(define-macro (dolines)
    (letex ((var (args 0 0))
                    (file (open (args 0 1) "read"))
                    (body (cons 'begin (rest (args)))))
        (while (set 'var (read-line file))
            body)
        (close file)))
 
(dolines (line "/path/to/file")
    (println line))

Variable capture

This is a real issue when writing newLISP. The function args can prevent capture by functions’ parameters, but there is no equivalent to Common Lisp’s gensym to dynamically create unique symbols. Fortunately, it is not difficult to write our own:

(define _gensym:_gensym)
 
(define (gensym:gensym (ctx MAIN))
    (if (_gensym (name ctx))
        (let ((new-sym (string "gensym-" (_gensym (name ctx) (+ 1 (_gensym (name ctx)))))))
                 (or (sym new-sym ctx) (gensym ctx)))
            (begin
                (_gensym (name ctx) 0)
                (gensym ctx))))

This demonstrates two other features of newLISP, contexts and dictionaries, which I will go into in more detail shortly. For now, it will suffice to explain that contexts are lexical namespaces (context symbols are typically denoted in the form context:symbol, similarly to CL packages) and dictionaries are a way to use contexts as an associative namespace.

Note that this implementation of gensym is dependent on the maximum integer value on the system. newLISP will wrap into negatives after the max integer, so the maximum number of symbols possible to create is generally twice the value of the max integer. However, symbols that have been deleted will be recycled.

Hash tables

newLISP does not use hash tables (see here for an explanation of why). This is a show-stopper for many. Without joining the (often heated) debate over this, newLISP does provide the same functionality as a hash through its dictionaries:

(define dict:dict) ; create the dictionary
(dict "foo" "bar") ; associates "foo" to "bar" in dict
(dict "foo") ; retrieves "foo" from dict

Or, programmatically:

(context dict "foo") ; => "bar"
(context dict "foo" "baz") ; sets foo:bar "baz"

Dynamic scope

newLISP is dynamically scoped. Because it is an interpreted language, much computation and memory is saved by avoiding the complexities of lexical scoping.

newLISP’s contexts do implement static scopes that can be used to create lexically enclosed modules and closures. Within the context, though, variable lookups are dynamic to the context. In the example above (gensym), _gensym is a context dictionary and gensym is a default functor (a function assigned to a symbol with the same name as its context is defined as a functor in newLISP). Within gensym:gensym, scope is dynamic to the gensym namespace, which is lexically isolated from the default namespace, labeled MAIN.

Garbage collection

newLISP uses a form of memory management called ORO (one reference only). From the newLISP documentation:

Memory management in newLISP does not rely on a garbage collection algorithm. Memory is not marked or reference-counted. Instead, a decision whether to delete a newly created memory object is made right after the memory object is created.

Empirical studies of LISP have shown that most LISP cells are not shared and so can be reclaimed during the evaluation process. Aside from some optimizations for primitives like set, define, and eval, newLISP deletes memory objects containing intermediate evaluation results once it reaches a higher evaluation level. newLISP does this by pushing a reference to each created memory object onto a result stack. When newLISP reaches a higher evaluation level, it removes the last evaluation result’s reference from the result stack and deletes the evaluation result’s memory object. This should not be confused with one-bit reference counting. ORO memory management does not set bits to mark objects as sticky.

newLISP follows a one reference only (ORO) rule. Every memory object not referenced by a symbol or context reference is obsolete once newLISP reaches a higher evaluation level during expression evaluation. Objects in newLISP (excluding symbols and contexts) are passed by value to other functions. As a result, each newLISP object only requires one reference.

The intuitive assumption is that this results in slow evaluation, but it does not. Nor is newLISP’s evaluation speed burdened by a garbage collector.

newLISP’s ORO rule forces newLISP to constantly allocate and then free LISP cells. newLISP optimizes this process by allocating large chunks of cell memory from the host operating system. newLISP will request LISP cells from a free cell list and then recycle those cells back into that list. As a result, only a few CPU instructions (pointer assignments) are needed to unlink a free cell or to re-insert a deleted cell.

The overall effect of ORO memory management is a faster evaluation time and a smaller memory and disk footprint than traditional interpreted LISP’s can offer. The lack of garbage collection in newLISP more than compensates for its high frequency of cell creation/deletion. Note that under error conditions, newLISP will employ a mark and sweep algorithm to free un-referenced cells.

Implicit indexing

Implicit indexing is syntactic sugar for indexed access to elements in a list:

(let ((lst '(1 2 3 4 5)))
    (lst 3))
=> 4

It also works for nested lists:

(let ((lst '(1 2 3 (4 5))))
    (lst 3 1))
=> 5

Prospective users often remark that this syntax breaks conventional semantics and makes it far more difficult to do meta-programming. The function nth exists in the language and may still be used for this purpose:

(let ((lst '(1 2 3 4 5)))
    (nth 3 lst))
=> 4

Concurrency and distributed computing

newLISP is not threaded. This in particular was a challenge for me. However, I have found that due to newLISP’s small size and quick start-up, forking new processes is quite painless. There are few programs where one would want to start hundreds or thousands of threads, and newLISP is able to launch a large number of processes in less space than a single instance of the Python interpreter.

Using the new Cilk-inspired API, concurrent programming is simple, cheap, and expressive. Additionally, there are few of the challenges associated with threaded programming.

newLISP also comes with built-in functions for distributed computing, permitting forms to be easily sent to other nodes over TCP/IP for evaluation:

(net-eval remote-server-ip expr-to-evaluate timeout response-handler)

Conclusion

newLISP packs a lot of power into its small size. Some of its more powerful built-in features include:

• a fast, simple concurrency API
• regular expressions (PCRE)
• native C library access
• XML parsing
• pattern matching
• HTTP networking
• sockets
• cross platform GUI server (using Java)
• bayesian training and probability analysis
• cross platform support – newLISP uses only standard libraries

newLISP does not deserve the pariah status to which many relegate it. While newLISP cannot replace a compiled language for the most intensive tasks, it remains a fun, artful language and is excellent for exploratory programming and rapid prototyping, while remaining fast and powerful enough for the final product.