Inlining closures / forms with free variables

[digikar99]

(in-package :specialization-store)
(defstore foo (a))
(let ((a "hello"))
  (defspecialization (foo :inline t) ((str string)) t
    (declare (ignore str))
    a))
(defun foo-caller (b)
  (declare (type string b))
  (foo b))

results in

; caught WARNING:
;   undefined variable: SPECIALIZATION-STORE::A
; 
; compilation unit finished
;   Undefined variable:
;     A
;   caught 1 WARNING condition

A solution suggested here was the use of code-walkers. A discussion about code-walkers is up at https://www.reddit.com/r/lisp/comments/itf0gv/determining_freevariables_in_a_form/

[markcox80]

Adding a code walker to specialization store is of no use here because the defspecialization macro function cannot obtain the containing form i.e. the top level let form.

The best I can do is attempt to compile the body of the specialization in a null lexical environment. That way warnings will be emitted when the specialization is defined rather than when it is used.

[digikar99]

That does seem like a lean solution, not sure if it has other limitations

[markcox80]

I have spent some time trying to figure out the best way to fix this without using a code walker.

Compiling the following using compile-file produces half decent error messages.

;;;; example.lisp

(eval-when (:compile-toplevel :load-toplevel :execute)
  (defmacro example ()
    (compile nil '(lambda () free-variable))))

(example)

;; (compile-file "example.lisp")
;; ; compiling file "/tmp/example.lisp" (written 21 SEP 2020 06:58:09 AM):
;; ; processing (defmacro example ...)
;; ; processing (example); in: lambda ()
;; ;     (LAMBDA () FREE-VARIABLE)
;; ;
;; ; caught warning:
;; ;   undefined variable: common-lisp-user::free-variable
;; ;
;; ; compilation unit finished
;; ;   Undefined variable:
;; ;     free-variable
;; ;   caught 1 WARNING condition

;; ; wrote /tmp/example.fasl
;; ; compilation finished in 0:00:00.009

However, very cryptic messages are shown when using Slime's C-c C-k

;;; Contents of *slime-compilation* buffer.
cd /tmp/
1 compiler notes:

*slime-source*:1:1:
  warning: undefined variable: common-lisp-user::free-variable

Compilation failed.

;;; REPL output
; compiling file "/tmp/example.lisp" (written 21 SEP 2020 07:00:43 AM):
; in: lambda ()
;     (LAMBDA () FREE-VARIABLE)
; 
; caught warning:
;   undefined variable: common-lisp-user::free-variable
; 
; compilation unit finished
;   Undefined variable:
;     free-variable
;   caught 1 WARNING condition

; wrote /tmp/example.fasl
; compilation finished in 0:00:00.003

Delaying the check until load time produces cryptic messages as well.

;;; example2.lisp

(eval-when (:compile-toplevel :load-toplevel :execute)
  (defmacro example ()
    `(progn
       (compile nil '(lambda () free-variable)))))

(example)

;; (compile-file "example2.lisp")
;; ; compiling file "/tmp/example2.lisp" (written 21 SEP 2020 06:50:54 AM):
;; ; processing (defmacro example ...)
;; ; processing (example)

;; ; wrote /tmp/example.fasl
;; ; compilation finished in 0:00:00.005

;; (load "example.fasl")
;; ; in: lambda ()
;; ;     (LAMBDA () FREE-VARIABLE)
;; ;
;; ; caught warning:
;; ;   undefined variable: common-lisp-user::free-variable
;; ;
;; ; compilation unit finished
;; ;   Undefined variable:
;; ;     free-variable
;; ;   caught 1 WARNING condition

Neither of these options are better than the error reporting for the following

;;;; example3.lisp
(eval-when (:compile-toplevel :load-toplevel :execute)
  (defmacro example ()
    'free-variable))

(defun foo ()
  (example))

;; Contents of *slime-compilation* buffer.
cd /tmp/
1 compiler notes:

example3.lisp:7:3:
  warning: undefined variable: common-lisp-user::free-variable

Compilation failed.

;; (compile-file "example3.lisp")
;; ; compiling file "/tmp/example3.lisp" (written 21 SEP 2020 07:06:56 AM):
;; ; processing (defmacro example ...)
;; ; processing (defun foo ...)

;; ; file: /tmp/example3.lisp
;; ; in: defun foo
;; ;     (EXAMPLE)
;; ;
;; ; caught warning:
;; ;   undefined variable: common-lisp-user::free-variable
;; ;
;; ; compilation unit finished
;; ;   Undefined variable:
;; ;     free-variable
;; ;   caught 1 WARNING condition

;; ; wrote /tmp/example3.fasl
;; ; compilation finished in 0:00:00.007

Adding a code walker dependency to specialization store for the purposes of detecting this error is not something I am willing to do. I am inclined to not do anything.

[markcox80]

It appears I have been hasty.

Firstly, the *slime-source* mentioned in the *slime-compilation* output above is actually a reference to an Emacs buffer with the name *slime-source*. This contains the form that failed to compile. I didn't know about that.

Unfortunately, it doesn't specify where the code actually came from. I can introduce a hint by introducing a macro which does nothing.

Consider compiling the following example:

;;;; example4.lisp

(eval-when (:compile-toplevel :load-toplevel :execute)
  (defmacro defspecialization-body-marker (store specialization-lambda-list)
    (declare (ignore store specialization-lambda-list))
    nil)

  (defmacro example ()
    (compile nil '(lambda ()
                   (defspecialization-body-marker 'hey-there 'example)
                   free-variable))))

(example)

;; (compile-file "example4.lisp")
;; ; compiling file "/tmp/example4.lisp" (written 21 SEP 2020 08:15:40 AM):
;; ; processing (defmacro defspecialization-body-marker ...)
;; ; processing (defmacro example ...)
;; ; processing (example); in: lambda ()
;; ;     (LAMBDA () (DEFSPECIALIZATION-BODY-MARKER 'HEY-THERE 'EXAMPLE) FREE-VARIABLE)
;; ;
;; ; caught warning:
;; ;   undefined variable: common-lisp-user::free-variable
;; ;
;; ; compilation unit finished
;; ;   Undefined variable:
;; ;     free-variable
;; ;   caught 1 WARNING condition

;; ; wrote /tmp/example4.fasl
;; ; compilation finished in 0:00:00.019


;;; Slime's C-c C-k

;; Buffer *slime-compilation*
;;
;; cd /tmp/
;; 1 compiler notes:
;;
;; *slime-source*:1:1:
;;   warning: undefined variable: common-lisp-user::free-variable
;;
;; Compilation failed.


;; Buffer *slime-source*
;;
;;   (LAMBDA () (DEFSPECIALIZATION-BODY-MARKER 'HEY-THERE 'EXAMPLE) FREE-VARIABLE)

;; REPL output
;; ; compiling file "/tmp/example4.lisp" (written 21 SEP 2020 08:18:48 AM):
;; ; in: lambda ()
;; ;     (LAMBDA () (DEFSPECIALIZATION-BODY-MARKER 'HEY-THERE 'EXAMPLE) FREE-VARIABLE)
;; ;
;; ; caught warning:
;; ;   undefined variable: common-lisp-user::free-variable
;; ;
;; ; compilation unit finished
;; ;   Undefined variable:
;; ;     free-variable
;; ;   caught 1 WARNING condition

;; ; wrote /tmp/example4.fasl
;; ; compilation finished in 0:00:00.014

I think this improves the error reporting a lot.

[markcox80]

Unfortunately, this idea isn't going to work. The error messages are either truncated or are produced after macro expansion, and thus the marker isn't included in the message.

[digikar99]

Wait! What's wrong with this?

CL-USER> (with-output-to-string (*error-output*)
           (compile nil '(lambda () a)))
"; in: LAMBDA ()
;     (LAMBDA () A)
; --> MULTIPLE-VALUE-PROG1 
; ==>
;   (PROGN A)
; 
; caught WARNING:
;   undefined variable: COMMON-LISP-USER::A
; 
; compilation unit finished
;   Undefined variable:
;     A
;   caught 1 WARNING condition
"

[markcox80]

That outputs an error message but it will be buried deep among a lot of other compiler messages.

What I want is something which integrates with the standard machinery which handles compiler warnings/errors. The key issue I faced is that the behaviour of the function compile changes if it is invoked inside compile-file or with-compilation-unit as shown below:

(defun compile-test ()
  (print (list "First" (multiple-value-list (compile nil '(lambda () free-variable)))))
  (with-compilation-unit (:override t)
    (print (list "Second" (multiple-value-list (compile nil '(lambda () free-variable)))))))

cl-user> (compile-test)
; in: lambda ()
;     (LAMBDA () FREE-VARIABLE)
; 
; caught warning:
;   undefined variable: common-lisp-user::free-variable
; 
; compilation unit finished
;   Undefined variable:
;     free-variable
;   caught 1 WARNING condition

("First" (#<function (lambda ()) {52E1BA9B}> t t)) 
("Second" (#<function (lambda ()) {52E1BCBB}> nil nil)) ; in: lambda ()
;     (LAMBDA () FREE-VARIABLE)
; 
; caught warning:
;   undefined variable: common-lisp-user::free-variable
; 
; compilation unit finished
;   Undefined variable:
;     free-variable
;   caught 1 WARNING condition

The branch feature/inlining-free-variables is another go at attempting to capture the warnings and outputting a decent error message as well as resignal the warnings.

[digikar99]

Just tried out that branch. This does look okay.

The simplest thing I was thinking of was capturing the *error-output* in a string and the top-level (defspecialization ... form itself, and then may be, search for "Undefined variable" string within the captured error string, and then do the resignalling. Don't know if this qualifies as ugly.

[markcox80]

I have merged feature/inlining-free-variables in to master.

String parsing is not portable across implementations and can be a maintenance nightmare as the message may change from version to version.

Thanks for reporting the issue.