compilation with LispWorks 7

[lispm]

There are minor issues.

The GC statements in shop3.lisp and explicit-search.lisp won't be executed, since lispworks is missing in #+(or ccl allegro sbcl clisp abcl ecl) . Similar for cmucl.

In minimal-subtree/package.lisp there is a duplicate package nickname .

[rpgoldman]

Probably for the GC, we should replace this with use of trivial-garbage, instead of redoing that work ourselves. I'll try to get to that soon.

And I will fix the issue in minimal-subtree/package.lisp; thank you for catching that.

FIXED

[lispm]

LispWorks can compile the code and tests (though IIRC, there are some errors later when running tests), but it will ask several times to enlarge the stack space. Thus it might be useful to enhance the stack under LispWorks upfront - then the compilation will work without interruption.

[lispm]

There are a bunch of dependencies I had to provide for compilation. Not sure if they are documented. Fiveam, pddl stuff, ...

The tests get compiled, but there was no plan validator command callable.

[lispm]

Probably for the GC, we should replace this with use of trivial-garbage, instead of redoing that work ourselves. I'll try to get to that soon.

How useful is it nowadays to explicitly call the GC?

[rpgoldman]

LispWorks can compile the code and tests (though IIRC, there are some errors later when running tests), but it will ask several times to enlarge the stack space. Thus it might be useful to enhance the stack under LispWorks upfront - then the compilation will work without interruption.

Would you please send me instructions for doing the stack space adjustment? If you do, I will be more than happy to put them in the test scripts, but you or someone else with a LispWorks license would have to test them.

Thank you for your many corrections and reports. I have added issues #4 and #5 to track the other issues separately.

[rpgoldman]

There are a bunch of dependencies I had to provide for compilation. Not sure if they are documented. Fiveam, pddl stuff, ...

The tests get compiled, but there was no plan validator command callable.

See #6

[rpgoldman]

Probably for the GC, we should replace this with use of trivial-garbage, instead of redoing that work ourselves. I'll try to get to that soon.

How useful is it nowadays to explicitly call the GC?

I believe it is still useful, especially when running the full test suite. It's also useful to minimize the amount of GC time when doing timed experiments.

[lispm]

Many implementations have generational GCs with more complex behavior. One might GC all generations beforehand, but on modern computers with multi GB RAM and fast memory, the difference is only mostly cosmetic I would guess. Generally I stopped looking at the GC performance for most Lisp applications. The most interesting thing I usually take from the GC is the amount of allocation that has been done for some computation -> that might give sometimes hints at inefficient code with lots of consing.

[lispm]

Would you please send me instructions for doing the stack space adjustment

I'll try that out in the next days. One can instruct LispWorks to automatically extend the stack, adjust the stack upfront to a certain size or even let the stack grow upto some limit.

Automatically extending the stack might be useful - but then this might have drawbacks with code which grows the stack indefinitely. Probably better to start with a stack size, which is large enough for compilation and testing. By default on a stack overflow the user can increase the stack manually via an error restart.

[lispm]

I managed to run shop3 on LispWorks 7.1.1 with the test suite:

 Did 522 checks.
    Pass: 522 (100%)
    Skip: 0 ( 0%)
    Fail: 0 ( 0%)

One way to avoid the stackoverflows is:

(setf system:*stack-overflow-behaviour* :warn)

Above will print a short notice about a resize, but then will continue automatically.

One can also set it to NIL. Then the stack resize will be silently done.

[rpgoldman]

So I propose to add this to the command in jenkins/run-tests.sh. See upcoming pull request.

[rpgoldman]

Rainer, will you try pull request #16 ?

To use it, you would need to set the environment variable LISPWORKS to point to a command-line version of lispworks on your machine (you might have to create one by dumping an image if you don't have one), and then just do

./run-tests.sh lispworks

in that directory. Assuming I didn't mess up the shell script, which I might very well have done.

[lispm]

Hi I'm not sure if I do the correct thing.

When I clone the repository. I have to update the submodules.

I do:

cd shop3/
git submodule init
git submodule update

Anything else I need to do?

Btw., the ASDF is quite old.

I don't try to use LispWorks yet (I have a command line version, which starts a threaded version of LispWorks).

I do for now:

sh run-tests.sh sbcl

The resulting error is:

+ sbcl --dynamic-space-size 4000 --noinform --end-runtime-options --no-userinit --no-sysinit --disable-debugger --eval '(load "build-shop3.lisp")'
While evaluating the form starting at line 190, column 0
  of #P"/Lisp/shop3a/shop3/shop3/shop3.asd":
While evaluating the form starting at line 18, column 0
  of #P"/Lisp/shop3a/shop3/jenkins/build-shop3.lisp":
Unhandled MISSING-COMPONENT in thread #<SB-THREAD:THREAD "main thread" RUNNING {10005205B3}>: Component "fiveam-asdf" not found

Any idea how to prevent that?

Btw.:

I see a line like

export CL_SOURCE_REGISTRY="${SHOP3DIR}:${THISDIR}//"

should it be this

export CL_SOURCE_REGISTRY="${SHOP3DIR}/:${THISDIR}/"

??

[rpgoldman]

When I clone the repository. I have to update the submodules.

I do:

cd shop3/
git submodule init
git submodule update

That should do it.

Anything else I need to do?

Did you do

git checkout lispworks-test-tweak

to get the branch with ./run-tests.sh updated for lispworks? That was just a guess about how it should be fixed, based on your earlier messages, since I don't have access to lispworks.

Btw., the ASDF is quite old.

I'm confused by this -- I don't have an ASDF directory anywhere in my repo, and it's not listed in my .gitmodules file, so I'm not sure why you have an ASDF there.

I don't try to use LispWorks yet (I have a command line version, which starts a threaded version of LispWorks).

I do for now:

sh run-tests.sh sbcl

I have been doing ./run-tests.sh (which defaults to SBCL). I'm not knowledgeable enough about the shell to know what the difference is between these two means of invoking the script. But I don't see the error you see when I try your invocation of the script.

The resulting error is:

+ sbcl --dynamic-space-size 4000 --noinform --end-runtime-options --no-userinit --no-sysinit --disable-debugger --eval '(load "build-shop3.lisp")'
 While evaluating the form starting at line 190, column 0
   of #P"/Lisp/shop3a/shop3/shop3/shop3.asd":
 While evaluating the form starting at line 18, column 0
   of #P"/Lisp/shop3a/shop3/jenkins/build-shop3.lisp":
 Unhandled MISSING-COMPONENT in thread #<SB-THREAD:THREAD "main thread" RUNNING {10005205B3}>: Component "fiveam-asdf" not found

Any idea how to prevent that?

Btw.:

I see a line like

export CL_SOURCE_REGISTRY="${SHOP3DIR}:${THISDIR}//"

should it be this

export CL_SOURCE_REGISTRY="${SHOP3DIR}/:${THISDIR}/"

The form I have, with the two final slashes, means that ASDF will search recursively below THISDIR instead of just searching THISDIR itself. So that is correct.

I think to see what's wrong it would help for you to

1. Send me a list of the jenkins/ext/ directory. This is what I have in there:

alexandria/  asdf-flv/  fiveam/  fiveam-asdf/  iterate/  pddl-tools/  sb-texinfo/  trivial-backtrace/  trivial-garbage/

2. Then check and make sure you don't have any empty subdirectories in jenkins/ext/. Git submodules are so confusing to interact with; maybe something didn't get populated properly.

If you can let me know about that, I will try to help you get this fixed. Sorry for the issues, thanks for being an early adopter and for sending the many helpful suggestions (I'm testing your fix to READ right now).

[lispm]

I'm confused by this -- I don't have an ASDF directory anywhere in my repo, and it's not listed in my .gitmodules file, so I'm not sure why you have an ASDF there.

I mean there is 3.2.1 asdf.lisp in the jenkins directory. It is older than current Lisps will have installed and older than the current version 3.3.3.

[rpgoldman]

You are right! I overlooked this months ago when I was updating the test scripts, and it was never loaded. I will remove it now. Thank you!

[lispm]

updated for lispworks

yes, I used that. But I wanted to get the script going with sbcl, since that is supported. When that works, I'll continue trying to get the script going with LispWorks.

Sorry for the issues, thanks for being an early adopter and for sending the many helpful suggestions

Hope it helps to get other users going.

I read the paper about shop3 and the improvements over shop2. Very impressive!

[lispm]

Seems like I am getting the run-tests.sh script working for sbcl under Ubunto 18.04 Linux.

git clone https://github.com/shop-planner/shop3.git
cd shop3
git submodule init
git submodule update
git checkout lispworks-test-tweak

then going into the jenkins/VAL directory

fixing the Makefile with

%.cpp : %.yacc
	${LEX} -+ -o src/Parser/lex.yy.cc src/Parser/pddl+.lex; ${YACC} $< -o src/pddl+.cpp

then running make

make

then going back to the jenkins directory

./run-tests.sh

Tests are executing right now.

[rpgoldman]

Great. If you can get lispworks to go, please let me know about the test results. I'd like to release a bugfix 3.0.1, but it would be good to get any fixes required by lispworks in place.

[rpgoldman]

If you get a chance, please try master with the new trivial-garbage dependency included (or it is available from Quicklisp). You will need to ensure your submodules are properly synced and populated before testing. Thanks!

[lispm]

Hi, I see the master. The code looks like this:

(defun find-plans (problem
                   &key domain (which *which*) (verbose *verbose*)
                        (gc *gc*) 
...
)
...
  #+(or ccl allegro sbcl clisp abcl ecl)
  (when gc
    (trivial-garbage:gc :full t))
...

This means that any other implementation which is not in the #+(or ccl allegro sbcl clisp abcl ecl) list will warn about an unused variable gc. Because GC is only used inside the conditional. If it would be LispWorks or CMUCL, then the variable GC would not be used in the source, since the reader removes the whole expression.

[rpgoldman]

Thank you for bringing that to my attention. I believe it might be a merge error. I will fix it and re-test.

[lispm]

Just declaring GC to be ignorable (can be ignore, but does not need to) removes the warning.

[lispm]

Great. If you can get lispworks to go, please let me know about the test results. I'd like to release a bugfix 3.0.1, but it would be good to get any fixes required by lispworks in place.

I have tried it, but have failed so far to get it to work with the script as such. There are two problems:

• the form to set the stack enlargement policy does not get transferred correctly via the lispworks call in the script. I have yet to find an easy way to fix that.

• I have to check also whether using no init file (?) when calling lispworks from the script is actually a good idea. I think it then for example might show the behavior of failing to compile files where it has a compilation warning. Something which I usually have just set to warn, but continue to compile.

I will get it sorted out, but it needs some more time.

Thanks for your patience!

[rpgoldman]

Thank you, Rainer. If I recall correctly, I put the stack enlargement instruction in an --eval argument in run-tests.sh. You might try moving it into the lisp test files (e.g., test-shop3.lisp) protected with a #+lispworks. Possibly also with (eval-when (:load-toplevel :execute))

There is supposed to be some control over the handling of warnings in ASDF, but I have never figured it out to my satisfaction, and I'm not sure it works on all the lisps.

[lispm]

For experimenting: In jenkins/build-shop3.lisp

(cond
  (*build-error*
   (uiop:die 1 "SHOP3 build failed with an error: ~a.~%" *build-error*))
;  (*build-warning*                                                                ; RAJ                                                                                                                                                                     
;   (uiop:die 2 "SHOP3 build failed with warnings:~%~{~t~a~%~}" *build-warning*))                                                                                                                                                                            
  (t
   (format t "SHOP3 build successful.~%")
   (uiop:quit 0)))

I think it is okay not to fail on build warnings. LispWorks signals some minor warnings and these would make the build to fail.

[rpgoldman]

I like to keep the build clean because I have worked on projects where warnings were allowed to build up and the warnings that did not matter hid the warnings that did.

By all means try withe warnings off, though. But please keep sending me the warnings so that I can inspect them and fix them.

[rpgoldman]

Build fixed with #46. Closing. Reopen or open a new ticket if you see other problems.