Problem of unknown words in the search order

[ruv]

Problem

Let's consider some use case for the quote recognizer recognize-word-quoted, that allows to get the xt of a word using the syntax like 'X.

' recognize-word-quoted set-perceptor-after

\ use this feature
: x [...] ;
: foo 'x [...] ;

unset-perceptor

Can we face any problem with that? Yes, we can. If the forth-wordlist contains a word 'x , our lexeme "'x" will be resolved into this word and our recognize-word-quoted will not get control, and we will get unexpected binding.

So, the only solution is to set our recognizer before other:

' recognize-word-quoted set-perceptor-before
[...]
unset-perceptor

NB: Recognizer v4 API does not guarantee that a user-defined recognizer can get control before the system's recognizer of Forth words.

On the other hand, recognize-word-quoted (or any other) cannot be set before the recognizer of the Forth words in the default perceptor — for the sake of backward compatibility.

Actually, we already have this problem with numbers and characters ('c' syntax). But usually the default search order does not contains names that conflict with these syntaxes. Nevertheless, nothing may prevent a Forth system to have such words, — and it will not make the Forth system non-standard (at the moment).

Conclusion and ideas

1. The standard should be tighter regarding the default search order and the words that it may contain.

2. Perhaps, in addition to the "standard program" notion, the "standard module" notion should be introduced.

3. A module may grant only limited scopes to additional recognizers.

4. Perhaps some level of sand-boxing can be implemented to load third party library modules. Another approach is to just automatically revert the perceptor, the search order, the base, etc., after loading a module.