API: idempotency vs side effects of recognizers

[ruv]

Should a recognizer be free of side effects, or at least idempotent?

Recognizer v4 API proposes that a recognizer may have any side effects. E.g., trying to recognize a lexeme may produce read the input source.

This looseness reduces safety of using recognizers. Particularly, when a program or library uses the perceptor (the recognizer currently used by the Forth text interpreter) that was not set by this program or library. Or when a lexeme is taken from an unsafe source (e.g., from the user).

On the other side, any recognizer can be implemented in an idempotent manner.

For example, see an implementation of string literals in such a way that the recognizer doesn't do additional parsing — see at the gist.

Also, recognizing is a generalization of finding definition names and conversion strings to numbers (FIND and >NUMBER or NUMBER?). And these actions never have any side effects.

See also: Issue #8

[ruv]

Examples of programs that work incorrectly due to side effect of some recognizers — see in the message news:[email protected] (raw archived) "Recognizer protocol" on 2020-07-04.

E.g., the following short correct program:

'!' parse "test passed"! rec-string cr .s cr .

produces "Invalid memory address" error in Gforth 0.7.9_20200618 (see also the corresponding bug report).

See also: news:[email protected] (raw archived) "Recognizer protocol" on 2020-06-28.