API: an approach to create descriptors

[ruv]

Recognizer v4 API proposes to create a descriptor (that it calls "rectype") by providing xt for the interpreting action, for the compiling action and for the reproducing action.

At least the compiling and reproducing actions should refer each component of a token. So, in the worst case we have three actions and have to thrice refer each component of the token.

An alternative approach is to automatically generate any required actions from the description of the components of a token, and perhaps only two actions: interpreting and compiling (see also the comment).

Example:

\ Create the token descriptor `td-3lit`
\ for a token `( xd x )` that is a tuple of two tokens: `xd` and `x` 
td-2lit td-lit 2 descriptor constant td-3lit

\ or alternatively
td-2lit td-lit 2 descriptor{ td-3lit }

One obvious advantage of this approach is simplicity of usage (less code): we should refer each component of a token only once.

The second advantage: we don't have to choose between the reproducing and postponing actions in the API.

Another advantage is that we can also generate an action for another useful method: drop-token. This method can also help to easily define recognizable word without catch/throw:

: recognizable ( c-addr u -- flag )
  recognize ( i*x td | 0 )
  dup if drop-token true then
;

Also, by design, all parts of a token in a standard program are located in the data stack and the floating-point stack only. No way to describe a token that is not a tuple of other already known tokens.

Disadvantage of this approach is some complexity of implementation.