Infrastructure for enso --docs option & signature generator

[JaroslavTulach]

Pull Request Description

Implements --docs flag to generate documentation of a library into its docs/api directory. The --docs flag can take an optional argument - the format of the documentation. The default value is md that invokes (not yet finished) DocsEmitMarkdown generator. Finishing that generator is a work for the future.

However there is another DocsEmitSignatures generator that can be invoked via --docs=api. It ignores all documentation and just emits info about visible elements and their signatures. Sample output for can be found below.

Arguments have their name, followed by : and their fully qualified type when specified. Arguments may be prefixed by ~ indicating they are suspended. Arguments may contain trailing = indicating there is a default value for such an argument.

Important Notes

Use following command:

sbt:enso> runEngineDistribution --docs=api --in-project ./built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Base/0.0.0-dev/

to generate signature snapshot for Standard.Base.

My plan is to use this infrastructure to signature check the public elements of Meta when #12031 is to be integrated.

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

• All code follows the
  Scala,
  Java,
• Unit tests written

[JaroslavTulach]

I believe this PR provides good enough infrastructure for generating the documentation for a library. Use:

sbt:enso> runEngineDistribution --docs --in-project ./built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Base/0.0.0-dev/

Please pick the PR up and cultivate the content of .md files! CCing @jdunkerley, @Akirathan

[enso-bot]

Jaroslav Tulach reports a new STANDUP for yesterday (2024-06-15):

Progress: - infrastructure to generate documentation: #10291

• negative review of removeContextLock: Fix context lock removal #10273 (review) It should be finished by 2024-06-21.

[JaroslavTulach]

• the goal of this PR is to help signature testing of Meta
• once this PR is integrated, Reducing helper methods in Standard.Base.Meta #12031 will be modified to use it
• there will be a test that generates --docs=api documentation for Meta
• and verifies the generated content is equal to expected content
• right now the signatures of Meta in Reducing helper methods in Standard.Base.Meta #12031 are as follows:

## Enso Signatures 1.0
## module Standard.Base.Meta
- type Type
    - constructors self -> (Standard.Base.Data.Vector.Vector Standard.Base.Meta.Constructor)
    - methods self -> Standard.Base.Data.Vector.Vector
    - qualified_name self -> Standard.Base.Data.Text.Text
    - name self -> Standard.Base.Data.Text.Text
    - find qualified_name:Standard.Base.Data.Text.Text -> Standard.Base.Any.Any
- type Atom
    - value self -> Standard.Base.Any.Any
    - fields self -> (Standard.Base.Data.Vector.Vector Standard.Base.Any.Any)
    - constructor self -> Standard.Base.Meta.Constructor
- type Constructor
    - value self -> Standard.Base.Function.Function
    - fields self -> (Standard.Base.Data.Vector.Vector Standard.Base.Data.Text.Text)
    - name self -> Standard.Base.Data.Text.Text
    - new self fields:(Standard.Base.Data.Vector.Vector|Standard.Base.Data.Array.Array) -> Standard.Base.Any.Any
    - declaring_type self -> Standard.Base.Meta.Type
- type Primitive
    - value self -> Standard.Base.Any.Any
- type Unresolved_Symbol
    - rename self new_name:Standard.Base.Data.Text.Text -> Standard.Base.Meta.Unresolved_Symbol
    - name self -> Standard.Base.Data.Text.Text
    - instrument self -> Standard.Base.Meta.Instrumentor
- type Error
    - value self -> Standard.Base.Any.Any
- type Polyglot
    - value self -> Standard.Base.Any.Any
    - get_language self -> Standard.Base.Meta.Language
Standard.Base.Any.Any.is_same_object_as self value:Standard.Base.Any.Any -> Standard.Base.Data.Boolean.Boolean
Standard.Base.Any.Any.is_a self typ:Standard.Base.Any.Any -> Standard.Base.Data.Boolean.Boolean
Standard.Base.Error.Error.is_a self typ:Standard.Base.Any.Any -> Standard.Base.Data.Boolean.Boolean
atom_with_hole factory:Standard.Base.Any.Any -> Standard.Base.Any.Any
meta ~value:Standard.Base.Any.Any -> (Standard.Base.Meta.Atom|Standard.Base.Meta.Constructor|Standard.Base.Meta.Primitive|Standard.Base.Meta.Polyglot|Standard.Base.Meta.Unresolved_Symbol|Standard.Base.Meta.Error)
is_same_object value_1:Standard.Base.Any.Any value_2:Standard.Base.Any.Any -> Standard.Base.Data.Boolean.Boolean
is_a value:Standard.Base.Any.Any typ:Standard.Base.Any.Any -> Standard.Base.Data.Boolean.Boolean
type_of value:Standard.Base.Any.Any -> Standard.Base.Any.Any
get_annotation target:Standard.Base.Any.Any method:Standard.Base.Any.Any parameter_name:Standard.Base.Any.Any -> Standard.Base.Any.Any
- type Language
    - Java
    - Unknown
is_atom_constructor ~value:Standard.Base.Any.Any -> Standard.Base.Data.Boolean.Boolean
is_atom value:Standard.Base.Any.Any -> Standard.Base.Data.Boolean.Boolean
is_error value:Standard.Base.Any.Any -> Standard.Base.Data.Boolean.Boolean
is_type value:Standard.Base.Any.Any -> Standard.Base.Data.Boolean.Boolean
is_polyglot value:Standard.Base.Any.Any -> Standard.Base.Data.Boolean.Boolean
get_source_location skip_frames:Standard.Base.Any.Any -> Standard.Base.Any.Any
get_simple_type_name value:Standard.Base.Any.Any -> Standard.Base.Any.Any
get_qualified_type_name value:Standard.Base.Any.Any -> Standard.Base.Any.Any
- type Instrumentor
    - uuid id:Standard.Base.Data.Text.Text -> Standard.Base.Any.Any
    - on_enter self fn:Standard.Base.Any.Any -> Standard.Base.Any.Any
    - on_return self fn:Standard.Base.Any.Any expression:(Standard.Base.Data.Text.Text|Standard.Base.Nothing.Nothing)= -> Standard.Base.Any.Any
    - on_call self fn:Standard.Base.Any.Any -> Standard.Base.Any.Any
    - activate self -> Standard.Base.Any.Any

[hubertp]

Looks mostly fine. Few nits to consider

[hubertp]

is this correct? It seems that the underlying value of docsArg is a String?

[JaroslavTulach]

The idea was:

• false - don't generate
• true - generate and use default "style"
• any String - generate and use the "style" specified by the string

[hubertp]

Isn't this available as a constant already somewhere? I'd like to avoid duplicates like that

[JaroslavTulach]

There are few occurrences and even some definitions:

The most suitable one would probably be BuiltinTypes.FQN_ANY, but it is not public and the dependency on package org.enso.compiler.pass.analyse.types doesn't seem correct either.

[hubertp]

Yeah, I figured BuiltinTypes.FQN_ANY should be the source of truth. Shame that we can't include it as a dependency.

[Akirathan]

LGTM. I would also prefer new methods in Compiler rather than adding a parameter, but it is fine.

[radeusgd]

This seems to have a lot in common with TypeInferenceSignatures pass.

I wonder if you could rewrite it to use the results of that pass instead? You could rely on the pass to check the arguments and return types, and also we have there already a way to render the type signature in textual form which we could re-use.

Maybe we could avoid duplication of some of the logic. I guess the problem is TypeInferenceSignatures does not currently handle names of arguments nor if they have default value. But I think this can be easily fixed by extending

--- a/engine/runtime-compiler/src/main/java/org/enso/compiler/pass/analyse/types/TypeRepresentation.java	(revision 1c0ce51bd71281f66428c19057b255bb69e5aa0c)
+++ b/engine/runtime-compiler/src/main/java/org/enso/compiler/pass/analyse/types/TypeRepresentation.java	(date 1737379428154)
@@ -72,7 +72,7 @@
     }
   }
 
-  record ArrowType(TypeRepresentation argType, TypeRepresentation resultType)
+  record ArrowType(String argName, TypeRepresentation argType, boolean hasDefaultValue, TypeRepresentation resultType)
       implements TypeRepresentation {
     @Override
     public String toString() {

[JaroslavTulach]

It would be good to share the code, but all fails since start:

enso/engine/runtime-compiler/src/main/java/org/enso/compiler/pass/analyse/types/TypeInferenceSignatures.java

Lines 130 to 133 in 17dcbd6

	if (hasAnyDefaults) {
	// TODO inferring types that have default arguments
	yield null;
	}

default arguments are prefixed with ~ in the --docs=api format. E.g. we have to process them.

[JaroslavTulach]

The TypeRepresentation classes are really solving similar issues. I recognize hacks I had to do myself. However the suspended arguments support is missing and I need to move forward with #12031 - e.g. I guess I leave the unification for the future.

[radeusgd]

The changes look great.

I'd suggest trying to re-use the TypeInferenceSignatures pass (or extract common logic from it) here. As otherwise are are getting two places that essentially do the same thing (extract the signatures of methods from IR). But that will require some slight changes to TypeInferenceSignatures although I think they should be relatively small.