CI check for Standard.Base API changes via snapshotting

[Akirathan]

Stability of Enso APIs, encapsulation of internals, detection of accidental API changes are of great value for Enso customers. Helping Enso runtime developers keeping these values is of high priority.

• Infrastructure for enso --docs option & signature generator #10291
• Reducing helper methods in Standard.Base.Meta #12031 (comment)

There have been two recent developments demonstrating how easy API snapshotting is and how it can be used in tests to ensure stability of API of a single module. The goal of this PR is to scale this up and apply such an API snapshotting to the whole Standard.Base and its APIs.

Steps

First of all we need to do few manual steps to prepare the infrastructure:

Generate API Snapshot

Executing following command:

sbt:enso> runEngineDistribution --docs=api --in-project distribution/lib/Standard/Base/0.0.0-dev/

generates distribution/lib/Standard/Base/0.0.0-dev/docs/api/ directory with a lot of API signatures of publicly accessible modules.

Store in Version Control

Let's put the current snapshot under version control:

enso$ git add distribution/lib/Standard/Base/0.0.0-dev/docs/
enso$ git commit -m "Initial snapshot of Standard.Base APIs"

The CI Check

Now we need to modify the CI to perform the check. E.g. to execute:

sbt:enso> runEngineDistribution --docs=api --in-project distribution/lib/Standard/Base/0.0.0-dev/
enso$ git diff --exit-code distribution/lib/Standard/Base/0.0.0-dev/docs/api/

The first command re-generates the Standard.Base API snapshot according to latest state of the .enso code. The second command checks for any differences. It succeeds if there are no differences. It fails with exit code 1 when any differences is found. In such case let us fail the CI build.

CI Failed. What Shall I do Now?

Once the above is implemented, we are likely to see the CI check to fail. Failing is good! Failure of this check means: we have discovered an unintended API change. Unintended API changes are bad. Thus detecting them is good!

Anyway, we need to fix the CI to be green. Here are the tips:

1. don't do the API change.
2. add private to the top of the .enso file containing the change
3. mark the method private
4. if you really wish to make an API change, turn it into intentional API change

Intentional API Change

When an API change is intended, it is not only changing the .enso code, but it is also updating the API snapshot. The simplest way to update the API snapshot is to regenerate it:

sbt:enso> runEngineDistribution --docs=api --in-project distribution/lib/Standard/Base/0.0.0-dev/

then review the changes via git diff and if they seem OK, integrate them into your PR.

Prior Work

Inspiration from NetBeans:

• https://github.com/jtulach/netbeans-apitest
• https://github.com/jtulach/netbeans-apitest/blob/master/src/test/resources/org/netbeans/apitest/org-netbeans-api-java.sig

[JaroslavTulach]

Prepare a design for review, Pavel.

[Akirathan]

As part of library compilation, we already generate cached Suggestions. They are later sent to the IDE in JSON format. The sigfile can thus be a JSON-formatted .suggestions. Implementation-wise this should be the most straightforward approach. Moreover, for the backward compatibility, suggestions are what we should care about the most.

[JaroslavTulach]

The sigfile can thus be a JSON-formatted .suggestions.

• Related to Implement a more efficient serde of Suggestions #9999

CCing @4e6 and @hubertp

[JaroslavTulach]

• another idea to reuse some other work: Infrastructure for enso --docs option & signature generator #10291
• let's have various "documentation formats"
• one of the formats would only generate signatures without and comments

The output format could be a simplified .md file:

- type `File_By_Line`
    - `new` `file:File` `encoding:Encoding=` `offset:Integer=`
    - `Reader` `file:File` `encoding:Encoding` `limit_lines:(Integer|Nothing)` `filter_func` `row_map` `file_end=`
    - ...
- `read_line` `file:File_By_Line line:Integer=` `~default=`

with such an infrastructure we would then:

• put the files for each library under version control
• create a CI job to regenerate the files
• fail the build whenever there is any change in files
• forcing human to regenerate the files and commit then
• to avoid unwanted API changes in our libraries

[Akirathan]

Reordering methods/types in a module is not an API change. That is why the API signature generator should sort the markdown file. That will not only fix this case, but also provide better readability. We just have to stick to a certain ordering convention.

[Akirathan]

Using git diff to check for change of the API is not a good idea. If a new file was added, git diff will not discover it, because it is still untracked. We should rather output the API signatures in a different directory and then use simple diff program.

[enso-bot]

Pavel Marek reports a new STANDUP for yesterday (2025-01-27):

Progress: - Started to work on #10507

• Added tests for the visitation order of IR elements in a module by the Documentation visitor.
• finally merged Dump compiler IR to IGV #12061 after many Windows job restarts It should be finished by 2025-02-04.

[Akirathan]

Important note to remember for some follow-up PRs: Exports should be part of the API.

[Akirathan]

There should be no API file generated for synthetical and empty modules. In other words, there should be no empty .md files (files with just title and no content).

[enso-bot]

Pavel Marek reports a new STANDUP for today (2025-01-29):

Progress: - The preparation PR for the CI job is ready to review: #12175 It should be finished by 2025-02-04.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2025-01-31):

Progress: - Generated API signatures in - #12203

• Will be followed by a PR that creates the API check job.

• Replace more Scala case classes with Java - Replace more IR Scala case classes with Java interfaces #12211

• Convert rest of global typing pass group to mini passes #11717 will be blocked until I fully understand pattern matching IR and runtime interpretation.

  • The whole NestedPatternMatch IR pass seems unnecessary and too complicated - creates a lot of IR nodes.
  • Before migrating this pass to a mini pass, we could try to simplify the whole pattern matching IR (both Scala and Truffle).
    • Update reported as Simplify case of and pattern matching implementation #12213

  It should be finished by 2025-02-04.