Conformance Reporting to reflect v1.1.0 release #3088

xtineskim · 2024-05-15T19:22:17Z

What type of PR is this?

/kind documentation
/area conformance
What this PR does / why we need it:
Added a function to get the latest version of Gateway API, reflects the new naming format (<GATEWAY|MESH>-<HTTP|TLS|GRPC>), and a start on how to "archive" the past versions of Gateway API

Currently, for "archiving", each version will have it's own implementation-table-.md. When a new released is cut (tbd frequency by the maintainers), the Comparison link in the nav will point to the latest version. Past releases are then moved to an info drop down - also open to other suggestions if this isn't ideal.

Which issue(s) this PR fixes:
Some of follow up tasks related to #3050

Fixes #

Does this PR introduce a user-facing change?:

NONE

k8s-ci-robot · 2024-05-15T19:22:27Z

Hi @xtineskim. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

youngnick · 2024-05-16T01:30:04Z

I agree that the dropdown for previous versions isn't ideal, but it's at least clear. Also, rotating this page as part of the release will encourage implementations to get their new conformance test results in quickly, since they won't show up until they do.

mlavacca

Thanks for this PR. @xtineskim! I think we may need to add instructions on the RELEASE.md page to rotate this table whenever a new release is going to be created.

mlavacca · 2024-05-16T08:04:16Z

/ok-to-test

robscott

Thanks for the quick update @xtineskim! This looks great!

hack/mkdocs-generate-conformance.py

robscott · 2024-05-16T23:33:48Z

RELEASE.md

@@ -70,6 +70,7 @@ For a **MAJOR** or **MINOR** release:
 - Run the `make build-install-yaml` command which will generate install files in the `release/` directory.
 Attach these files to the GitHub release.
 - Update the `README.md` and `site-src/guides/index.md` files to point links and examples to the new release.
+- Update the `site-src/implementation-table.md` to `site-src/implementation-table-<oldVersion>.md`. Add the path to the `past_versions` variable in `hack/mkdocs-generate-conformance.py`.


I think we probably want to add some kind of fixed and predictable timeline for this. We can't expect implementations to support the latest release on day 1, but maybe we switch over the default link 1 month after the release or after we've received 3 reports, whichever comes first?

/cc @shaneutt @youngnick @mlavacca

I like the suggestion that we either do 1month max, or at least 3 reports, that seems solid. We can adjust from there if anyone has complaints.

Yes, +1 on 1 month or 3 reports, whichever happens first.

Sounds good to me as well 👍

mlavacca

Thanks, @xtineskim!

/lgtm

robscott

Thanks @xtineskim! Some small nits about navigation structure, but otherwise LGTM.

robscott · 2024-06-10T21:31:43Z

hack/mkdocs-generate-conformance.py

- # gateway_tls_table = generate_profiles_report(reprots,'TLS')
+ The latest release of Gateway API is shown here. Past Gateway API implementations and feature support can be found here:
+
+ - [v1.0.0](./implementation-table-v1.0.0.md)


Thinking about this more, maybe we should aim for the following in terms of file structure:

/site-src/implementations/{VERSION}.md

Don't track patch versions, all reports for a minor version get lumped together

In published site, we end up with https://gateway-api.sigs.k8s.io/implementations/v1.1

This approach would mean that we wouldn't publish any reports for a release until we met the min criteria (one month or three implementations), but I think it might be simplest to keep track of. We could also choose to limit this to something like the 3 most recent Gateway API versions so the list doesn't become too long.

robscott · 2024-06-10T21:33:43Z

mkdocs.yml

+ - Comparison: implementation-table-v1.1.0.md
+ - Past Version Comparisons:
+ - v1.0.0: implementation-table-v1.0.0.md


Maybe something like this to line up with my comment from above

Suggested change

- Comparison: implementation-table-v1.1.0.md

- Past Version Comparisons:

- v1.0.0: implementation-table-v1.0.0.md

- Comparisons:

- v1.1: implementations/v1.1.md

- v1.0: implementations/v1.0.md

robscott · 2024-06-18T16:28:48Z

Thanks @xtineskim! I think once #3137 merges, we'll have 4 reports for v1.1, so if you rebase this PR and regenerate the v1.1 table, we should have enough to merge this.

mlavacca

We should wait for #3168 to be merged, as the cilium report is flawed.

xtineskim · 2024-06-19T17:12:13Z

@mlavacca just saw the errors now, it seems like it's tripping since only Istio provides the list of extended Supported Features, whereas all the other implementations don't support them 😅

robscott

Thanks @xtineskim!

/approve

hack/mkdocs-generate-conformance.py

k8s-ci-robot · 2024-06-20T18:52:46Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: robscott, xtineskim

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [robscott]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Update so GRPCRoute is listed before TLS Route Co-authored-by: Rob Scott <[email protected]>

robscott · 2024-06-20T20:06:26Z

Thanks @xtineskim!

/lgtm

* Reflect v1.1.0, new naming format * Add line to RELEASE.md * Update release, generated implementation table path file names * Chanch file structure per feedback * Check if extended features are present * Run against implementations * Update hack/mkdocs-generate-conformance.py Update so GRPCRoute is listed before TLS Route Co-authored-by: Rob Scott <[email protected]> --------- Co-authored-by: Rob Scott <[email protected]>

k8s-ci-robot requested review from robscott and youngnick May 15, 2024 19:22

mlavacca reviewed May 16, 2024

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 16, 2024

robscott reviewed May 16, 2024

View reviewed changes

k8s-ci-robot requested review from mlavacca and shaneutt May 16, 2024 23:34

xtineskim mentioned this pull request Jun 4, 2024

[Docs] Adopt Versioning #3115

Open

mlavacca reviewed Jun 5, 2024

View reviewed changes

k8s-ci-robot assigned mlavacca Jun 5, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 5, 2024

robscott reviewed Jun 10, 2024

View reviewed changes

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 18, 2024

xtineskim added 4 commits June 19, 2024 09:21

Reflect v1.1.0, new naming format

e310912

Add line to RELEASE.md

a992738

Update release, generated implementation table path file names

ffc335c

Chanch file structure per feedback

171606a

xtineskim force-pushed the conformance-reporting-v1.1.0 branch from 3273b0e to 171606a Compare June 19, 2024 13:22

mlavacca reviewed Jun 19, 2024

View reviewed changes

xtineskim added 2 commits June 19, 2024 15:15

Check if extended features are present

4410fee

Run against implementations

95a2c2e

robscott reviewed Jun 20, 2024

View reviewed changes

hack/mkdocs-generate-conformance.py Outdated Show resolved Hide resolved

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 20, 2024

Update hack/mkdocs-generate-conformance.py

562873b

Update so GRPCRoute is listed before TLS Route Co-authored-by: Rob Scott <[email protected]>

k8s-ci-robot assigned robscott Jun 20, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 20, 2024

k8s-ci-robot merged commit 02581a1 into kubernetes-sigs:main Jun 20, 2024
8 checks passed

xtineskim mentioned this pull request Jun 27, 2024

REQUEST: New membership for xtineskim kubernetes/org#5038

Closed

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Conformance Reporting to reflect v1.1.0 release #3088

Conformance Reporting to reflect v1.1.0 release #3088

xtineskim commented May 15, 2024 •

edited

Loading

k8s-ci-robot commented May 15, 2024

youngnick commented May 16, 2024

mlavacca left a comment •

edited

Loading

mlavacca commented May 16, 2024

robscott left a comment

robscott May 16, 2024

shaneutt May 17, 2024

youngnick May 22, 2024

mlavacca Jun 5, 2024

mlavacca left a comment

robscott left a comment

robscott Jun 10, 2024

robscott Jun 10, 2024

robscott Jun 10, 2024

robscott commented Jun 18, 2024

mlavacca left a comment

xtineskim commented Jun 19, 2024

robscott left a comment

k8s-ci-robot commented Jun 20, 2024

robscott commented Jun 20, 2024

Conformance Reporting to reflect v1.1.0 release #3088

Conformance Reporting to reflect v1.1.0 release #3088

Conversation

xtineskim commented May 15, 2024 • edited Loading

k8s-ci-robot commented May 15, 2024

youngnick commented May 16, 2024

mlavacca left a comment • edited Loading

Choose a reason for hiding this comment

mlavacca commented May 16, 2024

robscott left a comment

Choose a reason for hiding this comment

robscott May 16, 2024

Choose a reason for hiding this comment

shaneutt May 17, 2024

Choose a reason for hiding this comment

youngnick May 22, 2024

Choose a reason for hiding this comment

mlavacca Jun 5, 2024

Choose a reason for hiding this comment

mlavacca left a comment

Choose a reason for hiding this comment

robscott left a comment

Choose a reason for hiding this comment

robscott Jun 10, 2024

Choose a reason for hiding this comment

robscott Jun 10, 2024

Choose a reason for hiding this comment

robscott Jun 10, 2024

Choose a reason for hiding this comment

robscott commented Jun 18, 2024

mlavacca left a comment

Choose a reason for hiding this comment

xtineskim commented Jun 19, 2024

robscott left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Jun 20, 2024

robscott commented Jun 20, 2024

xtineskim commented May 15, 2024 •

edited

Loading

mlavacca left a comment •

edited

Loading