What is the state of our conformance tests?
One upcoming change will be a number of ineligible endpoints are now, in fact, eligible. Are any of these already covered by tests? What will be the drop of coverage if we include the ineligible tests?
select count(1) from conformance.ineligible_endpoint;
Some of these won’t ever be made eligible. If we look at the reasons for ineligiblity:
select distinct reason from conformance.ineligible_endpoint;
The deprecated endpoints won’t likely be made eligible. Similarly, the vendor-gated and optional ones. The endpoints with the highest possiblity are those whose reasons are “set to be made eligible in X”. So let’s filter the list to these:
select endpoint,reason
from conformance.ineligible_endpoint
where reason like '%conformance test in 1.25'
or reason like '%feature plan for GA 1.31';
Of these 151 (knowing some may be always ineligible), how many are being hit in our current test runs? 80
select count(distinct endpoint)
from audit_event
join conformance.ineligible_endpoint using(endpoint);
Of these, how many are tested by a conformance test? 18
select count(distinct endpoint)
from audit_event
join conformance.ineligible_endpoint using(endpoint)
where conf_test_hit is true;
Let’s dig deeper into those 18. What tests are they hit by?
select endpoint , count(distinct test) as tests
from audit_event
join conformance.ineligible_endpoint using(endpoint)
where conf_test_hit is true
group by endpoint
order by tests;
Most are only hit by 1 or 2 conformance tests, with some outliers at the bottom. The question is whether it seems like these endpoints are being hit intentionally.
One way we could look at it is to see the group and kind the endpoints are a part of, and the tests that hit them. If the test is part of that group and kind, it’s likely that the endpoint was within the scope of the test. Even better if the test’s description matches in any way this endpoint’s purpose.
I will reduce my look to just the tests hit by 1 or 2 endpoints and then expand.
with hit_endpoints as (
select endpoint , count(distinct test) as tests
from audit_event
join conformance.ineligible_endpoint using(endpoint)
where conf_test_hit is true
group by endpoint
order by tests
)
select oa.k8s_kind,
endpoint,
test.codename
from hit_endpoints
join audit_event ae using(endpoint)
join open_api oa using(endpoint)
join conformance.test test on (ae.test = test.codename)
where tests < 3
group by k8s_kind, endpoint, codename
order by k8s_kind, endpoint;
Of these, from my eyes, the storage endpoints and the ephemeral container endpoints are being hit by a kind/domain specific test.
We can remove the discovery test, since something
like getFlowControl... being hit by a discovery test just means that the test
wanted to know if it was there and it said yes, it…discovered it. It’s not
testing the functionality beyond that.
I am also iffy on the aggregator test. If we look at it specifically, it is hitting over 50 endpoints. I can’t say that this test is targeting any of these above endpoints specific functionality.
select count(distinct endpoint)
from audit_event
where test = '[sig-api-machinery] Aggregator Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance]';
So if we remove the aggregator and discovery tests, what do we have?
with hit_endpoints as (
select endpoint , count(distinct test) as tests
from audit_event
join conformance.ineligible_endpoint using(endpoint)
where conf_test_hit is true
group by endpoint
order by tests
)
select oa.k8s_kind,
endpoint,
test.codename
from hit_endpoints
join audit_event ae using(endpoint)
join open_api oa using(endpoint)
join conformance.test test on (ae.test = test.codename)
where tests < 3
and codename not like '%Aggregator Should be able%'
and codename not like '%Discovery%'
group by k8s_kind, endpoint, codename
order by k8s_kind, endpoint;
And of these endpoints, what are their reasons for ineligiblity?
with hit_endpoints as (
select endpoint , reason, count(distinct test) as tests
from audit_event
join conformance.ineligible_endpoint using(endpoint)
where conf_test_hit is true
group by endpoint,reason
order by tests
)
select endpoint, reason
from hit_endpoints
join audit_event ae using(endpoint)
join open_api oa using(endpoint)
join conformance.test test on (ae.test = test.codename)
where tests < 3
and codename not like '%Aggregator Should be able%'
and codename not like '%Discovery%'
group by endpoint, reason
order by endpoint;
So the only one that isn’t vendor-specific has an optimistic, open-ended reason. If it is decided for it to be conformant, then it’s likely already tested.
What is the breakdown of ineligible reasons?
select reason, count(*)
from conformance.ineligible_endpoint
group by reason
order by count desc;
The majority are optional or vendor-specific. Only a small amount have to do with deprecation.
select count(*)
from conformance.ineligible_endpoint
where reason ilike '%deprecat%';
Of these deprecated endpoints, how many are still in the open api spec? I assume all, but if any are missing then it would make sense to remove them entirely.
with ineligible as(
select endpoint
from conformance.ineligible_endpoint
where reason ilike '%deprecat%'
)
select endpoint
from ineligible
except
select endpoint
from open_api
where release = '1.27.0';
;
So all the deprecated endpoints are still in our API spec.
It is still a unclear to me the effect to coverage if any of these endpints are made eligible. For one, I am uncertain which of them are actually going to be made eligible. There are only 4 whose reason is an upcoming eligibility and three of them may no longer count toward that.
Of the currently ineligible that are hit by conformance tests, the majority are for vendor-specific features and likely wouldn’t qualify anyway. The remaining has a reason of ‘not eligible for conformance yet’ and so it is positive that, if it is later made eligible, it is already covered by a fairly well targeted test.
It would be good to revisit the ineligible endpoints list and standardize the reasons for ineligiblity. We can create consistent wording for deprecation, making it easier to filter these out in later queries. It woudl also be good to give more context to the ‘optional feature’ or ‘not eligible yet’ reasons. Will these endpoints always be optional and thus never conformant? What are the requirements for the ‘not eligible yet’ endpoints before they can be considered eligible? It might be good to have a field for “always ineligible” that will let us filter out those endpoints from our conversations entirely.