Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

discussion: Semantic validation checklist #54

Open
whitlockjc opened this issue May 29, 2019 · 3 comments
Open

discussion: Semantic validation checklist #54

whitlockjc opened this issue May 29, 2019 · 3 comments
Assignees
Labels
documentation Improvements or additions to documentation

Comments

@whitlockjc
Copy link
Member

whitlockjc commented May 29, 2019

I think it would be great to keep a running list of all "semantic validation" (validation not performed via JSON Schema either due to bugs, like 2.0's array items, or due to runtime processing) As I rewrite sway for 3.x support, I've come up with the following validations (of the OpenAPI Specification Document itself, not runtime processing like readOnly). Below is this list and I would like to verify my findings, fill in any gaps and eventually turn this into it's own documentation file for tooling authors.

Semantic Validation

Semantic validation is performing validations on top of the structural validation provided by JSON Schema.

Common

Description Type
Operations must have unique (name + in combination) parameters Error
Operations must have unique operationId Error
Parameters must be unique (in + name combination) at their definition path (path-scoped parameters can be overridden at the operation-scope) Error
Path parameters declared in the path string need matching parameter definitions (Either at the path-level or the operation) Error
Path parameter declarations do not allow empty names (/path/{} is not valid) Error
Path parameters definition (Either at the path-level or the operation) need matching parameter declarations Error
Path strings must be (equivalently) different (Example: /pet/{petId} and /pet/{petId2} are equivalently the same and would generate an error) Error
Referenceable definitions should be used by being referenced in the appropriate way Warning
References must point to existing documents or document fragments Error
Tag names must be unique Error
The default property for Schema Objects, or schema-like objects (non-body parameters), must validate against the respective JSON Schema Error
The discriminator property for Schema Objects must be a required field Error
The items property (where available) must be an Object (Unlike JSON Schema which Array or an Object) Error
The name used in Security Requirements must match a Security Definition/Schema name Error
The required properties for a Schema Object must be defined in the object or one of its ancestors Error

OpenAPI 3.0.x

Description Type
All names for #/components/* must match the following regex ^[a-zA-Z0-9\.\-_]+$ Error
Runtime expressions of Link Objects using $request.{parameter} must have matching parameter defined on its parent Error
Server variables declared in the server object's URL need matching variable definitions Error
The name used for Media Type Objects must have a matching properties entry for the schema Error
The operationId property of the Link Objects must have a matching Operation Object Error
The operationRef property of Link Objects must point to an Operation Object Error

Swagger 2.0

Description Type
Operations cannot have both a body parameter and a formData parameter Error
Operations must have only one body parameter Error
Parameter constraint properties should match the parameter type Warning
Parameters should not have a default property when required is true Warning
Parameters with type of file must have a consumes of multipart/form-data or application/x-www-form-urlencoded Error
Parameters with type of array must have an items property Error
@whitlockjc whitlockjc added the documentation Improvements or additions to documentation label May 29, 2019
@kylef
Copy link

kylef commented May 31, 2019

In Apiary and our surrounding tooling and OpenAPI parsers/validators, we are trying to follow the robustness principle in the sense that we lean towards warnings over errors wherever possible. We will not emit an error unless absolutely necessary. For us the semantics for error is that an error would cause our parsers and tooling to exit and not continue. Which is similar to most compilers (clang, gcc etc).

For example, in Apiary you cannot save an API Description Document that contains any errors. Thus, unless absolutely necessary (such as the parser cannot continue) we will lean towards a warning. Here's some examples:

  • If validation of an optional property fails we can treat that as a warning and discard the property. Such as if the required property in the 'Parameter Object' is set to a non-boolean value.
  • For path type 'Parameter Object's, the required property is required and must be set to true. Our parser will warn if the user does set this value correctly. We can continue and always interpret the parameter to be required.
  • The required paths property of the 'OpenAPI Object' is missing would cause a warning and the behaviour is equivalent of paths being set to an empty object.

We've seen that some of our users who are using OpenAPI 2 generators that end up with semantically invalid specifications which hinder the user's ability to use them in Apiary if we was to outright treat every validation failure as an error. We attempt to be lenient on the user where possible so they can continue to use their API Description Documents in Apiary. We've found cases in the past where users will discover that their (supposedly valid OpenAPI 2.0 document) could be used in other tooling but could not be used in Apiary due to more strict validation which prevented the user using Apiary without altering their (possibly very large and generated) OpenAPI specification.

Another reason behind this behaviour in our stack comes from API Blueprint, we attempt to never break compatibility with a document that has previously been parsable in Apiary. Thus we rarely attempt to add validations that cause an error to our existing parsers unless absolutely necessary for new functionality. In OpenAPI that's less likely to be possible as it is a more rigid YAML/JSON structure and new features would be added in new properties of an object. Perhaps an example of this in OpenAPI 3 (at least for us), is that we don't fully support the complete spec in regards to some of the optional properties. servers in OpenAPI Object is one example of this, if a user uses servers right now they are discarded and the user is shown a warning regarding the unsupported feature. In the future, when we add support for servers, if we introduced an error, it could mean that a previously supported document would emit an error and thus would no longer be parsable on newer parsers.

I wanted to bring this design point from Apiary for discussion and propose that some of these items above that are specified as type error may not need to be errors. For example:

Path parameters definition (Either at the path-level or the operation) need matching parameter declarations

Parser's can implicitly create a parameter definition in its parse result for the path parameter if missing.

References must point to existing documents or document fragments

If the reference is in an optional property (or optional tree of properties) in an object it can be discarded.

@darrelmiller
Copy link
Member

We've found cases in the past where users will discover that their (supposedly valid OpenAPI 2.0 document) could be used in other tooling but could not be used in Apiary due to more strict validation

The solution to this problem is not to be more lenient but to encourage tools to respect the rules of the specification. If tooling wants to implement "fix my description" mechanisms they are encouraged to do so. However I would absolutely not recommend that tools attempt to implicitly correct descriptions nor ignore errors. This approach severely hinders learning and leads to poor interoperability.

@handrews
Copy link
Member

I'm going to move this to the OASComply project, as it is addressing this sort of thing.

@handrews handrews transferred this issue from OAI/OpenAPI-Specification Jan 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation
Projects
None yet
Development

No branches or pull requests

5 participants