You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[x ] I am using the latest version of Schemathesis
Describe the bug
During positive testing against a simple POST that does not send a request body we receive back a 415 (unsupported media type). This test is marked as success and no further testing on the operation are carried out. The --checks switch was not used in the test run.
The mock was expecting a request body, responding with a 415.
The question is why only a single test which passed?
I assume that as no request body was in the specification, the test data was limited to a single test.
But why the pass?
I assume that as no request body was in the specification, the test data was limited to a single test.
You are right!
But why the pass?
I assume you'd expect Schemathesis tests to fail because 415 is not documented?
As you mentioned, the --checks option was not used hence only the not_a_server_error check was used which checks for 5xx status codes. The API returned 415, so the check passed.
Or its more a behavior mismatch that Schemathesis does not detect? I.e. the test is considered positive, but the API indicated a client error and it is not reported?
For the latter case I have this check implemented for my own usage, but I see it hard to detect such mismatches for an arbitrary case hence it is not included in the built-in checks. For example, Schemathesis may send some ID in the payload, but the API responds 404 (or 422, or 400, or something else), which is a reasonable response for an unknown ID, but its a positive test case from the spec perspective, even though the API indicates a client error.
Checklist
Describe the bug
During positive testing against a simple POST that does not send a request body we receive back a 415 (unsupported media type). This test is marked as success and no further testing on the operation are carried out. The --checks switch was not used in the test run.
The mock was expecting a request body, responding with a 415.
The question is why only a single test which passed?
I assume that as no request body was in the specification, the test data was limited to a single test.
But why the pass?
To Reproduce
🚨 Mandatory 🚨: Steps to reproduce the behavior:
st run sampleapi.yaml --base-url http://localhost:8080/crawl --cassette-path sampleapi-results.yaml
Please include a minimal API schema causing this issue:
Environment
Additional context
Attached cassette output
st-output.yaml.zip
The text was updated successfully, but these errors were encountered: