Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ProviderVerifier is not failing the test when PublishResults part has failed downstream #401

Open
xeroeugene opened this issue Jun 17, 2022 · 1 comment
Labels
bug Indicates an unexpected problem or unintended behavior upstream Indicates that an issue relates to an upstream problem (such as in pact-reference)

Comments

@xeroeugene
Copy link

BrokerSource (and UriSource in the future #392 ) have capability to publish verification results to the broker.

However underlying implementation might fail to publish the results (for many reasons), and that has no effect on the test results.

In other words your test would still be marked as passed and green, pact verification succeeded, but the results are not published anywhere and there is no way to know that, unless you are manually monitoring the logs.

This is especially important with webhook based scenarios, when the provider side builds are triggered on demand by the broker on pact contract changes. (no human would be involved to read the logs to understand that there was a problem)

Example from logs:
[ERROR][pact_verifier] Publishing of verification results failed with an error
when a read-only broker token was used to publish the results

Potential solution options:

  1. Make verification results publishing error propagation the default behavior, causing the provider verification test to fail
  2. Make verification results publishing error propagation an optional capability that can be switched on and off in the publishing options
@adamrodger adamrodger added bug Indicates an unexpected problem or unintended behavior ffi labels Jun 18, 2022
@adamrodger
Copy link
Contributor

This is probably a defect in the FFI not reporting the failure back to PactNet. I'll have to check the FFI source.

@xeroeugene xeroeugene changed the title Provide a configuration option for the provider verifier to fail the test when PublishResults has failed downstream ProviderVerifier is not failing the test when PublishResults part has failed downstream Jul 5, 2022
@mefellows mefellows removed the ffi label Jul 12, 2023
@mefellows mefellows added the upstream Indicates that an issue relates to an upstream problem (such as in pact-reference) label Aug 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Indicates an unexpected problem or unintended behavior upstream Indicates that an issue relates to an upstream problem (such as in pact-reference)
Projects
Status: New Issue
Development

No branches or pull requests

3 participants