Errors do not affect CLI exit code

[clementgarnier]

Describe your problem

Dredd CLI returns 0 when tests are errored, as long as none are failing, e.g.

complete: 0 passing, 0 failing, 1 errors, 0 skipped, 1 total

What command line options do you use?

$ dredd

What is in your dredd.yml?

dry-run: null
hookfiles: null
language: nodejs
sandbox: false
server: /foo/bar/baz
server-wait: 3
init: false
custom: {}
names: false
only: []
reporter: []
output: []
header: []
sorted: false
user: null
inline-errors: false
details: false
method: []
color: true
level: info
timestamp: false
silent: false
path: []
hooks-worker-timeout: 5000
hooks-worker-connect-timeout: 1500
hooks-worker-connect-retry: 500
hooks-worker-after-connect-wait: 100
hooks-worker-term-timeout: 5000
hooks-worker-term-retry: 500
hooks-worker-handler-host: 127.0.0.1
hooks-worker-handler-port: 61321
config: ./dredd.yml
blueprint: sample_blueprint.md
endpoint: 'http://127.0.0.1:3000'

What's your dredd --version output?

dredd v4.1.1 (Darwin 16.6.0; x64)

Does dredd --level=debug uncover something?

Can you send us failing test in a Pull Request?

To reproduce, run dredd with above configuration; note /foo/bar/baz is voluntarily inexistent, which will fail to start the server and mark all tests as error. A sample_blueprint.md can be created with any valid blueprint (I used the first example here) but not required to reproduce, as a missing blueprint file will also trigger the error state.

[michalholasek]

Hello @clementgarnier,

Thank you for the issue! As @honzajavorek said in #765 (comment), this is a serious issue and should be fixed ASAP - I'll start to dig into it right away.

[clementgarnier]

Thank you so much for the timely fix! When should we expect this to be released?

[honzajavorek]

It gets automatically released once this build passes. There are some flaky tests and we did not notice it failed and the release did not happen.

[honzajavorek]

@clementgarnier Here it is - v4.1.2