fail-fast config/template consistency checks break with concurrent on-demand pipelines

[gberche-orange]

Is your feature request related to a problem? Please describe.

The consistency check performed by COA between the template and the config repo is incompatible with the race condition that occurs when COAB submits concurrent multiple feature branches on the template repo, and a single config repo commit with multiple deployment activation.

This results into frequent failures of the update pipeline, resulting in timeouts of on-demand service instances

	
/tmp/build/140b8ac5/result-dir/lib/deployment_deployers_config.rb:29:in `load_configs': Inconsistency detected: deployment <c_211c0b64-aaaf-4ad6-a606-ee9d1810b78b> is marked as active, but no deployment-dependencies.yml, nor other deployer config found at ../templates/coab-depls/c_211c0b64-aaaf-4ad6-a606-ee9d1810b78b (RuntimeError)
	from /tmp/build/140b8ac5/result-dir/lib/root_deployment.rb:32:in `block in overview_from_hash'
	from /tmp/build/140b8ac5/result-dir/lib/root_deployment.rb:24:in `each'
	from /tmp/build/140b8ac5/result-dir/lib/root_deployment.rb:24:in `overview_from_hash'
	from /tmp/build/140b8ac5/result-dir/lib/pipeline_generator.rb:78:in `set_context'
	from /tmp/build/140b8ac5/result-dir/lib/pipeline_generator.rb:62:in `load_erb_context'
	from /tmp/build/140b8ac5/result-dir/lib/pipeline_generator.rb:46:in `execute'
	from ./scripts/generate-depls.rb:8:in `<main>'

Describe the solution you'd like

Either:

• systematically have the fail-fast consistency check instead report a warning (rather and a job failure)
• conditionally have the fail-fast consistency check instead report a warning (e.g. per root deployment, likely in configured in the template repo)

Describe alternatives you've considered

COAB to watch for the merge of the template feature branch before pushing commits to the config repo. However, the associated complexity is just to high for little end-user value.

[gberche-orange]

Discussion during coab backlog review:

• allow template authors to control consistency check per root deployment through shared-config.yml and override in private-config.yml
• default mode: fail
• alternative mode: warn. A log entry in the concourse log.