Feature request: Explicitly allow documentSelector to return multiple results

[MikaelElkiaer]

I love this project and it actually covers most of my use cases.

One thing though, which I can't seem to solve - other than templating my test files, which I want to avoid for the sake of simplicity/transparency.
I would like to do assertions on any documents of a certain kind - e.g. to check image across all workloads.
I would have hoped to be able to do this by setting multiple templates or having the documentSelector return multiple documents, but there is a hard limit on this right now allowing only a single document to be returned.

Perhaps documentSelector could be expanded, .e.g.:

tests:
  - asserts: []
    documentSelector:
      allowMultiple: true # defaults to false for backwards compatibility
      path: kind
      value: deployment

Then if allowMultiple is set to true, the asserts would be applied to all returned documents instead.

Please let me know if I can already do something like this - again, without templating test files.

[mszygenda]

I was also suprised that this is not the default behavior documentSelector.

I actually gave it a shot and prepared PR which adds that functionality.

https://github.com/helm-unittest/helm-unittest/pull/328/files

USAGE:

suite: Document Selector is matching many documents
templates:
  - "templates/deployments*.yaml"
tests:
  - it: deployment names should end with -deployment suffix
    documentSelector:
      path: kind
      value: Deployment
      matchMany: true
    asserts:
      - matchRegex:
          path: metadata.name
          pattern: -deployment$

It has one limitation though - every template that you select for your test-suite has to have at least one matching manifest. The test will fail otherwise.

So you can't just put:

templates:
  - "templates/*.yaml"

But rather

templates:
  - "templates/deployment-*.yaml"

Making that work either require greater refactor or would actually change the behavior of the way assertions are executed (skip execution if no manifests found in file) which would be a breaking change

[MikaelElkiaer]

Awesome work @mszygenda.

Regarding the "skip execution", I suppose it would not be changed behaviour if it only applies when matchMany: true.
It could also be a new option, e.g. noMatchBehaviour: <fail|skip>.

I am a bit surprised at how many code changes are needed for such a feature.

[mszygenda]

It could also be a new option, e.g. noMatchBehaviour: <fail|skip>.

Yeah, I guess this might be a direction to go to. But on the other hand - this option could lead to very unwanted behavior.

Imagine you have following test-case:

suite: Document Selector is matching many documents
templates:
  - "templates/deployments*.yaml"
tests:
  - it: deployment names should end with -deployment suffix
    documentSelector:
      path: kind
      value: Deployment
      matchMany: true
      noMatchBehaviour: skip
    asserts:
      - matchRegex:
          path: metadata.name
          pattern: -deployment$

And now you break your chart templates so that they do not output anything. The test passes - but it does not assert anything. Same would happen if you would break this way just one of the deployments.

Conceptually it's not wrong though - All out of selected documents passed the assertion. Maybe it should be separate test-case to make sure they're not empty.

Question is whether this should be property of selector itself - or maybe the assertion / test-case / suite (please note - you can get 0 documents even if you don't use the selector).

I'll think about and see if there's easy way to achieve it.

I am a bit surprised at how many code changes are needed for such a feature

Yeah, it seemed to me as well that this should be very small change (since it already worked fine for multiple documents if you did not use documentSelector). But it turned out to be a bit more complex due to the fact that the single document assumption lies at multiple layers and everything is just based upon "documentIndex" property (documentSelector is just concept build on top of it).

[mszygenda]

I prepared some PoC of "skipEmpty" at document selector level.

You can have a look here

#335

Example:

suite: Document Selector is matching many documents and skipping empty templates
templates:
  - "*"
tests:
  - it: deployment names should end with -deployment suffix
    documentSelector:
      path: kind
      value: Deployment
      matchMany: true
      skipEmpty: true
    asserts:
      - matchRegex:
          path: metadata.name
          pattern: -deployment$