Add PHPUnit internal helper(s)

[stloyd]

We often start to do copy-paste work in tests, thus we should begin to have some reusable helpers to make our internal tests easier to maintain.

Helpers:

• FilesystemHelper with: create, delete for files & directories,
• ...

[Bellangelo]

Hi @norberttech , I would like to help on this. If I am not mistaken there isn't any generic place to create the new helpers. Should we create a tests directory at the root for this case?

[norberttech]

hey @Bellangelo, awesome! I think I would put it inside of the src/core/etl/src/Flow/ETL/Test/... so it can be installed with ETL package together

[Bellangelo]

@norberttech, i checked and the Filesystem issue that the original PR was addressing seems to be fixed. I checked a little bit around the project and I noticed that some of the Extractor's tests logic is duplicated so I created a PR for it #1291. Not sure if all the changes are needed but it is a start.

Also, since the description of this issue does not contain much info I would really appreciate it if you could add what it is missing currently.

[norberttech]

Since this task was never properly explained, here is a proper description of what we are trying to achieve here:

#1291 (comment)

[norberttech]

I think we can create some very generic fake extractors and define schema for them,for example

• df()->read(from_flow_orders(limit:1_000))
• flow_orders_schema() : Schema
• do the same for products
• customers
• inventory

and we should make sure all of those datasets keep consistent schema and that they are using all possible entry types.
Those virtual datasets would need to follow
very strict backward compatibility policy and proper schema evolution.

This would make so much easier and more realistic to test not only entire pipeline but also stand-alone scalar functions as we can also create helpers that would just give us one row (and use them inside fake extractors)