Loader: add PipelineSpec test

[dilyand]

No description provided.

[dilyand]

This appears to be impossible. A PipelineSpec dry runs the entire job. It allows us to provide test inputs that go through the processing logic and populate a test output, on which assertions are performed. However, part of the Loader.run() logic is to load events into BQ and only failures make it all the way to the failedInserts output. The consequence of this is that whenever a PipelineSpec test is run, it tries to insert data in an actual BigQuery table.

There doesn't appear to be a way to provide a mock BQ table to the PipelineSpec test.

[dilyand]

A solution to this seems to be a dedicated test table in BQ where we can load data when the test runs.

We will need:

• a tf stack to deploy test infrastructure (prefixed with gcp_test_* pattern to distinguish from the other stacks)
• each test writes a small number of rows
• a regular job (sql runner?) to reset the table.