The recommended way to not skip auto materializations

[knowlix]

Context:

Auto-materialization policy defines the max_materializations_per_minute. Everything above this is skipped and requires a manual actions. The default value is 1. As for me the default is the recommended one.

Case:

I have an asset with dynamic number of partitions being created daily and in order to not to skip the processing of new partitions in the downstream assets I have to set max_materializations_per_minute=1000 (a magnitude bigger than the avg num of new partitions to be on the safe side).

Question:

I doubt that it is a stable solution. What is the recommended way and what are the potential implications of the huge max_materializations_per_minute?

[tacastillo]

Hmm, The auto-materialization policy API might not be the best use case for this.

Have you thought about doing a daily schedule that kicks off N number of RunRequests, one for each dynamic partition you have?

[knowlix]

Thank you @tacastillo for reply. There is a pipeline of assets A->B->C where the A (with a set of dynamic partitions) is kicked by schedule. B and C start as soon as their dependency is ready using auto-materialization policy. Kicking B and C by schedule is not a really elegant solution.