Quarantine data in silver layer

[sanketkaleda]

Normally we have a requirement of applying data quality rules from bronze to silver layer, so we would need to qurantine data in silver layer, currently its not supported to add quaratine feature in silver layer. will it be available soon ?

[ravi-databricks]

Usually bronze is entry point where customers do quarantine data and send back to source. We can introduce quarantine feature in silver too. It might be for v0.0.10 release since v.0.0.9 release is finalized

[sanketkaleda]

Thanks for accepting it as an enhancement for v0.0.10. In our case, we need to keep bronze layer without applying any data quality rule and send the quality data in silver and quarantine the bad data in silver only. The purpose of quarantine is not to send the data back to source but to have detailed data quality monitoring of rows are rejected. handshake with source system will still be manual and done by respective data product owner.

If we quarantine the data in bronze layer itself then we will loose the idea of keeping As-Is source data in bronze.