Orchestrator sometimes fails

[sw-lkinsella]

We run Airbyte on AKS and it will run well for several days and then the Orchestrator will get itself into some state it can't seem to get out of:

Has anyone seen something like this before? The logs in the UI show me nothing and the AKS control pane just says Error.

I have tried restarting the cluster etc but it seems there's persisted state somewhere that's causing it to keep getting stuck.

[sw-lkinsella]

Seems this may be down to local minio getting filled somehow -

java.lang.RuntimeException: Cannot end publishing: Cannot publish to S3: Storage backend has reached its minimum free drive threshold. Please delete a few objects to proceed. (Service: Amazon S3; Status Code: 507; Error Code: XMinioStorageFull; Request ID: 17E8D11CB1363625; S3 Extended Request ID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8; Proxy: null)

[sw-lkinsella]

I've upped it in values.yaml so we'll see what happens...

[nscheifler]

Hi @sw-lkinsella did this issue continue popping up for you and/or did you come up with a long-term fix?