FileManager: presignUrl 'intermittent' issue of getting Secrets Error

[raylrui]

There is an 'intermittent' issue in UI when try to access files through presign url.
Happened around 24 Apr 2025, 2:30-3:30 pm
Disappear around 24 Apr 2025, 4:00 pm

Slack link: https://umccr.slack.com/archives/CP356DDCH/p1745468268646679

This is the error message from ai call:
Error: {"message":"Secrets manager error: `no valid secret arn:aws:secretsmanager:ap-southeast-2:472057503814:secret:orcabus/file-manager-presign-user: service error`"}

[andrewpatto]

So I changed the API gateway lambda resource policy for GetSecretValue from

orcabus/file-manager-presign-user-*

to

orcabus/file-manager-presign-user*

(live in prod - as this was a reasonably safe change)

[andrewpatto]

I have no idea how GetSecretValue is being called - or whether this change has any effect! Just noting the changes that occurred soon after Joep put in the bug report

[raylrui]

So I changed the API gateway lambda resource policy from

orcabus/file-manager-presign-user-*

to

orcabus/file-manager-presign-user*

(live in prod - as this was a reasonably safe change)

Normally AWS will use "-" when add suffix. The secret arn is arn:aws:secretsmanager:ap-southeast-2:472057503814:secret:orcabus/file-manager-presign-user-vJctaO, so it should be okay

[andrewpatto]

Yeah I can't see why it would have changed anything. But I can't see how it was causing an issue in the first place either (the secret was definitely already there).

[andrewpatto]

Should I revert the policy change to make sure it isn't related?

[raylrui]

I have no idea how GetSecretValue is being called - or whether this change has any effect! Just noting the changes that occurred soon after Joep put in the bug report

yes, not sure about the how filemanager use GetSecretValue, but Is suspect the fallback mechanism in the code that tries to use the Lambda role if the access key fails

[raylrui]

Should I revert the policy change to make sure it isn't related?

sure, we can try. And I think we should revert as the policy is from the code, not sure if there will be conflict when next release

[raylrui]

Should I revert the policy change to make sure it isn't related?

Hi @andrewpatto do you revert your changes? it seems that presign url still works fine

[andrewpatto]

Yes - reverted

[raylrui]

Yes - reverted

thanks, the presign url still works fine, so it maybe more related to AWS side 'intermittent' issue.

[mmalenic]

I think if it was an authorization issue the error message would be something like: ... is not authorized to perform: secretsmanager:GetSecretValue ...?

Apparently there can be transient issues when fetching secrets, and the secrets cache library that filemanager uses has an "ignore_transient_errors" option. This wasn't turned on, but I can turn it on from now.

[raylrui]

It seems happen again around 3:30pm-4:00 pm today.
see slack: https://umccr.slack.com/archives/CP356DDCH/p1745992496713709?thread_ts=1745475092.570489&cid=CP356DDCH

[andrewpatto]

Given it seems to be happening with enough regularity - I'd suggest rolling out the "ignore_transient_errors" thing ASAP and then do a wait and see.