502 Bad Gateway on schema pull

[rriley99]

We suddenly lost access to our Redshift source. We are using v7 (trying to migrate over to v10, so we are EoL on this installation). The left panel does not allow me to pull schema data. When I use the inspect window in the browser I get a 502 Bad Gateway. However, I can complete queries, both ad-hoc and scheduled so I can rule out any network or permissions issues. Based on this it sounds like the backend is failing to communicate.

Please help, it's been a long day haha.

[justinclift]

You'll probably need to take a look at the backend docker logs to see the actual error that's happening.

Do you have ssh access to your server, and are you comfortable with using the command line?

[rriley99]

So I actually just took a slightly different path and exec'ed into the webserver and tried to make the problematic schema query and I got a different error, a cert error... I think that might be the culprit. I will chase that down and if that's not it I will give this a shot.

[justinclift]

Oh a cert error could make sense too. That kind of thing can cause all kinds of weird problems.

[rriley99]

So the cert issue was a bit of a red herring (And a lot of other ones!). But I've found out what the "error" is. In the UI, I attempt to refresh the schema, this query takes anywhere from 30s to 190s. In the webserver logs, I get this timeout after 30, 60, and 90 seconds and then a 90 second timeout is triggering a 502 on the request:
[1] [CRITICAL] WORKER TIMEOUT (pid:16)

So the webserver workers are timing out!

I thought there was a schema queue for this, but I don't see any activity on my schema worker pod either. Perhaps I am missing something? I see that there is a timeout arg passed to gunicorn in v10, but I'm not trying to upgrade to v10.

[justinclift]

So the webserver workers are timing out!

Ouch. Version 7 is before my time, but from what I've seen of notes ages ago there might be some kind of status/info (maybe even management?) pieces in the admin section of Redash v7. It'll probably titled something like "Celery" as that the queuing mechanism that version of Redash used.

That being said, it sounds like your current configuration has been working for you up until now?

So unless there was a change in the configuration (doesn't seem like it?) then it sounds weird for the existing queues to stop handling schema information requests.

On the configuration change note, has there been any access or permissions changes on the Redshift end?

As a diagnostic step, does Redshift have a command line tool you can run queries through? Pretty sure that'll be a yes, but I don't know Redshift at all.

If such a command line tool does exist, then I'd recommend trying to manually run the schema refresh query from that yourself, specifically using the same username/password that Redash is using. Just to 100% figure out if it's potentially some new authentication or permissions problem on the Redshift end.

[rriley99]

Yeah, so it looks like there is definitely an issue in a recent redshift patch, so I opened up a ticket with them, but through some patching I was able to add gunicorn timeout and increase the celery task timeout (these are all in v10). However, I've been hitting a 3 min timeout on the refresh query anyway, no idea where though.