Consider schema cache state on the health check

[steve-chavez]

As suggested in #2092 (comment)

However, what we can and should do is: Keep track of the status of the last schema cache refresh - and if that fails, e.g. because of something like #2024, we should return 503. Currently we are still returning 200, when the schema cache fails.

This change does prevent a healthy response in case something like #2024 happens, but only at startup. If the schema cache fails to load after startup, the main app will continue working and the health check will respond accordingly.

The main app failure happens at startup because of:

postgrest/src/PostgREST/App.hs

Lines 203 to 208 in 4f7ee6f

	dbStructure <-
	case maybeDbStructure of
	Just dbStructure ->
	return dbStructure
	Nothing ->
	throwError Error.ConnectionLostError

Should we turn the schema cache to Nothing if it fails loading? That would make the main app fail for all requests, I think it's better to leave the schema cache stale in that case, as it happens currently.

Edit: On second thought, I think it'd be better to void the schema cache, otherwise the app will continue running with a stale schema cache and the developers won't know about it.

[steve-chavez]

 Nothing -> 
  throwError Error.ConnectionLostError 

Fixed the above, it gave a misleading connection error("Database connection lost. Retrying the connection.") while the error was with the schema cache query(error now is "Could not query the database for the schema cache. Retrying.").

[wolfgangwalther]

Edit: On second thought, I think it'd be better to void the schema cache, otherwise the app will continue running with a stale schema cache and the developers won't know about it.

Agreed!

[steve-chavez]

I had difficulties with trying to fix the endless loop schema cache loading here.

I left a comment on #1766 (comment) to work on it on a later PR.

---

This PR should be good to go.

[wolfgangwalther]

I tested the ready endpoint by adding it as the wait_for_startup uri in the io tests in #2116. Before this PR here, using ready would fail half the io tests randomly. Now, they are passing.