Indexing completely fails if firehose goes down

[paymog]

Do you want to request a feature or report a bug?

Bug

What is the current behavior?

I'm running graph-node version 0.27.

My graph node is configured with both an RPC provider and a firehose source like so:

[chains.mainnet]
shard = "primary"
provider = [
    { label = "mainnet-default", transport = "rpc", url = "<url to rpc provider>", features = [] },
    { label = "firehose", details = { type = "firehose", url = "<url to firehose>"  }},
]

If I start graph-node and then kill my firehose instance, all indexing seems to stop, even though the rpc provider is still available.

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem.

1. start a firehose grpc server
2. start a graph node with a rpc and firehose provier
3. deploy a subgraph and verify that graph node starts indexing
4. kill the firehose server
5. deploy a new subgraph
6. see the new subgraph never starts indexing

What is the expected behavior?

I expect graph-node to gracefully fall back to the rpc provider when the firehose server isn't available (whether that's due to network issues or internal errors in the firehose server or anything else).

[paymog]

oh, I just tested killing the rpc provider after successful startup (instead of firehose) and I'm seeing the same issue. Looks like graph-node marks the whole chain as "dead" if one of the providers fails.

[matthewdarwin]

I'm not sure that failing back to RPC from firehose is the correct solution.

[paymog]

Interesting, why not? RPC on its own is sufficient for running graph node, why might falling back to it be invalid/incorrect?

[paymog]

I haven't tested this yet, but I'd guess that if I have two RPC providers and one fails, the whole chain will be marked as dead as well. I'd also guess the same thing would happen if I have two firehose providers and one dies.

[azf20]

cc @leoyvens or @lutter on current expected behaviour here? @paymog as you mentioned #3937 is looking to address this same area of robustness

[github-actions]

Looks like this issue has been open for 6 months with no activity. Is it still relevant? If not, please remember to close it.