make channel registry asynchronous

[AntoineDao]

We noticed that the Webhook sink was executing very slowly for high throughput events and concluded it was because the http requests were being executed synchronously. This commit fixes this by making the loop run in a go routine so it is not blocking.

I also added a quick test with a dummy slow sink.

Let me know if this change makes sense! 🙌

[AntoineDao]

Bump for attention 😄

[mustafaakin]

Hello, this is a bit dangerous because in the event of a slow sink and high flow events, it might spawn unlimited go routines, eventually getting OOM-killed. Can you also make your test by making channel bufferred, so that writes to channel will not be blocking?

[AntoineDao]

That makes sense. Should the channel buffer count be set in the config somehow so users can size their deployment as needed?

[mustafaakin]

Would be better, but with a default value of say 10? But I also guess we still need goroutines and a proper wait group because the consumer/processor would still be one goroutine.