CloudQuery Collaboration

[yevgenypats]

Hey Team!

I'm Yevgeny, Founder @ CloudQuery. I stumbled upon this project and was wandering if it can be interesting for your to consider using CloudQuery under the hood as the ELT step so you can focus on the analysis layer.

If yes, will be happy to chat and exchange notes.

Best,
Yevgeny

[Ice3man543]

@yevgenypats this seems like a very good idea. We can use the cloudquery Golang API to import it for ELT steps in cloudlist using the extensive provider list you have. We can talk more on Discord or anywhere you prefer. Let me know how you'd like to proceed.

[yevgenypats]

Cool! Can you jump on our discord this will enable all our team to help better. If not, we will join your that's also fine?

Few very high level thoughts:

Importing the API can work though I would suggest running CloudQuery as a standalone, this will eliminate GlueCode and also will give users more flexibility. For example there are some users that already use CloudQuery and they will be able to use some of the projectdiscovery SQL and security rules right out of the box without re-running CloudQuery.

Also, this should save a lot of duplicate documentation on your end. i.e you can have some short documentation on your end and point to CloudQuery docs for extraction and then just focus on the security, compliance side queries and documentation.

In general this is quite a popular approach in the modern data stack which we are big believers.

But the other way can also work though will be more involved imo.