You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Summary
As of now, there is some issues in the local setup of Dagger:
The local.properties assumes there is a TestMessageproto present. However, it is not the case. This causes the dagger not to start properly.
The other test protos which come inbuilt with Dagger are in test module which are not usable in local.properties.
The watermark delay is set to 0 in the local.properties which might cause issues in some cases, such as when events are coming in late due to issues with kafka. Also, this watermark delay needs to be much higher when Parquet daggers are being setup.
With multiple ways people can create daggers like with parquet source, kafka source, with preprocessors, etc, ideally there should be specific examples on how to do each
Proposed solution
These are some suggestions based on the feedback received from internal discussions as well as from users:
Creating a Dockerfile/Docker compose with all dependencies bundled( local kafka cluster, sample protos, etc). User just need to run it and check the logs for output. No manual setup required except Docker.
Organizing the documentation explaining how to do different things with Dagger in a single place. For example,
Create a Dagger with Kafka as a source
Create a Dagger with Parquet Files as a source
Create a Dagger using preprocessors
Currently, the reference is not streamlined and is organised in multiple sections.
What it will achieve
Following benefits are expected:
Faster onboarding of users: People can utilise the examples to try out different features of daggers without help by the committers/maintainers
Wider adoption: Faster ways to try out different features by minimal setup means people can quickly check whether the product satisfies their needs or not. Will help in a faster feedback loop.
Additional context
None
The text was updated successfully, but these errors were encountered:
Summary
As of now, there is some issues in the local setup of Dagger:
local.properties
assumes there is aTestMessage
proto present. However, it is not the case. This causes the dagger not to start properly.local.properties
.watermark delay
is set to 0 in the local.properties which might cause issues in some cases, such as when events are coming in late due to issues with kafka. Also, this watermark delay needs to be much higher when Parquet daggers are being setup.Proposed solution
These are some suggestions based on the feedback received from internal discussions as well as from users:
Creating a Dockerfile/Docker compose with all dependencies bundled( local kafka cluster, sample protos, etc). User just need to run it and check the logs for output. No manual setup required except Docker.
Organizing the documentation explaining how to do different things with Dagger in a single place. For example,
Create a Dagger with Kafka as a source
Create a Dagger with Parquet Files as a source
Create a Dagger using preprocessors
Currently, the reference is not streamlined and is organised in multiple sections.
What it will achieve
Following benefits are expected:
Additional context
None
The text was updated successfully, but these errors were encountered: