Dynamic Flink Jobs #3529
-
|
Hey guys, first of all: Great work! I am new to Flink and Nussknacker, but very much interested in how exactly you achieved it to create and submit Flink Jobs based on user input. However I am kind of lost in all the sourcecode. Maybe you could point me towards where I will be able to find the parts relevant for the job creation? Kind regards |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
Hey. Thanks. We have very short and high level page where we describe how nussknacker scenario is translated into Flink job in our documentation. Basically our designer sends jar with something that we call "model" (implementation of source, sink, custom node components + "executor") into Flink and execute this jar passing scenario json as an argument. You can start looking into source code from FlinkDeploymentManager.scala . It is service that works on designer side (BTW it is pluggable - you can write your own) and communicate with Flink via API. On Flink's side there is module that we call "executor". Here is entrypoint class with main method: FlinkProcessMain.scala and here is "registrar" that translates scenario AST into Flink's DSL: FlinkProcessRegistrar.scala. I hope that I've helped you. Feel free to ask for other details. |
Beta Was this translation helpful? Give feedback.
Hey. Thanks. We have very short and high level page where we describe how nussknacker scenario is translated into Flink job in our documentation. Basically our designer sends jar with something that we call "model" (implementation of source, sink, custom node components + "executor") into Flink and execute this jar passing scenario json as an argument.
You can start looking into source code from FlinkDeploymentManager.scala . It is service that works on designer side (BTW it is pluggable - you can write your own) and communicate with Flink via API. On Flink's side there is module that we call "executor". Here is entrypoint class with main method: FlinkProcessMain.scala and here is "regist…