Update, Extract Nifi Configuration
Deploy, undeploy, connect template
version | version NIFI |
---|---|
- | 0.X |
version 1.1.X download it from Maven Central | 1.0.X : to test 1.1.X : OK, Build on api nifi 1.1.0 with nifi-swagger-client 1.1.0.x 1.2.X : OK 1.3.X : OK 1.4.X : OK |
version 1.5.X download it from Maven Central | 1.5.X : OK, Build on api nifi 1.5.0 with nifi-swagger-client 1.5.0.x |
usage: java -jar nifi-deploy-config-1.1.15.jar [OPTIONS]
-h,--help Usage description
-b,--branch <arg> Target process group (must begin by root) : root > my group > my sub group (default : root)
-m,--mode <arg> mandatory, possible values : updateConfig/extractConfig/deployTemplate/undeploy
-c,--conf <arg> mandatory if mode in [updateConfig, extractConfig, deployTemplate] : configuration file
-n,--nifi <arg> mandatory : Nifi URL (ex : http://localhost:8080/nifi-api)
For more options see Chapter Advanced options
Requirement : You must have java 8 or higher installed on your machine
1 ) Create a template on nifi :
with this rule : each processor and each controller in a process group must have a unique name.
-
Download it
-
Extract a sample configuration with the command
java -jar nifi-deploy-config-1.1.15.jar \
-nifi http://ip-nifi-dev:8080/nifi-api \
-branch "root>My Group>My Subgroup" \
-conf /tmp/config.json \
-mode extractConfig
1a) undeploy the old version with the command
java -jar nifi-deploy-config-1.1.15.jar \
-nifi http://ip-nifi-prod:8080/nifi-api \
-branch "root>My group>My Subgroup" \
-m undeploy
1b) deploy the template with the command
java -jar nifi-deploy-config-1.1.15.jar \
-nifi http://ip-nifi-prod:8080/nifi-api \
-branch "root>My group>My Subgroup" \
-conf /tmp/my_template.xml \
-m deployTemplate
- update the production configuration with the command
java -jar nifi-deploy-config-1.1.15.jar \
-nifi http://ip-nifi-prod:8080/nifi-api \
-branch "root>My group>My Subgroup" \
-conf /tmp/PROD_config.json \
-mode updateConfig
Everything !
You can find all properties in your extraction file. Now just configure it with the production properties and update your production with the mode updateConfig.
Configuration is based on "name" properties, remember that each processor in a group must have a unique name.
It is also possible to create connections between process groups by adding a connections
JSON element to the configuration. The shortest route between the two process groups will be selected, with output and input ports named according to the connection's name used automatically, and created if they do not already exist. Note that it is only possible to connect to input and output ports, so this functionality is typically best used by including an input or output port already wired in to the template that is named the same as the connection.
sample :
{
"processors": [
{
"name": "ExecuteSQL",
"config": {
"properties": {
"Database Connection Pooling Service": "85f67694-015d-1000-5071-8cd46e8b2e47",
"Max Wait Time": "5 seconds",
"dbf-normalize": "false",
"dbf-user-logical-types": null,
"dbf-default-precision": null,
"dbf-default-scale": null
},
"schedulingPeriod": "0 sec",
"schedulingStrategy": "TIMER_DRIVEN",
"executionNode": "ALL",
"penaltyDuration": "30 sec",
"yieldDuration": "1 sec",
"bulletinLevel": "WARN",
"runDurationMillis": 0,
"concurrentlySchedulableTaskCount": 1,
"comments": "",
"lossTolerant": false
}
},
{
"name": "PutFile",
"config": {
"properties": {
"Directory": "c:\\tmp",
"Conflict Resolution Strategy": "fail",
"Create Missing Directories": "true"
},
"schedulingPeriod": "0 sec",
"schedulingStrategy": "TIMER_DRIVEN",
"executionNode": "ALL",
"penaltyDuration": "30 sec",
"yieldDuration": "1 sec",
"bulletinLevel": "WARN",
"runDurationMillis": 0,
"concurrentlySchedulableTaskCount": 1,
"comments": "",
"lossTolerant": false
}
}
],
"controllerServices": [
{
"name": "DBCPConnectionPool",
"properties": {
"Database Connection URL": "localhost:8080",
"Database Driver Class Name": "org.test",
"Password": "********",
"Max Total Connections": "3"
}
}
],
"connections": [
{
"name": "front_end_1",
"source": "root > front_end > front_end_1",
"destination": "root > back_end > back_end_1"
}
],
"name": "testController"
}
java -jar nifi-deploy-config-1.1.15.jar \
-nifi http://ip-nifi-dev:8080/nifi-api \
-branch "root>my group>my subgroup" \
-conf /tmp/test2.json \
-mode extractConfig
java -jar nifi-deploy-config-1.1.15.jar \
-nifi http://ip-nifi-prod:8080/nifi-api \
-branch "root>my group>my subgroup" \
-conf /tmp/test2.json \
-mode updateConfig
java -jar nifi-deploy-config-1.1.15.jar \
-nifi http://ip-nifi-prod:8080/nifi-api \
-branch "root>my group>my subgroup" \
-conf /tmp/my_template.xml \
-m deployTemplate
java -jar nifi-deploy-config-1.1.15.jar \
-nifi http://ip-nifi-prod:8080/nifi-api \
-branch "root>my group>my subgroup" \
-m undeploy
force mode actived
java -jar nifi-deploy-config-1.1.15.jar \
-nifi http://ip-nifi-prod:8080/nifi-api \
-branch "root>my group>my subgroup" \
-m undeploy
-f
-timeout 600
-interval 10
java -jar nifi-deploy-config-1.1.15.jar \
-user my_username \
-password my_password \
-nifi http://ip-nifi-prod:8080/nifi-api \
-branch "root>my group>my subgroup" \
-conf /tmp/test2.json \
-m updateConfig
java -Djava.security.krb5.conf=/etc/krb5.conf \
-Djavax.security.auth.useSubjectCredsOy=false \
-jar nifi-deploy-config-1.1.15.jar \
-accessFromTicket \
-nifi http://ip-nifi-prod:8080/nifi-api \
-branch "root>my group>my subgroup" \
-conf /tmp/test2.json \
-m updateConfig
-timeout <arg> allows specifying the polling timeout in second (defaut 120 seconds); negative values indicate no timeout
-interval <arg> allows specifying the polling interval in second (default 2 seconds)
-password <arg> password for access via username/password, then user is mandatory
-user <arg> user name for access via username/password, then password is mandatory
-accessFromTicket Access via Kerberos ticket exchange / SPNEGO negotiation
-noVerifySsl turn off ssl verification certificat
For accessFromTicket option, if you want use access via Kerberos ticket exchange / SPNEGO negotiation ; You must configure system properties java.security.krb5.conf (see https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/tutorials/KerberosReq.html) and javax.security.auth.useSubjectCredsOy to false. Sample access via Kerberos ticket exchange / SPNEGO negotiation
-connectionTimeout <arg> configure api client connection timeout (default 10 seconds)
-readTimeout <arg> configure api client read timeout (default 10 seconds)
-writeTimeout <arg> configure api client write timeout (default 10 seconds)
-placeWidth <arg> width of place for installing group (default 1935 : 430 * (4 + 1/2) = 4 pro line)
-startPosition <arg> starting position for the place for installing group, format x,y (default : 0,0)
-f,--force turn on force mode : empty queue after timeout
-noStartProcessors turn off auto start of the processors after update of the config
-enableDebugMode turn on debugging mode of the underlying API library
-keepTemplate keep template after installation (default false)
By default, nifi-config uses the controller declared on the parent group that has the same name, if any then deletes the controller declaration on the child group, otherwise uses the controller of the group.
If you want to use a controller declared on parent group without updating it, just declare the controller with no property on json file :
"controllerServices": [
{
"name": "DBCPConnectionPool"
}
add version management that undeploys the old version automatically (with a version # in comment?)
All ideas are welcome.
If you are behind a proxy, try adding these system properties on the command line:
-Dhttp.proxyHost=myproxy.mycompany.com -Dhttp.proxyPort=3128
See more at http://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html