Skip to content

Commit cfbdd91

Browse files
paxtonharepaxtonhare
committed
Merge branch 'release/1.0.0-rc.4'
2 parents 0dd796e + 341b4f6 commit cfbdd91

File tree

37 files changed

+861
-94
lines changed

37 files changed

+861
-94
lines changed

‎.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,3 +19,4 @@ assetInstallTime.properties
1919
/quick-start/plugins
2020
flow.properties
2121
node_modules/
22+
.tmp/

‎CHANGELOG.md

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,19 @@
11
# Change Log
22

3-
## [v1.0.0-rc.3](https://github.com/marklogic/marklogic-data-hub/tree/v1.0.0-rc.3)
3+
## [v1.0.0-rc.4](https://github.com/marklogic/marklogic-data-hub/tree/v1.0.0-rc.4)
44

5+
[Full Changelog](https://github.com/marklogic/marklogic-data-hub/compare/v1.0.0-rc.3...v1.0.0-rc.4)
6+
7+
**Implemented enhancements:**
8+
9+
- Add Spring Batch Example [\#263](https://github.com/marklogic/marklogic-data-hub/issues/263)
10+
11+
**Closed issues:**
12+
13+
- MLCP Command for CSV files is generated incorrectly [\#261](https://github.com/marklogic/marklogic-data-hub/issues/261)
14+
- update examples now that rc.2 is out [\#247](https://github.com/marklogic/marklogic-data-hub/issues/247)
15+
16+
## [v1.0.0-rc.3](https://github.com/marklogic/marklogic-data-hub/tree/v1.0.0-rc.3) (2016-08-25)
517
[Full Changelog](https://github.com/marklogic/marklogic-data-hub/compare/v1.0.0-rc.2...v1.0.0-rc.3)
618

719
**Fixed bugs:**
@@ -51,6 +63,7 @@
5163
- Performance tracing [\#193](https://github.com/marklogic/marklogic-data-hub/issues/193)
5264
- Refactor the Spring Boot API [\#145](https://github.com/marklogic/marklogic-data-hub/issues/145)
5365
- Support index configuration as a part of pushbutton deploy. [\#10](https://github.com/marklogic/marklogic-data-hub/issues/10)
66+
- UI rework based on new screen designs [\#70](https://github.com/marklogic/marklogic-data-hub/issues/70)
5467

5568
## [v1.0.0-beta.6](https://github.com/marklogic/marklogic-data-hub/tree/v1.0.0-beta.6) (2016-06-20)
5669
[Full Changelog](https://github.com/marklogic/marklogic-data-hub/compare/v1.0.0-beta.5...v1.0.0-beta.6)

‎examples/spring-batch/.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
build/
2+
.gradle
3+
.tmp

‎examples/spring-batch/README.md

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
# Spring Batch Job using the Data Hub
2+
3+
This example demonstrates how to run a custom Spring Batch job against the Data Hub Framework.
4+
5+
Learning [Spring Batch](http://docs.spring.io/spring-batch/reference/html/spring-batch-intro.html) is beyond the scope of this README. But let's pretend you know enough to be dangerous.
6+
7+
Now you want to use Spring Batch to load a bunch of data into MarkLogic. Maybe that data is coming from a message queue. Maybe it's a bunch of files in a folder. It can be anything really.
8+
9+
## What's the Big Idea?
10+
The idea is pretty simple. You read the data, do a little processing (maybe), and then write it into MarkLogic. But to properly integrate with the Data Hub Framework you will want to run your data through an [input flow](https://github.com/marklogic/marklogic-data-hub/wiki/The-MarkLogic-Data-Hub-Overview#ingest).
11+
12+
## How does it work?
13+
This example includes a sample Spring Boot Configuration [LoadAndRunFlow.java](https://github.com/marklogic/marklogic-data-hub/blob/develop/examples/spring-batch/src/main/java/example/LoadAndRunFlow.java) that configures a job to ingest some xml files and run a flow.
14+
15+
This example depends on a runtime class **com.marklogic.spring.batch.hub.HubJobRunner** that is responsible for reading command line parameters and connects to the Data Hub by reading your gradle project files.
16+
17+
## How do I Run this Example?
18+
19+
First you compile it.
20+
21+
`gradle installDist`
22+
23+
Then you launch it.
24+
25+
`./run.sh`
26+
27+
28+
## How do I add this to my existing Data Hub Project?
29+
30+
This is an example where the Data Hub Project artifacts have already been initialized. If you are wanting to add this ability to existing Data Hub Projects then you simply need to modify the build.gradle file.
31+
32+
```gradle
33+
plugins {
34+
// existing ids...
35+
...
36+
37+
// add application
38+
id 'application'
39+
}
40+
41+
dependencies {
42+
// existing dependencies
43+
44+
// add this one:
45+
compile "com.marklogic:marklogic-spring-batch-core:0.6.0"
46+
}
47+
48+
49+
// add the distributions section
50+
distributions {
51+
main {
52+
baseName = 'baseJob'
53+
}
54+
}
55+
56+
// add the mainClassName to specify the HubJobRunner
57+
mainClassName = "com.marklogic.spring.batch.hub.HubJobRunner"
58+
59+
```
60+
61+
Then drop in your custom Java Config class in src/main/java/.....
62+
63+
Next you simply Compile your code with `gradle installDist`.
64+
65+
Then you can run take a look at the [run.sh script](https://github.com/marklogic/marklogic-data-hub/blob/develop/examples/spring-batch/run.sh) to see how to run your custom config.
66+
67+
Note that this is not the only way to run it. It's merely the easiest. Java Ninjas can directly call the main() function of the [HubJobRunner class](https://github.com/marklogic/marklogic-data-hub/blob/develop/marklogic-data-hub/src/main/java/com/marklogic/spring/batch/hub/HubJobRunner.java). Or you can make your own class to start up Spring Batch by reading the HubJobRunner code and doing something similar.

‎examples/spring-batch/input/bigbird.xml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<monster>
3+
<name>BigBird</name>
4+
<color>yellow</color>
5+
</monster>

‎examples/spring-batch/input/elmo.xml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<monster>
3+
<name>Elmo</name>
4+
<color>red</color>
5+
</monster>

‎examples/spring-batch/input/grover.xml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<monster>
3+
<name>Grover</name>
4+
<color>blue</color>
5+
</monster>

‎examples/spring-batch/marklogic-config/databases/final-database.json

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
{
2+
"database-name": "%%mlFinalDbName%%",
3+
"range-element-index": [
4+
{
5+
"scalar-type": "dateTime",
6+
"namespace-uri": "http://marklogic.com/xdmp/dls",
7+
"localname": "created",
8+
"collation": "",
9+
"range-value-positions": false,
10+
"invalid-values": "reject"
11+
},
12+
{
13+
"scalar-type": "dateTime",
14+
"namespace-uri": "http://marklogic.com/xdmp/dls",
15+
"localname": "replaced",
16+
"collation": "",
17+
"range-value-positions": false,
18+
"invalid-values": "reject"
19+
},
20+
{
21+
"scalar-type": "unsignedLong",
22+
"namespace-uri": "http://marklogic.com/xdmp/dls",
23+
"localname": "version-id",
24+
"collation": "",
25+
"range-value-positions": false,
26+
"invalid-values": "reject"
27+
},
28+
{
29+
"scalar-type": "unsignedInt",
30+
"namespace-uri": "http://marklogic.com/data-hub/debug",
31+
"localname": "is-debugging-enabled",
32+
"collation": "",
33+
"range-value-positions": false,
34+
"invalid-values": "reject"
35+
}
36+
],
37+
"schema-database": "%%SCHEMAS_DATABASE%%",
38+
"triggers-database": "%%TRIGGERS_DATABASE%%",
39+
"triple-index": true,
40+
"collection-lexicon": true,
41+
"uri-lexicon": true
42+
}

‎examples/spring-batch/marklogic-config/databases/job-database.json

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
{
2+
"database-name": "%%mlJobDbName%%",
3+
"range-element-index": [
4+
{
5+
"scalar-type": "dateTime",
6+
"namespace-uri": "http://marklogic.com/xdmp/dls",
7+
"localname": "created",
8+
"collation": "",
9+
"range-value-positions": false,
10+
"invalid-values": "reject"
11+
},
12+
{
13+
"scalar-type": "dateTime",
14+
"namespace-uri": "http://marklogic.com/xdmp/dls",
15+
"localname": "replaced",
16+
"collation": "",
17+
"range-value-positions": false,
18+
"invalid-values": "reject"
19+
},
20+
{
21+
"scalar-type": "unsignedLong",
22+
"namespace-uri": "http://marklogic.com/xdmp/dls",
23+
"localname": "version-id",
24+
"collation": "",
25+
"range-value-positions": false,
26+
"invalid-values": "reject"
27+
},
28+
{
29+
"scalar-type": "unsignedLong",
30+
"namespace-uri": "http://marklogic.com/spring-batch",
31+
"localname": "id",
32+
"collation": "",
33+
"range-value-positions": false,
34+
"invalid-values": "reject"
35+
36+
},
37+
{
38+
"scalar-type": "string",
39+
"namespace-uri": "http://marklogic.com/spring-batch",
40+
"localname": "jobName",
41+
"collation": "http://marklogic.com/collation/",
42+
"range-value-positions": false,
43+
"invalid-values": "reject"
44+
},
45+
{
46+
"scalar-type": "dateTime",
47+
"namespace-uri": "http://marklogic.com/spring-batch",
48+
"localname": "createDateTime",
49+
"collation": "",
50+
"range-value-positions": false,
51+
"invalid-values": "reject"
52+
}
53+
],
54+
"path-namespace": [
55+
{
56+
"prefix": "msb",
57+
"namespace-uri": "http://marklogic.com/spring-batch"
58+
}
59+
],
60+
"range-path-index": [
61+
{
62+
"scalar-type": "unsignedLong",
63+
"collation": "",
64+
"path-expression": "/msb:mlJobInstance/msb:jobExecutions/msb:jobExecution/msb:id",
65+
"range-value-positions": false,
66+
"invalid-values": "reject"
67+
}
68+
],
69+
"schema-database": "%%SCHEMAS_DATABASE%%",
70+
"triggers-database": "%%TRIGGERS_DATABASE%%",
71+
"triple-index": true,
72+
"collection-lexicon": true,
73+
"uri-lexicon": true,
74+
"maintain-last-modified": true
75+
}

‎examples/spring-batch/marklogic-config/databases/modules-database.json

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"database-name": "%%mlModulesDbName%%",
3+
"collection-lexicon": true,
4+
"uri-lexicon": true
5+
}

‎examples/spring-batch/marklogic-config/databases/schemas-database.json

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
{
2+
"database-name": "%%mlSchemasDbName%%"
3+
}

‎examples/spring-batch/marklogic-config/databases/staging-database.json

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
{
2+
"database-name": "%%mlStagingDbName%%",
3+
"range-element-index": [
4+
{
5+
"scalar-type": "dateTime",
6+
"namespace-uri": "http://marklogic.com/xdmp/dls",
7+
"localname": "created",
8+
"collation": "",
9+
"range-value-positions": false,
10+
"invalid-values": "reject"
11+
},
12+
{
13+
"scalar-type": "dateTime",
14+
"namespace-uri": "http://marklogic.com/xdmp/dls",
15+
"localname": "replaced",
16+
"collation": "",
17+
"range-value-positions": false,
18+
"invalid-values": "reject"
19+
},
20+
{
21+
"scalar-type": "unsignedLong",
22+
"namespace-uri": "http://marklogic.com/xdmp/dls",
23+
"localname": "version-id",
24+
"collation": "",
25+
"range-value-positions": false,
26+
"invalid-values": "reject"
27+
},
28+
{
29+
"scalar-type": "unsignedInt",
30+
"namespace-uri": "http://marklogic.com/data-hub/trace",
31+
"localname": "is-tracing-enabled",
32+
"collation": "",
33+
"range-value-positions": false,
34+
"invalid-values": "reject"
35+
},
36+
{
37+
"scalar-type": "unsignedInt",
38+
"namespace-uri": "http://marklogic.com/data-hub/debug",
39+
"localname": "is-debugging-enabled",
40+
"collation": "",
41+
"range-value-positions": false,
42+
"invalid-values": "reject"
43+
}
44+
],
45+
"schema-database": "%%SCHEMAS_DATABASE%%",
46+
"triggers-database": "%%TRIGGERS_DATABASE%%",
47+
"triple-index": true,
48+
"collection-lexicon": true,
49+
"uri-lexicon": true
50+
}

‎examples/spring-batch/marklogic-config/databases/trace-database.json

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
{
2+
"database-name": "%%mlTraceDbName%%",
3+
"range-element-index": [
4+
{
5+
"scalar-type": "dateTime",
6+
"namespace-uri": "http://marklogic.com/xdmp/dls",
7+
"localname": "created",
8+
"collation": "",
9+
"range-value-positions": false,
10+
"invalid-values": "reject"
11+
},
12+
{
13+
"scalar-type": "dateTime",
14+
"namespace-uri": "http://marklogic.com/xdmp/dls",
15+
"localname": "replaced",
16+
"collation": "",
17+
"range-value-positions": false,
18+
"invalid-values": "reject"
19+
},
20+
{
21+
"scalar-type": "unsignedLong",
22+
"namespace-uri": "http://marklogic.com/xdmp/dls",
23+
"localname": "version-id",
24+
"collation": "",
25+
"range-value-positions": false,
26+
"invalid-values": "reject"
27+
},
28+
{
29+
"scalar-type": "string",
30+
"namespace-uri": "",
31+
"localname": "traceId",
32+
"collation": "http://marklogic.com/collation/codepoint",
33+
"range-value-positions": false,
34+
"invalid-values": "reject"
35+
},
36+
{
37+
"scalar-type": "string",
38+
"namespace-uri": "",
39+
"localname": "identifier",
40+
"collation": "http://marklogic.com/collation/codepoint",
41+
"range-value-positions": false,
42+
"invalid-values": "reject"
43+
},
44+
{
45+
"scalar-type": "unsignedInt",
46+
"namespace-uri": "http://marklogic.com/data-hub/debug",
47+
"localname": "is-debugging-enabled",
48+
"collation": "",
49+
"range-value-positions": false,
50+
"invalid-values": "reject"
51+
},
52+
{
53+
"scalar-type": "dateTime",
54+
"namespace-uri": "",
55+
"localname": "created",
56+
"collation": "",
57+
"range-value-positions": false,
58+
"invalid-values": "reject"
59+
}
60+
],
61+
"schema-database": "%%SCHEMAS_DATABASE%%",
62+
"triggers-database": "%%TRIGGERS_DATABASE%%",
63+
"triple-index": true,
64+
"collection-lexicon": true,
65+
"uri-lexicon": true
66+
}

‎examples/spring-batch/marklogic-config/databases/triggers-database.json

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
{
2+
"database-name": "%%mlTriggersDbName%%"
3+
}

‎examples/spring-batch/marklogic-config/servers/final-server.json

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
{
2+
"server-name": "%%mlFinalAppserverName%%",
3+
"server-type": "http",
4+
"root": "/",
5+
"group-name": "%%GROUP%%",
6+
"port": %%mlFinalPort%%,
7+
"modules-database": "%%mlModulesDbName%%",
8+
"content-database": "%%mlFinalDbName%%",
9+
"authentication": "digest",
10+
"default-error-format": "json",
11+
"error-handler": "/MarkLogic/rest-api/error-handler.xqy",
12+
"url-rewriter": "/MarkLogic/rest-api/rewriter.xml",
13+
"rewrite-resolves-globally": true
14+
}

0 commit comments

Comments
 (0)