Consider JSON Lines rather than JSON

[timgdavies]

Dealing with lots of data in single JSON objects can bring a big overhead.

We should consider using JSONLines, which allows easier stream processing, and use of command line tools (e.g. simply grep over files) to work with large collections data.

[timgdavies]

There will be a couple of issues to address here:

(1) Do we use JSON Lines , JSON-L or NDJSON

JSON-L appears to be an expired internet draft

NDJSON and JSONLines appear to be (almost?) the same, and there is discussion of whether they should be merged.

It looks like the difference is ultimately very limited, with perhaps something about whether comments or blank lines are accepted or not. Assuming, if there are differing opinions here, that there will be greater tooling support for files without comments or blank lines, that we should apply this constraints to BODS.

I can't see any of the above formats having made it into IANA Media Types list - and it looks at a quick glance like tooling that reads one should read the others.

This might mean this is more a case of referring to the most stable and clear spec, rather than choosing between different formats.

(2) What exactly a line should contain

For example, would each line still require a statementGroup as it's root element. Or could it even allow an array of statementGroups?

Would we still enforce the constraint in this case that a statementGroup should contain all relevant statements, or would we allow that a statementGroup might be repeated on multiple lines, with the merged set of these containing all relevant statements.

(3) Do we have JSON Lines instead of, or in addition to a full object serialisation?

E.g. we could say that publication can be either:

• A JSON document, with a statementGroups array as the root element or
• A JSON Lines document, where each line contains a statementGroup containing one or more statements.

[timgdavies]

We allow for JSON Lines in our serialisation guidance. http://standard.openownership.org/en/0.2.0/schema/guidance/serialization.html