JSON for the curl tool

[bagder]

If you feel you want to comment on or contribute to the brainstorm around json support. Here's your chance!

[chrisdickinson]

I saw that JSON display is handled by piping to jq, but some users are attracted to tools like HTTPie for pretty JSON output in interactive contexts. This might be a bit of solutioneering on my part, but it'd be interesting if curl supported the PAGER env var and fed results to it when run in a TTY context. If PAGER is not set, a default PAGER command could be provided to allow for pretty-printing functionality JSON output?

[bagder]

PAGER would be a completely different and independent feature so let's not discuss that in this thread.

[seripap]

Evolving a bit from your ideas- maybe combining bracket annotations to create lists versus :list

--jp []="Monday,Tuesday,Wednesday,Thursday"

[
  "Monday",
  "Tuesday",
  "Wednesday",
  "Thursday"
]

Since these are probably the same --jp prime[]=13

{
   "prime": [ 13 ] 
}

[bagder]

Thanks that seems reasonable if we go with this approach. My initial take there was just to bring out some ideas to show a direction and I can't say they were "complete" in my mind. I'm personally doing far too little JSON posting myself to be the one to make it perfect. I would need assistance on what is needed (and what can/should be left) out. I'm especially looking for a way to design something basic that works for an initial take with simple JSON but that can (optionally) get expanded and more complete later.

I don't expect nor desire curl to completely be able to replace very capable json tools such as jq and jo.

Update: I updated the wiki to use this style now.

[seripap]

Great- I think your initial direction is simple and to the point.

Perhaps not assume spaces as delimiters- Given --jp '[]="Monday, Tuesday, Wednesday, Thursday"', my expected output would be;

[
  "Monday",
  " Tuesday",
  " Wednesday",
  " Thursday"
]

A nice addition would be the ability to do something like this --jp map=europe --jp prime[]=13 --jp prime[]=17 --jp target[x]=-10 --jp target[y]=32 with one command; --jp 'map=europe prime[13,17] target[x=13,y=32]'

[dch]

It's important to remember that:

• it should be less work including shell quoting rules, than just writing JSON
• not end up just writing pseudo-json anyway with a different syntax
• usage should be intuitive, assuming a reasonable familiarity with json
• interpolating variables from the shell should be trivial and not require more quotes than necessary
• generally I expect that {} or [] is the smallest valid JSON text,
  and "a" : true is not. YMMV here depending on which JSON standard you choose, and precisely how you apply it.

Our worst-case effort is --json ["Monday", "Tuesday", "Wednesday", "Thursday"] which is pure valid json. We can (in this case) leave out all the " and the list can be inferred. And finally, we can leave out the [] entirely here as a list of items can only be a json array.

• for an object, (set of key:value pairs) it's perhaps not so clear?

--json {"Monday":1, "Tuesday":2}

could that be simplified? monday:1 tuesday:2 is probably OK.

If you need to have variables in the array, the "" just make it harder. Not a lot harder, through. But then, if you require the quotes, and the [] then why not just write the actual json directly, instead of having to remember an additional syntax?

[bagder]

Initial trial balloon for --json in #8314

[timmc]

I think this is the one part I would actually use -- I'd rather compose and manipulate JSON with a JSON-specific tool, but I frequently find myself wanting a shorthand in curl for the JSON content-type.

[kernalphage]

+1, the -jp flag may or may not end up with a lot of feature creep, but the --json flag hits 80% of my "POST JSON to test an endpoint" use cases, especially if it works the same way as -d with stdin and files.

I actually just wrote a surprisingly similar alias to set the content-type yesterday before finding this thread.

[bagder]

This --json as implemented right now works like -d for stdin and files, and even shares the code doing that. (To be exact, it actually does what --data-binary does for reading files - it will read and send the exact contents as-is. -d strips off CRs and LFs when reading from files.)

[yonatan8070]

I have a concern about data types with --jp.

In the example --jp a=b --jp c=d --jp e=2 --jp f=false the output is this:

{
   "a": "b", // string
   "c": "d", // string
   "e": 2, // integer
   "f": false // boolean
}

My concern is that there doesn't seem to be a way to specify what type you want, what would I do if i wanted to send "557" or "true"?

I suggest adding an optional way to specify types, here's an example of how I see it:

--jps a=4 // Will always output a string, even if the input is a number or boolean
--jpn b=haha // Error: b must be a number
--jpb c=54.5 // Error: c must be a boolean

[dch]

I think this covers 90% of use case from the perspective of "I'm a terminal user and I want to send some JSON but not run an actual scripting language". If I've missed something, happy to add it.

Per IRC it is very important to pick a JSON "standard" to comply with. i-json or ECMAscript 404 perhaps. Things like duplicate key behaviour is not obvious to many people, but has been exploited for security vulnerabilities in the past.

A plain = or ":" as a Do What I Mean interpration is sensible - anything that looks like a boolean or a number would be treated as such, things that look like arrays should end up that way, nested stuff can be done via . separators and object keys are merged.

• a:4 -> {"a": 4} 0-9.- and Ee+=0-9 as well per https://www.json.org/json-en.html
• a:true -> {"a": true} ditto for false, null as well
• a:foo -> {"a": "foo"}
• a:foo,bar,baz -> {"a": ["foo", "bar", "baz"]}
• a:1,2,3 -> {"a": [1,2,3]}
• a:1 b:true -> {"a": 1, "b":true} implied top-level JSON object
• nested.a:1 nested.b:2 -> {"nested": {"a":1, "b":2}} merge the keys
• a:true a:false -> {"a": false} drop duplicate object keys
• a:@/path/to/file -> {"a": <contents_of_file>}

This assumes that you know the contents of whatever shell variable you're injecting here, and that it contains the data type you expect. If you send a bare string, but it has "," inside, this would assume you mean a list.

For all other cases, what you want is to force an auto-detected type/field to be treated as a string. Add another operator for raw fields - a plain = perhaps?

• a=4 -> {"a": "4"} a string
• a=true -> {"a": "true"} a string
• a=foo,bar,baz -> {"a":"foo,bar,baz"} a single string, not an array
• a=@/path/to/file -> {"a": <contents_of_file>} assumes file contains a valid JSON value

For generating arrays and more complex nested json objects, I think we need to be pragmatic - if you're building a small json object for an API call, then curl should be able to handle it. For larger or more elaborate structures use a more appropriate scripting language instead.

• @/path/to/file -> use data from file as elsewehere in curl, assumed to be valid JSON

I also expect that people will assume that they can do both "binary" PUT/POST, and form encoded data.

Regarding setting headers, I would like to see both content-type: application/json and accept: application/json set by default.

See some prior art at bat and its fork gurl.

[bagder]

I really like this syntax proposal. This could create a fairly straight forward way for users to generate basic JSON from a command prompt and script, without getting completely lost in quote problems as often is the case otherwise. (And a simple way that doesn't cover all the use cases and complexities is still okay, as mentioned already.)

My plan is now to first split off the --json option and proceed with that initially (and that PR is already close to land), since that's simpler and basically just works as a shortcut. The separate work on an option that can produce JSON is probably going to become a larger work, and also potentially cause more friction that trigger opinions and emotions.

This also means that we haven't yet decided on if we actually will merge the --jp proposal.

[docteurklein]

I'd argue that as you prefer to delegate output parsing to jq, you could prefer to delegate input building to jo (or something similar).
It looks similar to what you're trying to achieve with --jp.

Maybe the automatic configuration of Content-Type is a good idea, IDK.

Another problem I see with letting the curl monolith entirely build the json body is that it's not composable.
Is there a way to build a pipeline that would allow to modify the json incrementally, something like:

cat body.json | jq '.foo |= "baz"' | curl --jp 'baz=bar' ...

[bagder]

That's not a problem per se, that would just be a known limitation. Whatever ambition we end up going with, I don't think anyone thinks that the curl native options will (completely) replace jo and jq. For the easy and trivial cases it could perhaps.

[oliv3r]

After first having raid some of the mailing post messages, I was like, what are you guys on about, of course is having a json cmdline great. It's not that different from what the other encoding options do, right? I hadn't yet looked at the proposal.

As a user of cURL to do API requests mostly, I also ran into inconveniences when posting JSON data. Now forgive me if there IS a way to properly escape the JSON payload and I was just ignorant ;)

The issue I ran into initially was escaping new lines 0, but lets also add a brief summary first.

When doing REST API calls, we suffer two issues really, one that's caused by the shell, the other by the content. The shell we can do little about, it will consume quotes in certain forms. For completeness sake, I'm speaking of

$ curl ... {"key": "value"}

won't work, because the shell will eat the quotes and you end up with invalid JSON, and so obviously a cheap trick is to quote the whole thing (which very often people will do anyway.

$ curl ... '{"key": "value"}'

This then of course breaks again once you want to do actually some fun stuff, like including variables that the shell needs to expand, and so you end up with having to escape the double quotes which are required as part of the JSON format (and so it might be argued that it is a more accurate representation :p). This is just shell shenanigans that cURL can't do much about anyway, other then 'deciding' to properly quote things in the payload, which makes this feature explode from MVP complexity.

$ curl ... "{\"key\": \"${value}\"}"

Now to the beefy parts. JSON actually doesn't allow you to send various characters, and they need to properly be escaped. My feeling is, this is what most people complain about when talking about 'having to escape JSON'. When looking at a decently common JSON implementation, we can also see this clear as day 1 what in addition needs to be escaped. If a common JSON parser does this, would it not make sense that also cURL does this (Again, if there is a different way to escape exactly this set of characters in a different way, ignore my ignorance and delete my post :p)?

I'm sure the above can easily be solved using jq and what not, but I'm a CI user, and not wanting to have to build or use random containers, when ALL I want to do is do a few REST requests, I prefer to use the official cURL Docker container and so I end up using shenanigans like 2.

To get back to the original proposal, I think that including the proper content type by default behind --json (or maybe I'm talking about --data json as mentioned elsewhere ...) makes good sense, as that is probably always needed when sending json, and I also see the advantage of doing --jp to actually parse json content. But this will be a deep rabbit hole quickly. I guess for the MVP having only basic support is fine, but once you get to speak of more complex queries ... And even here, what if --jp key="${value}" also contains characters that need to be escaped ...

Further more, another upcomming hot potato is graphQL posting over REST. While still 100% JSON, it is a bit ugly, in that it looks like:

$ curl ... --json '{"query": "<graphql content>"}'

One could argue though, that once the popularity of graphql increases and json parsing already exists within cURL, this could easily be wrapped internally, so that

$ curl ... --quaphql '<graphql content>'

gets turned into the earlier internally (BUT I'm not certain if the keyname (query) is a GitLab specific thing or a graphql library specific thing or a graphql spec thing).

Sorry for the very long post @bagder, but I hope this help clarify some problems and thoughts?

P.S. I do fully agree this should only be about input. If a consumer actually needs to process the output, for that, there are better ways as things become very complex very quickly to support that and you'd end up just building a mini jq for output handling. While nice 'from the JSON dump give me only value of keyX, I think here it IS better to follow the UNIX philosophy :p

[jzakrzewski]

Well, I don't like the input transformation already because it is not what curl is about in my opinion. Shortcuts for setting the right headers? - sure and very welcome. But where do we draw the line with input support? Why only JSON? Why not XML, YAML, or something else?
Nevertheless, I'm not going to spread negativity about this feature. Said it once now and that's enough.

[whiteinge]

I like the idea of --json as a convenience flag when making a request. Although many APIs default to JSON for the response nowadays it's still common to require an Accept header for that determination (any API that does any kind of content-negotiation). Anecdotally maybe half the APIs I frequent require both Content-Type and Accept headers to be set to application/json for a full JSON exchange.

Seconding @dch for a convenience flag to set both headers. I'm not sure --json should do both but how about a second, longer flag? Perhaps --json-full or --json-repreq or --json-request-response or --json-both or --json-bidirectional ?

[bagder]

If we would set the Accept: header as well, shouldn't it then be set to Accept: application/json, */* to provide a fallback ?

[jzakrzewski]

That's actually a good question and probably without a straight-forward answer. Sometimes I'd prefer the request to give me any answer, and sometimes I'd rather get 406 instead of getting weird errors a hundred lines down the script and having to debug that.

[bagder]

I'm not sure I understand. Are you saying the default accept header will give you the 406 you want while application/json, */* wouldn't? Also: isn't it a bit problematic to write scripts that then assume a 4xx error when things are fine?

[whiteinge]

@bagder If I understand your follow-up question correctly (please correct me if not): I think it would indeed be incorrect for a server to return a 406 if the Accept header contained a */* fallback value since the client is effectively asking for JSON or whatever else is available. Since this proposal is for a convenience flag to make curl interactions more quick and less verbose, the primary winners of using that flag are one-off CLI commands or simple shell scripts -- both uses that are unlikely to gracefully handle multiple possible response formats.

For example: Say I was writing a CLI command or a short script that sent the response from curl to jq. I'd check for a 406 and exit the script gracefully (the -f flag works great for this). However if I was writing a longer script that sent the response from curl to either jq or tidy depending on the Content-Type header in the response it wouldn't be much out of my way to just use the full -H flag instead -- in fact, I'd prefer to craft the Accept header specifically for whatever formats my script was written to support.

So for a hypothetical --json-full flag I think a fallback is not desirable -- JSON or bust.

[jzakrzewski]

I meant that sometimes I want to accept JSON and JSON only, so send Accept: application/json and get 406 if that cannot be fulfilled. Other times I'd prefer JSON but can work with whatever and then Accept: application/json, */* is what is needed.

And 406 is not 'fine' - it means I have requested something the server cannot give me. If cannot process anything else than JSON, this is the way to go from my perspective.

[dch]

`Accept: application/json, */*` is what is needed.

This is fine for browsers. It should not be the default for a command line tool that expects to parse the response reliably. Are there actual APIs that people use with curl where this is necessary? I’ve not encountered one where this would be a useful fallback. Or does this come up in pretending to be a browser, maybe in testing?

[bagder]

I don't know, that's why I'm asking. I'm trying to determine what to do about Accept: by default when --json is used!

[jzakrzewski]

I basically would love to have a simple way to specify both. I mean I am perfectly capable of crafting my own Accept header but for quick curl in the terminal it's annoying to have to specify Content-Type and Accept by hand each time.

Nevertheless, if we can have only one, it should probably be just Accept: application/json as this is less surprising than receiving XML ;). Although, there are servers that won't care anyway :(.

[bagder]

For this single option there can only be one default! 😁 Users will still of course be able to override the default when that isn't good enough.

[bagder]

There. Now #8314 defaults to setting both Content-type and Accept while accepting both to be overridden as you'd expect.

[SamuelMarks]

In July I made this project, I can relicense under CC0 if you want… maybe (only question is that its mostly based on your libcurl examples): https://github.com/SamuelMarks/curl-simple-https

In addition to simplified https verb functions, it also has simplified json ones, like so:

struct ServerResponse
https_json_post(CURLU *urlp, const char *body, struct curl_slist *headers) {
    return https_wrapper(urlp, make_request_post, body, set_json_headers(headers));
}

Where that return type is defined as:

struct ServerResponse {
    CURLcode code;
    /* What else would be good? */
    struct curl_slist *headers;
    long status_code;
    const char *body;
};

My code is nothing special, so feel free to just think of this as a concept suggestion; though I can send through a PR if you would be interested.

Can I encourage you to have simple ${protocol}_${verb}_${encoding} functions? - You could probably go further and use an X macro to generate them for all the [relevant] protocols curl supports; alongside a new enum.

Thanks for your consideration

PS: the big issue with my codebase, apart from lack of docs and tests [!!], is that I don't have any principles stance around allocations. Like calling curl_easy_cleanup and curl_global_cleanup everywhere. Oh, and I have touched the multi-interface (yet), but that would also be handy with this new ultra-simplified interface.

[abraakie]

Hi there,
I recently worked on a similar problem and developed CLON (Command Line Object Notation). It's inspired by some ideas of this thread and supports both key based and value based notation. This means a JSON like:

{
    "name":"max",
    "age":30,
    "kids":[
          "kirk",
          "john"
    ],
    "car":{
         "name":"speedo",
         "speed":200,
         "electric":false
    }
}

could be written in object based notation:

[name=max,age=30,kids=[kirk,john],car=[name=speedo,speed=200,electric=false]]

or in key based notation:

[name=max,age=30,kids[]=kirk,kids[]=john,car[name]=speedo,car[speed]=200,car[electric]=false]

Which is especially useful if your JSON structure is deep without many values.
Of course you can also mix both notation techniques.

If anyone is interested here is a link to the (Apache 2 licensed) Java project and here is one to the actual parser that does all the magic.

Maybe this is inspiring/helpful in any way :)

[dfandrich]

Isn't urlon just a convention for structuring URLs? curl supports URLs just fine (it's even in the name).

[nbanb]

Hi
I think it would be a huge progress to add a json parser / json builder to curl (most API are sending replies in json format).
Don't know if it's possible, but why not "simply~" merge curl and jq in an uniq tool and let acces jq from curl with something like 'curl --jq' (and from jq like 'jq --curl') ?
nbanba

[bagder]

Everyone is of course free to experiment with something like that and tell us how it runs and what works and how we could use that going forward!