
Jq – Work with JSON on the command line - preek
http://200ok.ch/jq-my-new-favorite-tool-to-work-with-json-on-the-command-line/index.html
======
haswell
It's always surprising when I learn that this isn't more widely known - it's
such a core part of my daily workflow!

The links to the project aren't immediately obvious, so here they are:

[https://stedolan.github.io/jq/](https://stedolan.github.io/jq/)
[https://jqplay.org/](https://jqplay.org/)

~~~
dozzie
It's always surprising when I learn that people think jq is a rich tool.

[https://metacpan.org/pod/App::RecordStream](https://metacpan.org/pod/App::RecordStream)

~~~
setheron
I remember using recs at Amazon. Is that where it originated ?

~~~
dozzie
I don't know, but supposedly Amazon is somewhat hostile to publishing open
source, so I doubt it.

------
asamarin
I _love_ jq. It's been an incredibly useful tool for me since I discovered it
~6 months ago. However, article mentions "jq -n"; I personally find jq syntax
less appealing when it comes to generating JSON instead of parsing it. For
that particular task, I prefer using "jo":

[https://github.com/jpmens/jo](https://github.com/jpmens/jo)

jq & jo are the Calvin & Hobbes of the shell: lots of fun together!

A few examples:

\- Generate a simple JSON object:

    
    
      $ jo foo=12 bar=true baz=null
      {"foo":12,"bar":true,"baz":"null"}
    

\- Want a nested object?

    
    
      $ jo foo=12 bar=true baz=null thing=$(jo number=57 test=string)
      {"foo":12,"bar":true,"baz":"null","thing":{"number":57,"test":"string"}}
    

\- What about arrays?

    
    
      $ jo thing=$(jo number=57 test=string) anarray=$(jo -a 1 2 3 4 5)
      {"thing":{"number":57,"test":"string"},"anarray":[1,2,3,4,5]}
    

\- Now let's add some jq magic to sum all the values in the array together:

    
    
      $ jo thing=$(jo number=57 test=string) anarray=$(jo -a 1 2 3 4 5) | jq '.anarray | add'
      15
    

Great stuff :)

------
jedisct1
An alternative that completely replaced jq for me: rq
[https://github.com/dflemstr/rq](https://github.com/dflemstr/rq)

Not limited to JSON, and super fast.

~~~
azdle
rq is super great, it's been indespsible for me while working with CBOR. It's
also way more useful than jq if you want to filter down your data. Some
examples:

    
    
        ~> curl -s http://httpbin.org/ip | rq
        {
          "origin": "184.100.112.11"
        }
        ~> curl -s http://httpbin.org/ip | rq 'at origin'
        "184.100.112.11"
        ~> curl -s http://httpbin.org/ip | rq 'at origin | split "." | spread | drop 3'
        "11"
        ~> curl -s http://httpbin.org/ip | rq -C
        ?foriginn184.100.112.11⏎
    

(These are a bit contrived, but that was the only public API that returns JSON
I could think of.)

The only thing I haven't figured out is how to unquote string elements once
filtered down to just that string.

~~~
dflemstr
I'm the author of rq, and it's great to hear about your use-cases!

Feel free to submit a feature request about the raw string use-case, I've been
thinking about it for a while but some people have wanted me to add line
support (maybe -l and -L?) and some people have wanted raw text support (maybe
-r and -R?) while other people still have been talking about TSV/CSV support
(not sure how that would look tbh...). I'm not sure what to do exactly, so
feel free to tell me!

~~~
danso
Aw dang, I wish I knew about this earlier. May have made teaching command-line
data processing to beginners quite a bit easier. Example lesson using Spotify
API (which is still public and free):

[http://www.compciv.org/recipes/data/touring-the-spotify-
api/](http://www.compciv.org/recipes/data/touring-the-spotify-api/)

(just noticed that you used the Spotify API too for the demo :)
[https://github.com/dflemstr/rq/blob/master/doc/demo.md](https://github.com/dflemstr/rq/blob/master/doc/demo.md))

I personally like using jq but not sure how easy-to-grok it is for people who
are new to both the command-line and to serialized data structures in general.
Will have give rq a spin (waiting for it to compile on my machine) but it
looks quite nice, at least for my own uses, which heavily involve YAML.

In terms of TSV/CSV support, you may want to checkout csvjson, which is part
of the csvkit command-line suite:
[http://csvkit.readthedocs.io/en/0.9.1/scripts/csvjson.html](http://csvkit.readthedocs.io/en/0.9.1/scripts/csvjson.html)

csvkit is my go-to tool for data, so much that I rarely ever use SQL for
anything other than when I actually need a database. I suppose if rq were to
be my go-to data-processing tool, I'd probably do this:

    
    
          csvjson mydata.csv | rq ...
    

Unless I'm missing it in the documents, there doesn't appear to be a way to
convert to CSV? That would be helpful. FWIW, csvkit has a tool named in2csv,
which will read from line-delimited JSON:

    
    
         curl -s 'https://api.spotify.com/v1/search?q=rihanna&type=track&limit=50' \
            | rq 'at "tracks.items" | spread | flatMap artists' \
            | in2csv -f ndjson
    
    

[http://csvkit.readthedocs.io/en/stable/scripts/in2csv.html](http://csvkit.readthedocs.io/en/stable/scripts/in2csv.html)

------
kbenson
Some prior submissions:

2 years ago:

[https://news.ycombinator.com/item?id=9446980](https://news.ycombinator.com/item?id=9446980)
(91 comments)

[https://news.ycombinator.com/item?id=7895076](https://news.ycombinator.com/item?id=7895076)
(55 comments)

4 years ago:

[https://news.ycombinator.com/item?id=4679933](https://news.ycombinator.com/item?id=4679933)
(36 comments)

[https://news.ycombinator.com/item?id=4985250](https://news.ycombinator.com/item?id=4985250)
(10 comments)

[https://news.ycombinator.com/item?id=5734683](https://news.ycombinator.com/item?id=5734683)
(29 comments)

------
13of40
I'm sure someone's going to throw a shoe at me for this, but...this sort of
thing is why you want an object-oriented shell.

PS C:\> $x = ConvertFrom-Json -InputObject '[{"banana" : {"stem" : "brown",
"peel" : "yellow", "flesh" : "white"}}]'

PS C:\> $x.banana.peel

yellow

~~~
fabianfabian
$ echo '[{"banana" : {"stem" : "brown", "peel" : "yellow", "flesh" :
"white"}}]' | jq '.[].banana.peel'

"yellow"

~~~
13of40
Come on now, you're missing the point. If the lingua franca of your shell is
objects instead of text, you can import from whatever format you want and work
with it in high fidelity:

PS C:\> $x =
[XML]"<banana><stem>brown</stem><peel>yellow</peel><flesh>white</flesh></banana>"

PS C:\> $x.banana.peel

yellow

PS C:\> $x = ConvertFrom-Json -InputObject '[{"banana" : {"stem" : "brown",
"peel" : "yellow", "flesh" : "white"}}]'

PS C:\> $x.banana.peel

yellow

PS C:\> $x = New-Object PSObject -Property @{banana = New-Object PSObject
-Property @{stem = "brown"; peel = "yellow"; flesh = "white"}}

PS C:\> $x.banana.peel

yellow

PS C:\> $x = @{banana = @{stem = "brown"; peel = "yellow"; flesh = "white"}}

PS C:\> $x.banana.peel

yellow

~~~
fabianfabian
Good point, thanks for clarifying.

------
iopuy
Hey guys, I work with JSON on a daily basis on machines that don't have access
to jq. My advice to you in similar situations is as follows:

To pretty print JSON:

    
    
        cat myjson.json | python -m json.tool > prettyPrinted.json
    

To validate json:

    
    
        cat myjson.json | python -m json.tool
        echo $?
    

If the result is 0 it is well formed and 1 otherwise. You can also pretty
print JSON by opening the scratchpad in firefox and clicking the "Pretty
print" button.

~~~
dajohnson89
Why do you have access to Python, but not jq?

~~~
rhubarbquid
I would guess: a policy against installing unapproved software on production
servers, systems that don't have compilers installed (for security, jq appears
to be C code), or debugging on client/customer machines

------
nikolay
If you use AWS CLI [0], you can embed JMESPath [1] queries via the --query
switch, i.e. you don't ned jq. I was a loyal supporter of it, but switched to
JMESPath [1], and I love its query languages more.

[0]: [https://aws.amazon.com/cli/](https://aws.amazon.com/cli/)

[1]: [http://jmespath.org/](http://jmespath.org/)

~~~
svs
JMESPath is nowhere near us powerful. No unique(), for example, which is very
important when munging JSON

~~~
nikolay
Okay, but in 99.9% of the cases it's powerful enough.

------
dekhn
i want to love this tool but it's almost impossible to do anything
complicated. the docs have a lot of examples but they sort of stop sort

~~~
ajmurmann
I've been using jq quite a bit lately and agree that the docs would benefit
from more complex examples. However, I've been able to get some fairly complex
solutions done with jq. Do you have a concrete example of something you are
trying to do?

~~~
teej
I have server logs in jsonlines format in a file. My goal is to display some
basic information (URL, referrer, time stamp, IP address, and user agent) for
each log line with a particular header set to a specific value. Headers are in
$.request.headers and are an array of key-value pair arrays.

Figuring out this use case from the jq docs or jqplay has been a major
struggle for me. I feel that this use case reflects 90% of what I would want
to use jq for. I feel that if I can't get over this hump just from docs,
there's no way I could justify bringing this tool on board with my team.

~~~
ekimekim
Sounds like you're after the select() filter, which evaluates a boolean
expression and passes the input to the output only if the expression is true.

So your example would be something like: select(.request.headers|map(.[0] ==
"X-My-Header" and .[1] == "my target value")|any)|<pick out some keys to
display>

I realise though that your point wasn't the specific example, it's that the
docs for these kind of cases are poor, and you aren't wrong.

I guess my main advice is to not think of jq as something like cut or sort -
instead, it's more like awk or sed. You can do lots of crazy things with sed,
but it's not immediately obvious how just from reading the man page. These
kinds of tools require a little more time investment but are very powerful.

(edit for code typo)

------
dasil003
jq is awesome, but what stands out to me in the article is the use of curl.
You can shrink those ~200 bytes of curl by 2/3rds using
HTTPie([https://httpie.org/](https://httpie.org/)):

    
    
      http put your.api.endpoint email=your@email.address password=swaggerrocks
    

HTTPie and jq go together like peanut butter and chocolate, except without the
caloric bloat.

~~~
paulddraper
And swift death to those with allergies.

------
edmundhuber
Hi, I wrote a similar tool: [https://github.com/edmund-
huber/jsonq](https://github.com/edmund-huber/jsonq)

------
a3n
Looking at the jq github repo led me to also find ag, an ack-grep replacement.
Both are in my distro's repositories.

[https://github.com/stedolan/jq](https://github.com/stedolan/jq)

[https://github.com/ggreer/the_silver_searcher](https://github.com/ggreer/the_silver_searcher)

"The command name is 33% shorter than ack!"

~~~
danso
I like ag, but unless I'm missing something, ack has this killer flag which ag
lacks:

    
    
        --output=expr
           Output the evaluation of expr for each line (turns off text
           highlighting) If PATTERN matches more than once then a line is
           output for each non-overlapping match.  For more information please
           see the section "Examples of --output".
    
    

Basically, it allows you to use captured groups in the output:

    
    
          cat file | ag '(\w+) (\d+)' --output '$1 is $2 years old'
    
    
    

On the other hand, ag allows multiline matching. So I end up using ag and ack
together frequently.

~~~
mVChr

        sed -rn 's/([a-zA-Z]+) ([0-9]+)/\1 is \2 years old/p' file

~~~
danso
sed doesn't support the same PCRE regex ack or ag

~~~
mVChr
No, but the `-r` option makes it use extended regex[1]. Here's some PCRE:

    
    
        cat file | perl -ne 'print if s/(\w+) (\d+)/\1 is \2 years old/'
    

My only point is sometimes old school standard tools are plenty.

1: [https://www.gnu.org/software/sed/manual/sed.html#Extended-
re...](https://www.gnu.org/software/sed/manual/sed.html#Extended-regexps)

~~~
danso
Right...but there's a point in having a tool that does PCRE, and all the
things useful in grep, in one command, rather than kicking out to Perl.

~~~
a3n
Agreed, it's nice to consolidate disparate functionality into a single
invocation.

But when I saw mVChr's sed line, and then again the perl line, I was reminded
"Oh, right, old school." And I wondered for the Nth time how much of old
school has been unknowingly reinvented, and how many times.

Nevertheless, I like the new(er) tools in this thread too.

------
vemv
My jq killer:

    
    
      function ruby_json {
        ruby -e "require 'json'; require 'active_support/all'; puts JSON.parse(STDIN.read)$1"
      }
    

Performs/prints arbitrary ruby over a readily available representation of the
JSON object coming from stdin.

Example:

    
    
      curl the_endpoint | ruby_json "['funnels'].map{|a| puts a.to_i * 2 }"

~~~
svs
Cannot accept user inputted script for json transform because Security
nightmare! That's JQ's and JMESPath's secret sauce.

------
kbenson
Related HN discussion, but more encompassing of things you might need to do on
the command line with data: _Command-line tools for data science_ (2013) [1]

1:
[https://news.ycombinator.com/item?id=6412190](https://news.ycombinator.com/item?id=6412190)

------
smetj
The Wishbone event stream processing framework
([http://wishbone.readthedocs.io](http://wishbone.readthedocs.io)) has the
wishbone.flow.jq module ([https://github.com/smetj/wishbone-flow-
jq](https://github.com/smetj/wishbone-flow-jq)) to do pattern matching on JSON
streams. The article
[http://smetj.net/processing_webhooks_using_wishbone_part_2.h...](http://smetj.net/processing_webhooks_using_wishbone_part_2.html)
shows how it's used.

------
ericHosick
Friend wrote a feedforward neural network using jq.

[https://github.com/kevin-albert/jq-neural-network](https://github.com/kevin-
albert/jq-neural-network)

------
zyang
Recently discovered jq and it has been indispensable part of my toolbox. A lot
of json tasks I used to write scripts for, such as filtering and multi file
concat, are now possible for a short command.

------
s_m
A few months ago I wrote a blog post about how to use jq in conjuction with
pup to debug HTML-in-JSON endpoints:
[https://the1x.engineer/2016/08/20/efficient-endpoint-
testing...](https://the1x.engineer/2016/08/20/efficient-endpoint-testing-curl-
jq-pup/)

jq is one of my favourite tools - can't recommend it enough.

------
koolba
jq is awesome. While it can do a lot of really cool processing to filter out
nested details, the majority of my usage of it is:

    
    
        $PROGRAM_THAT_OUTPUTS_JSON | jq .
    

Which simply pretty prints the JSON input with two spaces per indentation.
Also, $PROGRAM_THAT_OUTPUTS_JSON is usually a script that simply outputs the
clipboard (alias for "xclip -o").

~~~
grayrest
My usage is generally pretty simple as well but if people are looking for a
more complex example, I wrote one yesterday for putting together some test
data. Turns DynamoDB responses into newline separated flat arrays:

    
    
        pbpaste | jq -c '.actionResponses[].data.items[] | [.attributes[] | .stringValue // (.numberValue | tonumber)]'

------
preek
This is actually awesome for demos in lectures! Great to have better tooling
for such a regular task on the command line.

------
purple-dragon
Conceptually similar is Mike Bostock's wonderful ndjson-cli[0]

[0]: [https://github.com/mbostock/ndjson-
cli](https://github.com/mbostock/ndjson-cli)

------
spinningarrow
jq is really nice and I always install it as part of my tool chain. However I
don't use it very frequently because of which I can never remember the syntax
which imho is not very intuitive :(

------
zimbatm
For shell scripting, jshon is also very useful (
[http://kmkeen.com/jshon](http://kmkeen.com/jshon) ).

------
abraae
I wish jq was available from Java. We built a microservice recently that was
just an aggregator of a bunch of other api calls. Jq would have been perfect.

~~~
wryun
jmespath is:

[http://jmespath.org/libraries.html](http://jmespath.org/libraries.html)

(also, a nicer language IMO, even if it has slightly fewer features)

------
DrAwesome
I actually just discovered this tool a few days ago when I wanted to do
something with the Gitlab API from a bash script. It's really cool!

------
bootload
Q? How does Jq handle crap (malformed) JSON? ... quickly scanned the docs,
tutorial and readme. I see no FAQ.

------
taormina
For all of you on Macs, it's as easy as `brew install jq`. One of the first
things I grab on a new box.

------
foota
Seems like you could totally write a graphql command line utility to filter
through json output.

------
chewmieser
This basically becomes a necessity when working with the AWS cli. Great tool!

~~~
empath75
The aws cli tool basically has this built in with --query

------
Zombieball
Anyone here used jsawk that could compare and contrast to jq?

------
agumonkey
jq embeds fp bits, you can define functions too, crazy.

------
telmich
Very nice!

