
Jq – A Command-line JSON processor - geekrax
http://stedolan.github.io/jq/
======
useerup
jq is awesome, a real achievement.

At the same time it demonstrates everything that is wrong with traditional
shells and what PowerShell gets so right.

jq is _not_ a "Unixy" tool in the sense that it should do one thing and do it
right. jq implements its own expression language, command processor complete
with internal "pipelines". Why would a tool need to do that? find is another
utility that does many, many other things than to "find" items: It executes
commands, deletes objects etc.

Consider this challenge that included parsing json, filtering, projecting and
csv-formatting output:
[https://news.ycombinator.com/item?id=9438109](https://news.ycombinator.com/item?id=9438109)

Several solutions uses jq - to good effect. But the PowerShell solution uses
_PowerShell_ expressions to filter, sort and project items.

The problem is - at the core - that the traditional command tools are severely
restricted by only being able to rely on a text pipeline convention. You
cannot parse json and send the "objects" along to another tool. Well, you can,
if the json tree is extremely basic - like 2 levels.

PowerShell also has a json parser tool: [https://technet.microsoft.com/en-
us/library/hh849898.aspx](https://technet.microsoft.com/en-
us/library/hh849898.aspx). It is called ConvertFrom-Json (follows the verb-
noun convention) and is distributed as part of PowerShell (built-in if you
want - but that's a misnomer for PowerShell modules).

It is extremely simple - it doesn't even take any parameters - it just
converts a string (parses it as json) and outputs the object/objects which can
be arbitrarily complex. It truly does only one thing. If you want to select
specific properties, sort, group etc you do it with other PS tools like select
(alias for Select-Object), sort (alias for Sort-Object), group (alias for
Group-Object) etc. If you want to convert back to json you use ConvertTo-Json.

~~~
fragmede
Depends how you look at it.

jq _is_ a "Unixy" tool in the sense that it should do one thing and do it
right - and that 'one thing' is parse json data to output it in the desired
format.

Awk, arguably one of the oldest unix tools, also has its own language. Would
you consider it non-Unixy or is it grandfathered in due to age?

~~~
DonHopkins
It's "Unixy" in that it has a short cryptic name!

Some times "one thing" happens to be a big thing that you have to do 100% or
not at all, and doing it well requires a hell of a lot of complexity.

Does a XSLT processor do "one thing": processing XML? Does a DSSSL processor
for doing "one thing": processing SGML? (Hint: It has a Scheme interpreter
built into it!)

~~~
useerup
You raise some interesting points.

As I said, jq is impressive. It is a JSON processor that implements it's own
functional language, complete with variables, function definitions,
arithmetic, recursion. There is _no question_ that it is useful and will be
the absolutely right tool for a number of jobs.

Likewise with awk. I really, really like awk. It is such an elegant solution
to a common problem (text parsing and productions). It is _also_ a functional
language.

XSLT is also a functional language. An XSLT processor is expected to implement
a standard (as opposed to jq and awk) - but that's a minor difference.

It is when you are required to blend the disciplines that the limitations
sneak up on you. In powershell it works the same regardless of whether I
process data from JSON, from XML, from line-oriented text, from CSV, fixed
columns or or from database queries. Sorting, grouping, comparing, mapping,
presenting, converting - it is the same skill set.

That is not so say that something like PowerShell can or should replace tools
like awk, XSLT or jq. It should not. Period.

But for tasks where one has to use several formats, the lack of a sufficiently
strong inter-command protocol causes the tools to bloat and include
functionality that has nothing to do with the core task. Consider, for
example, why jq needs a "@csv" function (called a "format string") - and
@html, @sh, @base64!

My claim is that those functions have nothing to do with JSON processing.

The equivalent in PowerShell is handled by ConvertTo-Csv, ConvertTo-Html, Out-
String or the [Convert] class.

It really comes down to a n*m problem. jq is just one tool that implements csv
or html formatting (m formats). There are n tools (jq, awk, xslt, ...). Should
they all use the same rationale as jq and implement csv formatting? Or should
they stay away from that and just concentrate on the core task.

In PowerShell the equivalent of jq - the ConvertFrom-Json cmdlet - just
converts from JSON. Regardless of whether the objects were converted from
JSON, line-oriented text, database queries, fixed columns, XML - the output
formatting is handled by tools that are optimized for that.

If I'm dissing anything it is the limited inter-command protocol of
traditional shells. In this day and age we process complicated structures that
are not easily expressed in line-oriented plain text.

------
xahrepap
I've been using this for a couple months now. With httpie[1], it's become as
essential to my work day as Grep.

[1]
[https://github.com/jakubroztocil/httpie](https://github.com/jakubroztocil/httpie)

~~~
discardorama
Wow, thanks for that link. I hadn't heard of httpie, but it looks pretty
slick.

~~~
yunong
Isn't that the same as curl(1)?

~~~
jkbr
Nope. HTTPie is much, much more focused than cURL (the CLI tool). It is
exclusively an HTTP client designed primarily for interactive use. cURL, on
the other hand, is meant to be used mostly non-interactively and is more of a
general client for "URLs" (it also supports DICT, FILE, FTP, FTPS, GOPHER,
IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS,
TELNET and TFTP).

HTTPie's goal is user-friendliness, so it comes with features like built-in
JSON support, output formatting, syntax highlighting, extensions, etc.
Comparison of the two in one screenshot can be seen in the README:

[https://github.com/jakubroztocil/httpie#readme](https://github.com/jakubroztocil/httpie#readme)

(Disclaimer: I created HTTPie)

~~~
giovannibajo1
I use and love httpie but I start to regret that it's written in Python as the
Python platform is a very very late adopter of modern TLS standards. Even
today, in 2015, it's still a hurdle to use httpie with a SNI endpoint, and
forget about it making it talk to a ECDSA-based endpoint like those exposed by
the free tier of Cloudflare (so called "modern TLS"). I wonder how long it
will take to HTTP2 to land.

I tried a random rewrite of httpie in Go and it worked perfectly with those
endpoints, as expected, and Go is getting full HTTP2 with 1.5 in June. Too bad
the rewrite was still feature incomplete, when I tried it.

~~~
jkbr
The situation is not ideal but more recent versions of Python (2.7.9 and 3.4)
have a much improved TLS support and things like SNI work out of the box.

For older Python versions there are some additional dependencies to be
installed manually (will happen automatically in next version of HTTPie):

[https://github.com/jakubroztocil/httpie#sni-server-name-
indi...](https://github.com/jakubroztocil/httpie#sni-server-name-indication)

As for HTTP/2, there is an experimental extension for now:

[https://github.com/jakubroztocil/httpie-
http2](https://github.com/jakubroztocil/httpie-http2)

------
danso
Love, _love_ jq...after awhile, I realized I was twisting my brain too much to
get bash to do what it wasn't meant to do, but I got a lot of mileage of just
playing with APIs because of how jq made it so easy. I came up with some fun
lessons on how to use jq and other CLI-tools to tour the Spotify API [1] and
to mash up images from Instagram [2]

[1] [http://www.compciv.org/recipes/data/touring-the-spotify-
api/](http://www.compciv.org/recipes/data/touring-the-spotify-api/)

[2] [http://www.compciv.org/recipes/data/api-exploration-with-
gma...](http://www.compciv.org/recipes/data/api-exploration-with-gmaps-
instagram/)

(note: while I say that I like the command-line, I'm not die-hard enough to
have stopped to learn awk, so take that fwiw)

------
piva00
Using it for parsing AWS CLI responses, it's been a sweet tool to parse those
JSON responses from AWS, most of the time I'd fire some python script to parse
the data and now with jq I can keep most of the one-time scripts limited to
bash only, if I need something more maintainable or longer living then I'll
drop a script in python or ruby.

~~~
Terretta
JmesPath is built into the AWS CLI.

You don't need separate JQ. See the --query option:

[http://docs.aws.amazon.com/cli/latest/userguide/controlling-...](http://docs.aws.amazon.com/cli/latest/userguide/controlling-
output.html)

Check out [http://jmespath.org](http://jmespath.org) for details.

~~~
piva00
Only saw your comment today, thanks so much for this, I really overlooked this
part from the docs.

------
themartorana
Lots of previous discussions about this on HN (for instance [0]) but still a
sweet tool.

[0]
[https://news.ycombinator.com/item?id=5734683](https://news.ycombinator.com/item?id=5734683)

------
devy
Jq is truly amazing swiss army knife for JSON viewing/processing in command
line.

Btw, I've seen people posting this github link at least 10 times in the past.
I thought HN filters dupes, what gives?

[https://hn.algolia.com/?query=http:%2F%2Fstedolan.github.io%...](https://hn.algolia.com/?query=http:%2F%2Fstedolan.github.io%2Fjq%2F&sort=byPopularity&prefix&page=0&dateRange=all&type=story)

~~~
kbenson
I think it filters dupes within a certain time frame. You don't want important
front-page news at a site being considered a dupe because something was big on
the front page a year ago.

~~~
devy
Gotcha. In this particular case though, it does feel like every few months
someone will remind us there is this great tool some of us have been already
using for 2 years now:)

~~~
Zecc
FWIW, I hadn't heard of it before.

(actually that's a lie, I saw it mentioned in the CSV challenge thread. But
that was just a few days ago, so the point still stands)

------
yunong
For a similar tool written in javascript, check out json
([https://github.com/trentm/json](https://github.com/trentm/json)). I use
both.

------
andrewd18
Love JQ, been using it for the past year. I'm very much looking forward to the
regex support in 1.5.

------
_RPM
I love that it is written in C and not Go, or Rust, Java

------
rmgraham
I know it is right on the linked page, but it is useful enough I think it
deserves a direct link in any conversation:

[https://jqplay.org/](https://jqplay.org/)

------
blacksqr
Compare: rl_json -- Extends Tcl with a json value type and a command to
manipulate json values directly. The commands accept a path specification that
names some subset of the supplied json entity. The paths used by [json] allow
indexing into JSON arrays by the integer key.

[https://github.com/RubyLane/rl_json](https://github.com/RubyLane/rl_json)

------
aggregator-ios
For those on OS X, check out JSON Query:
[https://itunes.apple.com/us/app/json-
query/id953006734?mt=12](https://itunes.apple.com/us/app/json-
query/id953006734?mt=12)

It's not nearly as powerful as jq, but for simple querying and downloading
JSON (with POST body and HTTP headers), I find it quite useful.

Disclaimer: I built it.

~~~
lectrick
"brew install jq"

------
platz
I recently had to consume an xml service; it would've been great to have
something like Jq for xml. For some reason I wasn't really interested in xml
-> json conversion as a first step. I also found a bunch of xml cli tools, but
they are older, had to learn things like XQuery (which actually few tools
today seem to support).

~~~
xaduha
Well there's XProc. But as with everything XML - you shouldn't really edit it
by hand without proper editor support at the very least (preferably DTD/XML
Schema aware).

~~~
platz
I understand XML has namespaces, but I'm not sure what "edit by hand" means in
this context. The tools I found previously were xqilla and xmlstarlet, but
they seem pretty old and just not as convienient as Jq.

I still think there is space for an XML cli tool that is more convenient for
the majority of common tasks working with XML, at simply treating the data as
a hierarchical structure, without requiring an interface that is full fully-
qualified/needs XSLT/etc..

------
doki_pen
Nice. I made jipe[0] a while back to handle common troubleshooting tasks with
streams of json objects. I still use it quite a bit. I didn't see anything
about streaming in the jq docs.

[0] [https://github.com/dokipen/jipe](https://github.com/dokipen/jipe)

~~~
slapresta
It does streaming. You can pass a stream of JSON objects through standard
input and jq will process them on stream.

The master branch also has experimental support for JSON sequences.

------
feld
This appears to be in FreeBSD ports already, so just

    
    
      pkg install jq
    

and you can have it :)

------
kalehrishi
We are using jq to parse JSON output from various API in shell scripts. It
served us well even for large JSON strings. If you want to write small scripts
and parse some JSON in it then jq is best fit.

------
AdrianRossouw
i love it so much.

i've used it for things like migrating between json schemas, and even
converting normal json into elasticsearch bulk imports on the fly.

------
XorNot
How weird - I just started using this for some simple scripting against a rest
interface. Its a great tool.

------
klinquist
I love JQ. I use it along with Ansible's AWS dynamic inventory script to
manage my instances.

------
jimmy0x52
No mention of jsawk in here? We use it quite a bit for shell scripts, works
great.

------
beamatronic
I appreciate jq and use it almost every day. At the same time, I recently had
a task that was easier to do in Node.js. I had a file containing a JSON object
with a structure like so: [ { }, { }, ... ] and I wanted to turn each object
into a separate file.

~~~
TheDong
That task is trivially accomplishable in jq. I have no clue why you say it's
"easier" in nodejs.

    
    
        i=1
        for line in `echo '[{"a":"b"},{"c":"d"},{"e":"f"}]' | jq '.[]' -c`; do
          echo $line | jq '.' > jq_out_$i.json
          i=$((i+1))
        done
    

The above prints each element of the array to a file named by its index,
including pretty-printing it.

If you use it every day, the above should have come to you easily enough.

------
drdoom
Looks interesting. Given this json, [{"name":"john", "grades":[1,2,3]},
{"name":"jane", "grades":[4,5,6]}], can it print it out as this: john,1 john,2
john,3 jane,4 jane,5 jane,6

~~~
aquassaut
Yes, kind of like so:

echo '[{"name":"john", "grades":[1,2,3]}, {"name":"jane", "grades":[4,5,6]}]'|
jq '.[]| "\\(.name), \\(.grades|.[])"'

~~~
drdoom
Thanks! Now, it looks more interesting...

~~~
devy
Based on @aquassaut's answer, I tweaked it a little, and this should do it.

echo '[{"name":"john", "grades":[1,2,3]}, {"name":"jane", "grades":[4,5,6]}]'
| jq -r '.[] | "\\(.name),\\(.grades|.[])"' | paste -sd' ' -

------
misiti3780
one of the most useful command lines tools in my toolbox these days - thank
you for all that have contributed to this masterpiece.

------
shmerl
I'm using it in bash scripts - it's a very good tool.

