
Mastering Jq: Part 1 - code-faster
https://codefaster.substack.com/p/mastering-jq-part-1-59c
======
anarcat
I know I might be a dissenting opinion here, but I can never wrap my head
around `jq`. I can manage `jq .`, `jq .foo` and `jq -r`, but beyond that, the
DSL is just opaque to me.

And every time I try to learn, I get lost in a maze of a manpage:

[https://manpages.debian.org/jq](https://manpages.debian.org/jq)

I mean there are useful tips in there, but I rarely find what I need. I
usually find it simpler to write a Python script (which ships with `json`
because it's "batteries included") and operate on lists and dicts the normal
way...

It's longer, but at least I don't need to learn a new programming language (if
we can call jq that...)

~~~
enriquto
> the DSL is just opaque to me.

May I suggest "gron"? As compared to jq it is simpler (good!) has less
features (good!) has no DSL (good!) and it is less powerful (good!, in many
cases). It is a tool that "expands" json into standalone lines, and you can
further process them using the standard tools grep, sed, cut, sort, awk...

EDIT: for example, if you have this json:

    
    
        {
          "outdir" : "out3",
          "data" : [
            {"img" : "img_01.jpg"},
            {"img" : "img_02.jpg"}
            ],
          "roi" : {
              "x" : 150,
              "y" : 150,
              "w" : 700,
              "h" : 700
          },
          "margin_h": 20,
          "margin_v": 5,
          "tile_size" : 300,
          "resolution": 0.5
        }
    

running it through gron produces this

    
    
        outdir = "out3"
        data[0].img = "img_01.jpg"
        data[1].img = "img_02.jpg"
        roi.x = 150
        roi.y = 150
        roi.w = 700
        roi.h = 700
        margin_h = 20
        margin_v = 5
        tile_size = 300
        resolution = 0.5
    

that some people find really convenient to deal with.

~~~
fold_left
[https://github.com/tomnomnom/gron](https://github.com/tomnomnom/gron)

------
MattyRad
By far the most useful jq operator I've used is recursive descent: `..`. It
can take a positively heinous json blob and pick out the attributes/keys you
care about. For example:

Yikes:

> _wget -O - -q
> '[https://reddit.com/r/unixporn/new.json'](https://reddit.com/r/unixporn/new.json')
> | jq_

I just want the links:

> _wget -O - -q
> '[https://reddit.com/r/unixporn/new.json'](https://reddit.com/r/unixporn/new.json')
> | jq '..|.permalink? | select(.)'_

~~~
hiram112
Thank you. This will be added to my 'How To JQ' list, since I'm convinced now
that I'll never fully understand its syntax enough to master it outright.

~~~
mahalol
Oh, would you mind sharing your list please?

------
ryannevius
One of the most useful ways I've found to learn/test queries in `jq` is by
using it as a semi-REPL via fzf with `--preview`:

    
    
        echo '' | fzf --print-query --preview 'cat example.json | jq {q}'
    

Here's a rough example of it in action:
[https://asciinema.org/a/y4WGyqcz1wWdiyxDofdPXKtdC](https://asciinema.org/a/y4WGyqcz1wWdiyxDofdPXKtdC)

~~~
ctippett
Wow, thank you for sharing this. I had no idea you could do that with fzf.

------
llimllib
This example is weirdly spelled:

    
    
        echo '{"k1": [{"k2": [9]}]}' | jq '.k1 | .[0] | .k2 | .[0]'
    

is equivalent to

    
    
        echo '{"k1": [{"k2": [9]}]}' | jq '.k1[0].k2[0]'
    
    

I kept waiting for the author to explain this, but they did not

~~~
code-faster
It is a weird way to write it, I never use this notation for myself.

When teaching though, I'd rather make the it more visually explicit that jq
commands are a sequence of filters. With the former notation, a | explicitly
tells the reader that it's the end of one filter and the start of a new one,
the latter is more implicit.

~~~
llimllib
That's possible! I've never had to teach it, good point.

------
JamesSwift
`jq` is up there with `nix` for me where I can sort of remember the DSL but
always have to open the docs. I recently found
[https://jqplay.org/](https://jqplay.org/) and its a huge help. I can crank
scripts out much quicker now that I'm iterating with that.

~~~
yingw787
[https://jqterm.com](https://jqterm.com) is also a good one :)

------
jbotz
jq is to cloud and "Infrastructure as Code" software what grep and awk are to
unix software. It's the glue that lets you combine the pieces into a system.
It's an odd language though, and not that easy to master. Tutorials like this
one are helpful and needed... Thanks, Tyler!

However, there are quite a few typos in the examples... even the very first
one is missing a quote and won't parse if cut-and-pasted as is.

~~~
cle
I used to use shell scripts and jq a lot, but got tired of my scripts growing
into monstrous mudballs of Bash hackery.

I do my scripting mostly in Go now and it's much easier to structure my code
and grow it over time. I sometimes use Python, but Go is more flexible for
what I do. I can compile an exe and drop it onto a host and run it, without
worrying about VM and library versioning (or in Bash's case, making sure jq is
installed and all the Unix utils have compatible versions).

~~~
thechao
How’s go’s batteries-included story? I’ve got nigh-on 50k lines of Python2
that IT is about to boot. My choice is to port it to ... something. I was
going to do Python3 but I’ve always thought that a memory-management-free
systems-like language would’ve always been better.

~~~
cle
There is not a language as batteries-included as Go, that I've found. The
standard lib includes almost everything, including an excellent HTTP server
and client (even an HTTP proxy), a test framework, JSON/XML/CSV libs, SQL
driver, CLI parser, a template library, crypto, etc. The "go" command includes
tooling like a package manager, unit test runner, code formatter, static
analyzer, race condition checker, code coverage, profiler, cross-compiler,
etc.

I don't know what you mean by "memory-management-free" though, Go is still a
garbage collected language.

~~~
Evidlo
I'd say Python has more batteries than Go.

Also I recently learned Go's CLI parser is pretty weak compared to argparse.

~~~
cp9
ymmv there but for what I'm looking for Go is usually more helpful than python
straight out of the box, specifically because you don't have to reach for a
third party library to make simple http requests, which is what I often end up
using a scripting language for. I did just pick up argparse though and was
very pleasantly surprised about how nice it is, go's `flags` module in the
stdlib is unfortunately not quite as nice and is one of the things I reach for
a third party library on. However, because it's pure go it still compiles down
to a single binary which you can just scp around.

------
Rapzid
I don't know what it is, but anything more than moderately simple stuff gets
super annoying for me with JQ.

After getting super frustrated with the documentation while trying to
accomplish stuff that should have been straight forward, I created jsling so I
could pipe output through node for JavaScript one-liners. Anything moderately
complicated that doesn't need to be portable; I just use that.

EDIT: I should mention that by "moderately complicated" I mean stuff that
starts to get into the realm of joins, correlated sub queries, and the like.

------
ojhughes
When I find myself reaching for jq, that’s a signal that maybe this bash
script should be implemented in a language such as python or go

------
kureikain
Another good one about JQ:

[https://programminghistorian.org/en/lessons/json-and-
jq#the-...](https://programminghistorian.org/en/lessons/json-and-jq#the-dot-)

------
xonix
I myself am developing a very similar tool [1] with somewhat saner (IMHO!)
DSL. In fact this is a side-project from a JS lib I developed long ago [2].

My approach uses very simple ideas and is heavily based on JS internally. This
is simple once you understand it [3] (thus saner DSL) and gives the whole
power of JS in your hands.

I've also prepared a basic comparison jq vs jsqry based on some examples in
article [4].

It's worth noting that currently the CLI tool [1] is written in Java using
Graal VM polyglot native image compilation. Thus HUGE executable size of 95 MB
(sic!) because it bundles JS engine. I'm considering rewriting this to QuickJS
by Fabrice Bellard [5]. This should make it MUCH smaller.

[1] [https://github.com/jsqry/jsqry-cli](https://github.com/jsqry/jsqry-cli)

[2] [https://github.com/jsqry/jsqry](https://github.com/jsqry/jsqry)

[3] [https://jsqry.github.io/#filtering](https://jsqry.github.io/#filtering)

[4]
[https://gist.github.com/xonixx/d6066e83ec0773df248141440b18e...](https://gist.github.com/xonixx/d6066e83ec0773df248141440b18e8e4)

[5] [https://bellard.org/quickjs/](https://bellard.org/quickjs/)

------
labelbias
Just recently used xq to convert and clean up a bunch of weird xml files to
JSON. It was such a breeze.

I love how nested lists inside objects can expand into a particular one with
the .[] operator.

For example:

{ a: [{b: 5}, {b: 3}], c: 5} can be transformed into: [{ b: 5, c: 5}, {b: 3,
c: 5}] using jq '{c: .c, b: .a[].b}'

For heavily nested XMLs I can get a nice flat output.

~~~
code-faster
Nice! I've found xml transforms is one of the best applications of jq (yes,
jq, not xq).

In a future post, I'll cover how to use jq not just for json and xml but any
data format.

------
moreati
I wrote [https://github.com/moreati/jq-filter](https://github.com/moreati/jq-
filter) for using jq with as a filter in Ansible.

------
Ericson2314
The author of jq is quite talented. I hope jq becomes a gateway drug.

------
coding123
Jq is quickly becoming one of the most useful tools in devops.

~~~
ing33k
Mhm. Can you please add some details on how you are using it ?

~~~
coding123
Generally we use it to manipulate JSON configuration files. So often we'll
have a single Config file across many microservices. We just use the same
config for simplicity. Then we deploy many different services that have access
to that same config. Some of the docker containers know how to natively access
the config file. Others, things were we might be running one-off jobs
especially, are often built from outside software that has no idea how to read
our config file. Instead of programming a harness around it, we just typically
use jq to read the config elements we need and save them as variables in a
script. Then we can pass those variables as needed to whatever command line
arguments.

Now, admittedly anywhere we have Node available, it pretty much negates a lot
of it. But even with node, it's sometimes easier to just throw in a couple jq
calls to extract data.

------
macca321
I reckon JMESPath should be first choice in 2020, as it's properly specified,
so you can find x-language implementations that behave the same.

------
maddyboo
jq is honestly one of my favorite tools ever. Super powerful yet simple.

I've written some relatively complex programs with it, e.g. this one [1] I
just wrote to manipulate the output from neuron [2] to generate index files
based on tags. It's probably overkill/not as efficient as it could be, but I
wanted to be able to expand upon it later.

I think once you grok the basics, which can be a bit confusing at first, you
can accomplish some amazing things with the help of a few of the more advanced
features like reduce, to_entries/from_entries, etc.

I really wish it was more actively developed, I feel like it actually has
potential to be a semi-general-purpose functional programming language.

1:
[https://gist.github.com/b50a047115d1dcf1f15c16a6f7b71e3c](https://gist.github.com/b50a047115d1dcf1f15c16a6f7b71e3c)

2: [https://github.com/srid/neuron](https://github.com/srid/neuron)

------
jolux
Is there stuff jq can do that I can't do with Invoke-RestMethod and
manipulating objects in PowerShell?

~~~
dragonwriter
There's nothing in jq you can't do with any programming language with a JSON
parser and serializer implemented.

But I find jq more pleasant than most for lots of JSON-to-JSON
transformations, and Powershell worse than not only jq but also Node or Python
for that task.

~~~
jolux
It’s a one-liner in PowerShell to grab JSON from an endpoint and deserialize
it.

~~~
dragonwriter
> It’s a one-liner in PowerShell to grab JSON from an endpoint and deserialize
> it.

Yes, and that’s great. I don't prefer jq for downloading JSON (which it
doesn't do), but, as I said, for JSON-to-JSON transformations.)

~~~
jolux
That’s fair, it probably makes more sense for that usecase than PS. It just
feels like yet another DSL to learn

------
ryanmccullagh
My favorite way to use jq is as follows:

`tail -f my-log-file.json | jq`

This can be great if you've setup Nginx to log as JSON.

------
yiyus
At first I thought this was about github.com/timestored/jq and was very
confused.

------
pyuser583
jq really is an awesome tool.

