
JSON on the Command Line with Jq - pcr910303
https://shapeshed.com/jq-json/
======
aasasd
Every time I try to do something involved with jq (i.e. more than a couple
commands deep) and inevitably end up wandering through the manual, I get a
persistent feeling that jq could be cleaned up quite a bit. That it either has
structure throughout that I fail to grasp, or that it's really rather
disorganized and convoluted in places. This is exacerbated by me never being
able to find anything in the manual without skimming half of it each single
time.

E.g.:

Filtering and transforming objects by reaching a few levels deeper in them
seems to be way easier to slap together in Python's list comprehensions or
some functional notation. Stuff like [{ * * x, qwe: x.qwe*2} for x in a.b if
x.y.z == 1].

Add nested lists to the above, and I can spend another fifteen minutes writing
a one-liner (i.e. `a` is a list of objects with lists in some field).

I once had to deal with a conditional structure where a field might be one
object or a list of them. Hoo boy.

Every time I want to check the number of entries my filters print out, I need
to wrap them in an array. Repeat this a couple dozen times during writing a
complex query. Weirdly, this is where strict structure gets in the way―and
while I understand the reasoning, I feel that something could be done for such
basic need. (Like, aggregate filters should be able to aggregate instead of
acting on individual items? Maybe it's already in the language, but I'm not in
the mood to go over the manual again.)

Variables seem to be bolted on as an afterthought, or at least the syntax
doesn't exactly accommodate them. Meanwhile, they're necessary to implement
some filters omitted in the language. Compare that to Lisp's simple `(let)`.
IIRC the manual also says that jq has some semblance of functions, but I'm
afraid to think about them with this syntax.

I like the idea a lot, but execution not so much. Frankly I'll probably end up
throwing together a script in Lumo or something, that will accept Lisp
expressions and feed JSON structure to them. (I'd use Fennel, but JSON has
actual null while Lua... doesn't.)

Btw, I have pretty much the same sentiment about Git. Git structures, great.
Git tools, oy vey. Maybe I need Lisp for Git, or at least Python for Git.

~~~
gregopet
Personally I prefer
[https://github.com/jmespath/jp](https://github.com/jmespath/jp) \- it has a
bit saner syntax based on an actual standard (at least for me it's easier to
remember than jq).

And perhaps more importantly it doesn't mess with your numbers! Some other
comment mentioned doubles, but I had a problem with Longs, jq would silently
truncate them when they became to large to fit inside a float (I was working
with random Long numbers used as secrets in some schema so it happened quite
quickly in my use case).

~~~
michaelcampbell
Thanks for this; for the few cases/year I need something like `jq`, I just
find its selector syntax too hard to remember (seems a common theme). `jp`
seems way more usable.

------
dang
Thread from 2016:
[https://news.ycombinator.com/item?id=13090604](https://news.ycombinator.com/item?id=13090604)

2015:
[https://news.ycombinator.com/item?id=9446980](https://news.ycombinator.com/item?id=9446980)

2014:
[https://news.ycombinator.com/item?id=7895076](https://news.ycombinator.com/item?id=7895076)

2013:
[https://news.ycombinator.com/item?id=5734683](https://news.ycombinator.com/item?id=5734683)

2012:
[https://news.ycombinator.com/item?id=4985250](https://news.ycombinator.com/item?id=4985250)

[https://news.ycombinator.com/item?id=4679933](https://news.ycombinator.com/item?id=4679933)

(for the curious)

------
DC-3
For those interested in JSON as a command line protocol, I think it's
interesting to see how Andreas Kling has integrated it into his Serenity OS
project. While the project is aesthetically retro, its architecture is
forward-facing, and the usage of JSON for the representation of such data as
the process tree is, I think, an interesting take on old Unix ideas.

~~~
akling
It definitely makes writing the userspace code a lot nicer!

The main issue I've run into so far is that my JSON parser can generate quite
heavy in-memory representations of parsed data. It's far from optimized
though, and a lot can be done before needing to worry about the limitations of
the format itself. :)

------
inopinatus
jq is the AWK of today. I'm in favour of jq in *nix base. I use it almost as
much as grep | cut | sort.

JSON is ubiquitous now. So much so that even my <conservative enterprise
customers>, who 20 years preferred data in CSV, 10 years ago preferred XML,
now want to exchange data in JSON.

~~~
lenkite
Something better than jq please. It has the most arcane and inconsistent
syntax that I have ever seen in a CLI tool. There are newer tools like jp with
simpler language that can be understood by mere mortals with limited brain
power.

~~~
kmelve
You should check out GROQ ([https://groq.dev](https://groq.dev)), it's a query
language for JSON.

------
tejtm
A line of `jq` to turn any .json into a set of xpaths.

[https://github.com/TomConlin/json2xpath](https://github.com/TomConlin/json2xpath)

~~~
GhettoMaestro
This is useful! Thank you for sharing!!!!

------
dan-robertson
Jq can do lots of things that are similar to what’s normally done with awk,
cut, sed, and grep. The languages pipeline model is slightly opaque (eg as
soon as you use a function like select) and I feel tricky to get the hang of
but overall I think it works well for the kind of job jq does.

That said, I usually get on well with a halfway house of piping jq -c ... into
eg grep or sort -u for easy processing (-c writes one json object per line,
aka compact form), or selecting a few elements, putting them in an array, jq-
piping into @csv, and passing -r to get raw output which I can then process
with awk or sort or other Unix commands.

One subtle caveat is that jq normalises all the json it processes, so for
example escaped characters in strings may change and numbers which are not
IEEE doubles get converted (eg if someone writes a 64-bit int or a bignum as a
number not a string in the json)

------
Gys
Jq is nice and I regularly use it.

Does anyone know a more interactive json tool? With, for example, auto-
completion? Showing possible fields to select next and maybe even possible
values if fields are choosen.

~~~
frou_dh
> Does anyone know a more interactive json tool? With, for example, auto-
> completion?

[https://github.com/antonmedv/fx](https://github.com/antonmedv/fx)

It just uses JS for the query language and can be hooked in to bash
completion.

Personally I don't get on with jq, because I don't feel that plinking with
JSON is something that warrants memorising a bespoke language.

------
jayd16
Being reminded of JQ has me thinking... I actually don't hate powershell or at
least I like it's attempt at typed pipes.

Has anyone gone down the road of posix like Json pipes? Or maybe just Json
bridging the gap between fully typed pipes and the string based Unix cli
world?

Even just having a Json output format for the core posix apps would be good.
If it had OS support, maybe you wouldn't even have to serialize to strings if
both sides of the pipe happened to be compatible.

~~~
imglorp
Yes, Oil Shell is one:

[https://www.oilshell.org/preview/doc/json.html](https://www.oilshell.org/preview/doc/json.html)

------
llimllib
I like jq, but the documentation is super thin. Every time I go to do anything
involved, I inevitably end up at stack overflow hoping somebody else has had
to do it before.

I really wish there were a much greater selection of usage examples in the
manual.

------
NinoScript
Jq is a great tool, I wish more people knew about it!

~~~
smitty1e
It's a bit arcane to use, but has proven itself over and over when doing
little tasks with AWS CLI returns.

~~~
GhettoMaestro
Exactly right.

I use jq to mangle get/list requests from AWS (and others) and then
recursively hit each element for whatever reason. Without jq none of my
workflows in this area would be remotely easy.

TLDR: Learn JQ! It is like sed and grep, but for JSON.

~~~
dastx
If you're using AWS you're better using the built in option available for the
cli. The --query option (and often combined with the --output option) does
this, and uses JMESPath. Especially when you're scripting, it eliminates the
need for having yet another tool.

~~~
dragonwriter
> If you're using AWS you're better using the built in option available for
> the cli. The --query option (and often combined with the --output option)
> does this, and uses JMESPath.

Jq is so much more capable than jmespath that I find myself using jq on the
json output of AWS CLI commands more than using the built-in query option.
Sure, it requires another tool, but it’a often the best tool for the job.

------
dxxvi
I'd like to have a command line tool that allows me to modify the json with
javascript. Does jq provide any feature to process json that it's very
difficult/cumbersome to do in js?

~~~
dastx
Jq allows you to fully transform a json structure. Batteries included. Similar
to awk/sed etc you can even write your jq code in a separate file, and point
jq to it.

