
Show HN: jql – Easier jq Alternative with a Lispy Syntax Written in Go - cube2222
https://github.com/cube2222/jql
======
cube2222
Hey, author here. Christmas Eve, 22 pm, after having eaten with my family, I
was browsing HN (because what else could you be doing then), and stumbled upon
this comment:
[https://news.ycombinator.com/item?id=21860107](https://news.ycombinator.com/item?id=21860107)

It inspired me to create an alternative to jq with more of a Lispy syntax, as
I think the original is awesome but also fairly cryptic for anything more
advanced than single field selection.

Overall it was a fun few-day project, and was also very educational in terms
of writing a parser in Go. (I used goyacc before in the sql parser of
OctoSQL[1], however, that one is copied from vitess, so I’ve never built one
from scratch, only customised an existing one. It’s really pleasant overall
and the code is very simple, so I encourage you to take a look[2].

I'd love to hear any feedback, comments or potential improvements you can
think of.

[1]:[https://github.com/cube2222/octosql](https://github.com/cube2222/octosql)

[2]:[https://github.com/cube2222/jql/tree/master/jql/parser](https://github.com/cube2222/jql/tree/master/jql/parser)

~~~
jsd1982
Looks good but there seems to be a missing explanation gap between switching
from `(elem "countries")` and dropping the `elem`s to just `("countries")`.
Why is `elem` able to be dropped? Is it the default function or something? I
didn't follow that big jump in the readme.

~~~
aasasd
This feature can probably be thought of similarly to Clojure's map access via
keywords: if you write `(:something map-var)`, you'll get the value of the
field ‘:something’ in the ‘map-var’ variable. I.e. keywords can be ‘executed’,
and such calls get translated to map access.

~~~
cube2222
Indeed, it just extends to values, and works recursively if it's an array or
map.

------
EdSchouten
So I guess we all know [https://jsonnet.org](https://jsonnet.org), right? It's
advertised as a data templating language. Interestingly enough, you can also
use it as a querying tool. The first four examples on the jql page can be
translated to Jsonnet as follows:

    
    
        $ jsonnet -e "(import 'test.json').countries[0]"
        $ jsonnet -e "(import 'test.json').countries[0:2]"
        $ jsonnet -e "[x.name for x in (import 'test.json').countries]"
        $ jsonnet -e "std.objectFields((import 'test.json').countries[0])"
    

With that in mind, I never bothered to learn how to use a tool like jq, rq,
jql, etc.

~~~
FragenAntworten
Another option which looks interesting:
[https://github.com/jmespath/jp](https://github.com/jmespath/jp)

I haven't used it yet, but the syntax seems simpler than some.

~~~
edaemon
Thanks for the link, this straightforward syntax is what I've been after for
dealing with JSON on the command line.

------
xvilka
There is more universal alternative to jq that supports also protobuf,
msgpack, cbor, etc - rq[1].

[1] [https://github.com/dflemstr/rq](https://github.com/dflemstr/rq)

~~~
cube2222
Thanks, didn't know about it, looks interesting!

Though from the tutorial I don't really understand how I can use it to
transform data, other than converting between formats.

~~~
dan-robertson
Looks like query support was dropped.

------
mharju
Looks cool!

I'm using
[https://github.com/borkdude/babashka](https://github.com/borkdude/babashka)
with with [https://github.com/borkdude/jet](https://github.com/borkdude/jet)
and a few aliases along with curl for the same effect.

The syntax of jq is so hard to remember a familiar lisp always beats it :)

~~~
cube2222
Nice, this looks great, and I love Clojure too, so I'll make sure to check it
out!

------
cookiecaper
To be frank the ascendance of jq has always seemed weird to me. There are many
alternatives that felt much more natural, as this thread demonstrates.

JSONata remains pretty obscure but it's been my favorite JSON query/mangling
DSL for a while now: [https://github.com/jsonata-
js/jsonata](https://github.com/jsonata-js/jsonata).

jmespath is also quite natural, but I think it was hurt by reference
implementation in Python, meaning it's quite slow.
[http://jmespath.org/](http://jmespath.org/)

~~~
rajangdavis
jq is simple to install, can handle large datasets (have used it to parse GB's
of JSON data), and works well with bash/shell.

~~~
cookiecaper
Yeah -- I've ended up defaulting to jq just because of its ubiquity, but many
of these other tools have similar attributes without the esoteric nature of
the jq DSL. It'd be great if something more natural came along and actually
started to displace it.

~~~
rajangdavis
I've never felt it was an unnatural API, but it can be hard to grok/remember.
I was primarily using jq either to extract data from large JSON files or to
concatenate a bunch of smaller JSON files into one, large file.

I think it can get a little weird as far as error handling goes, but I feel
it's been pretty approachable.

------
preek
If you're into jq, here's an interface so you can have live feedback (in
Emacs): [https://github.com/200ok-ch/counsel-jq](https://github.com/200ok-
ch/counsel-jq)

~~~
cube2222
You can probably also use an analogue to:

    
    
      echo '' | fzf --print-query --preview-window wrap --preview 'cat test.json | jql {q}'
    

Something close to:

    
    
      echo '' | fzf --print-query --preview-window wrap --preview 'cat test.json | jq {q}'

------
fiatjaf
Shameless plug to my list of interesting `jq` stuff:
[https://github.com/fiatjaf/awesome-jq](https://github.com/fiatjaf/awesome-jq)

It even features a complete rewrite of the actual jq in Go.

~~~
bloopernova
Thank you very much for shamelessly plugging your list, it's great!

I had no idea about jq-mode for Emacs, it even has org-babel support. So cool!

------
murat124

      echo '' | fzf --print-query --preview-window wrap --preview 'cat test.json | jql {q}'
    

Trivial note. echo instead of echo '' or echo "".

    
    
      $> [[ "$(echo)" == "$(echo '')" ]] && echo $? || echo $?
      $> 0

~~~
cimnine

      $> [[ "$(echo)" == "$(echo '')" ]] && echo $? || echo $?
      $> 0
    

Trivial note. `; echo $?` instead of `&& echo $? || echo $?`

    
    
      $> [[ "$(echo)" == "$(echo '')" ]]; echo $?
      $> 0

------
pololee
I have used fx for a few days and enjoyed it.
[https://github.com/antonmedv/fx](https://github.com/antonmedv/fx)

Highlights \- interactive mode \- Use full power of JavaScript.

$ curl ... | fx '.filter(x => x.startsWith("a"))'

\- Access all lodash (or ramda, etc) methods by using .fxrc file.

$ curl ... | fx '_.groupBy("commit.committer.name")' '_.mapValues(_.size)'

~~~
cube2222
I've seen fx recently and it looks very interesting. jql is obviously not as
feature-rich, but you can achieve interactivity with fzf!

    
    
      echo '' | fzf --print-query --preview-window wrap --preview 'cat test.json | jql {q}'

------
aasasd
Nice. However, a part of my problem with jq is that _changing_ data structure
also gets awkward pretty quick. So I hoped to hijack some small Lisp that's
still likely to have a suite of functional tools―`map` and such. This would
also rely on the fact that I can represent JSON's maps and arrays in most such
Lisps. Afaict jql doesn't have that.

So far I'm planning to just fire up Hy the next time I need anything in this
vein. Since it's on Python, both JSON parsing/encoding and functional stuff
should be there out of the box. Alternatively, I still can try using
ClojureScript with Lumo. But I don't expect too fast startup with either of
them, compared to, say, Fennel/Lua.

BTW, there's a Lisp that translates to Go:
[https://github.com/jcla1/gisp](https://github.com/jcla1/gisp). Though not too
popular, and I've heard an opinion that it's toy-like in its capabilities.

~~~
cube2222
Indeed, I know I didn't really solve your problem (and create your idea), as I
decided to go in a different direction.

Thanks for the inspiration though!

------
onionisafruit
Nice project. I'm a heavy jq user and still find myself regularly tinkering
around with queries on [https://jqplay.org](https://jqplay.org). It would be
nice if there were a similar page for jql, even if it is just a local server.

~~~
cube2222
I was thinking of an interactive terminal app, but that would be much more
work.

Though you can surely put a oneliner together which achieves the same effect
using fzf, as per: [https://paweldu.dev/posts/fzf-live-
repl/](https://paweldu.dev/posts/fzf-live-repl/)

~~~
preek
Here's an interactive implementation (in Emacs): [https://github.com/200ok-
ch/counsel-jq](https://github.com/200ok-ch/counsel-jq)

------
markandrewj
This is fairly short talk, but it might interesting to some people.

[https://youtu.be/PS_9pyIASvQ](https://youtu.be/PS_9pyIASvQ) [Serious
Programming with jq?! A practical and Purely Functional Programming Language!]

~~~
cube2222
Thanks for posting! I'll make sure to watch it!

~~~
markandrewj
No problem. I hope it is helpful and interesting for you.

------
michaelmrose
If this was lispy then (elem "countries" (elem 0)

Would be an error.

Expression evaluation ought to be inside out.

~~~
cube2222
Why?

It is, it's just that everything returns a function! The topmost function will
in the end be called with the json as an argument.

Check out these parts of the README:
[https://github.com/cube2222/jql#attention](https://github.com/cube2222/jql#attention)
[https://github.com/cube2222/jql#type-
cheatsheet](https://github.com/cube2222/jql#type-cheatsheet)

(elem 0) returns a function which takes a JSON array as input and returns its
first element.

~~~
michaelmrose
Can you find many lisps or even any lisps that evaluate like that? A familiar
syntax that actually works different from any lisp in existence may be more
hindrance than help.

~~~
cube2222
It’s first class functions. Clojure has them.

It's just that the query ends up having a type JSON -> JSON, not just JSON.

~~~
michaelmrose
Take

    
    
       (elem "countries" (elem 0))
    

Inside out elem 0 has to return a function since it doesn't have enough to do
anything. I'm not sure why 0 couldn't be a key necessitating a different
function to differentiate but whatever.

Elem countries has to return json and one can imagine using the json in the
first position like a map in clojure and taking a fn in following position.
This would mean that on net it would be exactly the same as.

    
    
        (elem 0 (elem countries))

~~~
cube2222
It really ends up being called like this: ((elem "countries" (elem 0)) JSON)

Yours is more akin to (elem 0 (elem "countries" json))

and you can see that the hierarchy of the query is now inverted in respect to
the json. This is what I like about the continuation based approach, a big
query will be readable because it fits the data very well.

~~~
michaelmrose
Is it more readable than

    
    
        (->> JSON 
            (elem "countries")
            first)

~~~
cube2222
This looks very readable and matches the structure of the json too.

To me it's basically the same readability wise with proper indentation.

Though with deeply nested data and a lot of map functions I think you'd have a
lot of unnecessary (->> JSON ...). But those can probably be eliminated with
another macro.

Also, there's the pipe function in jql which basically lets you do this.

------
spikepuppet
I'm super excited to sink my teeth into this. I usually have to use jq in
intense bursts and no matter what i end up back in the manual because it just
get's so damn confusing.

From a quick look on the repo, this seems so much simpler. Thanks!!!

~~~
cube2222
I'm happy that you're so enthusiastic! Make sure to let me know what you think
about it after having tried it out!

------
dfc
Documentation Suggestion: I'm not familiar with fzf. So having the first
example the README use fzf but not subsequent examples is a little confusing.
Maybe start with a couple small examples and then use fzf towards the end?

~~~
cube2222
I've added this after comments here asked for it, but I may move it towards a
small "interactivity" chapter towards the end.

It just evaluates jql continuously while you can write and edit your query.

------
bloopernova
Good Lawd'amercy.

jq _and_ lisp?

In all seriousness, thank you for writing this. I've had trouble wrapping my
small brain around some of jq's more advanced syntax, so being able to look at
things from a different direction seems like a great idea to me.

~~~
cube2222
Thanks!

Let me know if it was understandable/confusing overall!

~~~
bloopernova
It was very well laid out and understandable. I found myself wishing for a
parenthesis-matching capability in my zsh though! I also liked the
informal/joking conversational style of the tutorial/walkthrough. It helped to
put me at ease while learning something new. And having it written in Go, and
with Go's simple installation method, makes me way more likely to use it in
the future :) Nice job!

~~~
cube2222
Very glad to hear that! Feel free to open issues if you have problems or
feature requests!

There's an issue for at least adding trailing and starting parentheses
automatically, so I'll try to add at least the trailing ones.

------
moondev
Personally I like
[https://github.com/mikefarah/yq](https://github.com/mikefarah/yq)

It also converts between yaml and json with ease for writing and reading

------
ldd
only tangentially related, I made a VS Code extension to help out with jq
development[0].

Also, jq supports modules. which is awesome!

[0][https://marketplace.visualstudio.com/items?itemName=ldd-
vs-c...](https://marketplace.visualstudio.com/items?itemName=ldd-vs-code.jq-
preview)

------
dan-robertson
I tend end to struggle with this sort of lispy syntax as I use a regular shell
(rather than one in emacs) which makes balancing parens hard, and the syntax
is otherwise verbose. If I’m using a query tool like this it will be
interactively writing a one-liner so I want syntax that’s easy to get right
(balancing parens by hand is hard), easy in the usual case (eg I think it
should accept some way to use symbols to name fields in a record because
typing and balancing double quotes all of the time is a waste of time) and
easy to append to (I will run a query, press up, move to the end of the query,
and modify. In this case I think moving to the end involves counting parens).

Perhaps I would want some implicit pipe or threading at the top level (which
could be turned off with some option for scripts I guess).

Another helpful feature could be adding the magic closing paren: ] which will
close all open parens to the top level. E.g. these two lines are equivalent:

    
    
      (foo (bar (baz) whizz))
      (foo (bar (baz) whizz]
    

A completely different (and less powerful) paradigm I like is one of match-
format where the pattern looks a lot like what it’s trying to match. In this
case most functions and fancy transformations are impossible. So you could
write something like this to extract the list of countries:

    
    
      j '{countries: %%}'
    

Or to extract each country’s name:

    
    
      j '{countries: [ .*, { name: %% }, .*]}'
    

(Presumably you would have syntax for trying to match against each array
element). (I also would make commas optional but I’m trying to make this
syntax obvious).

Or to list the keys of the 0th element:

    
    
      j '{countries: [ { %%: . }, .* ]}'
    

Or to get name and population:

    
    
      j '{countries: [ .*, { name: %name, population: %pop }]}'
    

The output might look like:

    
    
      { "name": "Poland", "population": "38000000" }
    

————————

An alternative query tool which I haven’t really seen before would be one
which is interactive rather than repeatedly run: you pipe your json (or
whatever) into it and it spins up some kind of (curses) gui shoeing you your
data with a nice interface to interactively write your query and see its
results. It could finish by outputting a command line to run the query non-
interactively.

The closest thing to this I know of is a small programming by example demo
that came out of Microsoft research. The goal was to figure out how to extract
some desired data from some big blob of json (or maybe html, I don’t
remember). The user would see the json in some web gui and could click on the
data they wanted and the app would try to figure out the “most obvious” query
to get it. If they clicked on the country name, I think it would probably
suggest a query for all the countries’ names (but maybe that would be the
second option). I think it would also try to be clever about grouping, so if
the user picked name and population it would hopefully figure out that they
should come in pairs rather than being two independent lists that might not be
the same length.

~~~
cube2222
Thanks so much for so much feedback!

There's already an issue to automatically add trailing parens and I definitely
think it's a good idea, as balancing parens really is annoying in-shell. I
didn't think of it before.

I decided that adding interactivity would be too much work, but you could
definitely put together a oneliner (or alias) which achieves that using fzf:
[https://paweldu.dev/posts/fzf-live-repl/](https://paweldu.dev/posts/fzf-live-
repl/)

Overall I encourage you to try write a tool that has the syntax you described,
as it sounds plausible! It kinda looks a little bit like graphql, so maybe
that would fit?

~~~
dan-robertson
I wonder if it’s worth writing some read line macros (and instructions for
setting them up) for eg inserting ( ) <move-cursor-backwards>. (Bound to M-(
in emacs) I think they help for simple sexp editing. I also wish I had
something like “run this command but when it’s done show me a prompt with the
same command and my cursor in the same place” but I have no idea if it’s easy

~~~
cube2222
I didn't know about readline macros, thanks for mentioning them.

I'll think about it.

