

Jq - a lightweight and flexible command-line JSON processor - denzil_correa
http://stedolan.github.com/jq/

======
atdt
It'd be great if the help text contained some examples rather than simply a
pointer to the website, and if you added a man page. When you're in the middle
of a terminal session, switching to a browser can be really disruptive.

Also, if you tag releases with a version number and make them available as
downloads on GitHub, you'll have an easier time getting Jq into Homebrew, the
OS X package manager. I wrote a formula for you, but AFAIK you'll need to add
in a real version number before submitting a pull request to Homebrew:
<https://gist.github.com/3928074>

~~~
alexchamberlain
Arch PKGBUILD would be much easier with a version number too...

~~~
stedolan
They exist now!

------
nekedorff
Very nice tool! I'm getting syntax errors from the Twitter examples though
(and everytime I try to use underscores):

<edit>Created an issue on Github</edit>

    
    
      curl 'http://search.twitter.com/search.json?q=json&rpp=5&include_entities=true | jq '.results[] | {from_user, text, urls: [.entities.urls[].url]}' 
    
      error: Invalid character
      .results[] | {from_user, text, urls: [.entities.urls[].url]}
                        ^
      error: syntax error, unexpected IDENT, expecting '}'
      .results[] | {from_user, text, urls: [.entities.urls[].url]}
                        ^^^^
      2 compile errors
        % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                       Dload  Upload   Total   Spent    Left  Speed
      100  5970  100  5970    0     0  11969      0 --:--:-- --:--:-- --:--:-- 23050

~~~
stedolan
(author here) That bug was fixed a while ago, but I forgot to upload new
binaries. Try again, they should be working now.

------
tmcw
See also, underscore-cli <https://github.com/ddopson/underscore-cli>

~~~
zootm
Also see also RecordStream: <https://github.com/benbernard/RecordStream>

Someone mentioned jsawk in a sibling comment, too.

------
alexchamberlain
> You can download a single binary, scp it to a far away machine, and expect
> it to work.

Not if my machine is x86 and the server is SPARC.

Cool utility though.

~~~
evincarofautumn
Git clone and make, then. As long as you have a C compiler. And if you don’t,
you probably aren’t using JSON.

~~~
hnriot
Actually I bet there are a ton of people using JSON that don't have a c
compiler. The technologies are spaces by several decades and I would imagine
the majority of JSON people are using much higher level languages than c.

If I want to munge JSON I find it easier to do so in python, just convert it
back to data structures and manipulate them and serialized it back to JSON.

~~~
evincarofautumn
I would usually use some other language with a JSON library as well, but this
utility is conveniently one-function, after Unix fashion. I like
composability. Also, a C compiler is usually on hand whether you use it or
not; even if none is, they’re not exactly hard to obtain.

~~~
ville
"they're not exactly hard to obtain"

Except, of course, in Mac OS X where you have to download (a couple of
gigabytes) Xcode from Apple and install it to get GCC :)

------
055static
Wow, this compiles fast. And the Flex and Bison files are there. This seems
hackable. Nice work!

My only questions are (they are always the same with any purported sed/awk
"replacement"):

1\. what was the problem you were trying to solve where sed and awk failed
you, and

2\. does this program operate line by line or does it read entire files into
memory?

I had to deal with some JSON a while ago and threw together some sed like this
just so I could read it:

    
    
        sed '
        s/,/&\
        /g;
        /^$/d;
        s/^[{][^}]/\
        &/g; 
        /\"/s/,/<##eol##>/g;
        s/ *//;
        ' |tr '\012' '\040' |sed '
        s/<##eol##>/\
        /g;
        s/\[/\
        &\
        /;
        s/\]/\
        &\
        /;
        s/,  /, /g;
        '
    

But any difficuly I have dealing with JSON I attribute to the pervasive use of
JSON, not sed.

~~~
stedolan
Thanks!

1\. I think your example shows sed/awk's failings with JSON data :) I don't
want to write a JSON parser by hand every time I want to pull a field out of
an object, and parsing recursive structures with regexes is never a good plan.

2\. It reads JSON items from stdin into memory, one at a time. So if the input
is a single giant JSON value, it all goes into memory, but if it's a series of
whitespace-separated values they'll be processed sequentially.

It's cat-friendly: if you do

    
    
        cat a | jq foo; cat b | jq foo
    

then it's the same as doing

    
    
        cat a b | jq foo

~~~
055static
1\. But those are general statements. Opinions. What I mean is give me a
specific case. A specific example, a specific block of JSON and a specific
task. Once I have that, then I can ask myself "Is this something I would ever
need to do or that I have to do on a regular basis?"

Sometimes I need to write one-off filters. There is just no getting around it.
I have to choose a utility that gives maximum flexibility and is not too
verbose; I don't like to type. Lots of people like Perl, and other similar
scripting languages for writing one-off filters. But Perl, _out of the box_,
is not a line-by-line filter. It's unlike sed/awk; it needs more memory. That
brings us to #2.

2\. If I understand correctly, jq is reading the entire JSON block into
memory. This is what separates your program and so many other sed/awk
"replacements" from sed and awk, the programs they purport to "replace".
sed/awk don't read entire files into memory, they operate line-by-line and use
a reasonably small buffer. Any sed/awk "replacement" would have to match that
functionality. Given that sed/awk don't read in an entire structure (JSON,
XML, etc.) before processing it, they are ideal for memory constrained
environments. (As long as you don't overfill their buffers, which rarely
happens in my experience.)

Anyway, so far I like this program. Best JSON filter I've seen yet (because I
can hack on the lexer and parser you provided).

Well done.

------
draegtun
Also see App::PipeFilter - <https://metacpan.org/module/App::PipeFilter>

The module is a framework/collection of shell pipelines which include JSON
processing:

    
    
      curl -s 'http://api.duckduckgo.com/?q=poe&o=json' |
      jsonpath -o '$..Topics.*.FirstURL' -o '$..Topics.*.Text' |
      grep -i perl |
      jmap -i col0 -o url -i col1 -o title |
      json2yaml
      

Output:

    
    
      title: Perl Object Environment, a library for event driven multitasking for the Perl programming language
      url: http://duckduckgo.com/Perl_Object_Environment
    

Other structured data can be added to this module (Looks like Ingy maybe
adding YAML - <https://github.com/ingydotnet/app-pipefilter>)

------
almog
Got the following compile error when running make (Lion + XCode 4.5.1):

    
    
      bison -W -d parser.y -v --report-file=parser.gen.info -o
      parser.gen.c
      bison: invalid option -- W
      Try `bison --help' for more information.
      make: *** [parser.gen.c] Error 1

~~~
stedolan
(author here) I haven't tried to build on a mac in a while, bison there seems
to support fewer options (must be an older version).

For now, I've just checked in the autogenerated parser, so that bison won't
have to run when you build master. git-pull and try again :)

~~~
almog
Works great now, thank you:)

------
trustfundbaby
FYI if you have python installed (and most do) you can just do

curl
'[http://search.twitter.com/search.json?q=json&rpp=5&i...](http://search.twitter.com/search.json?q=json&rpp=5&include_entities=true)
| python -mjson.tool

------
Draiken
Very nice tool, congrats!

This is something everyone ends up needing at least one time, and we always
end up using a language specific tool.

Nothing beats the command line piping

------
metabrew
See also, jsawk: <https://github.com/micha/jsawk>

------
bryogenic
When doing the tasks this seems to be targeting, Jansson has worked well for
me.

<http://www.digip.org/jansson/>

------
xargs12
Built for Windows using cygwin, works a treat.

~~~
MatthewPhillips
Builds with MinGW as well, although I did have to manually edit a header file
because I couldn't get sed to work properly.

------
ohaal
See also jsontool, <http://trentm.com/json/>

------
evincarofautumn
We periodically have to munge JSON at work—will definitely be trying this out.

------
littlemerman
This is awesome. Thank you.

------
ruggeri
Thanks; very nicely done. :-)

------
vhost-
I vote for a man page!

~~~
codereflection
<http://stedolan.github.com/jq/manual/>

------
nsp
D. DQ,-"l

