
Jsonnet – A data templating language - fhrow4484
https://jsonnet.org
======
arianvanp
Also check out [https://dhall-lang.org](https://dhall-lang.org)

It's GCL but with a strong type system and termination guarantees.

I personally like the syntax a bit more than jsonnet. It's a bit less noisy.

I'm also a bit biased because I work on [https://github.com/dhall-lang/dhall-
kubernetes](https://github.com/dhall-lang/dhall-kubernetes) . We're currently
working on a 2.0 release so the current state of the repo is a bit in turmoil.

Programmable configuration language like dhall an jsonet are very powerful and
I like them way more than for example helm charts, where templating happens on
the string level instead of at the data level.

~~~
Trufa
Just my two cents but I would go against the first interaction with the user
being find the bug, I didn't didn't find it and missed the point, granted I
didn't take much time and I'm on mobile, but I guess that's the reality for
many people if not most :)

~~~
cyphar
The bug is that your SSH keys are in /home/bill/.ssh/. But I agree the first
page really don't help sell the idea.

------
jpgvm
All the people suggesting to use JS instead of Jsonnet are completely missing
the point.

Jsonnet is a functional lazy evaluated language that supports powerful
referential concepts. Unfortunately the website sucks at getting this across.

JS on the other hand is an imperative programming language. If you say "use
JS" you may as well say "use Python", the 2 are equivalent in this context and
neither are in the same class as Jsonnet in terms of robustness and
simplicity.

I didn't get it until I used it but after using it I don't think I would go
back to test templating when it comes to dealing with JSON/things that are
equivalent. i.e I would use Jsonnet to create JSON to convert to YAML rather
than templating YAML directly. Same goes for generating objects directly in
JS/Python/jq - yes it can be done, yes sometimes you can write a nice DSL but
in most cases Jsonnet is going to allow you to write said DSL faster using
functions and it's going to be simpler and less prone to errors.

~~~
mruts
JS is a lot closer to Scheme and Lisps than an imperative language. In fact,
it used to have a lispy syntax before Brenden Eich was forced to change it to
make it more approachable.

~~~
h8hawk
It's not. [http://journal.stuffwithstuff.com/2013/07/18/javascript-
isnt...](http://journal.stuffwithstuff.com/2013/07/18/javascript-isnt-scheme/)

~~~
mruts
As a person who has done _every_ exercise in SICP, that argument is total
bullshit.

Let's go through the list of things that the author says Javascript doesn't
have:

2\. Lexical block scope

JS has lexical function scope, so it would trivial to convert a "lexical
javascript" to a "function lexical javascript." I think it's unfortunate that
JS doesn't have lexical block scope, but it's trivial to implement through
nested function definitions, so I don't think it's a real problem.

3\. TCO

This isn't a feature of a language, it's the feature of a compiler. It just so
happens that the Scheme spec requires the compiler to have TCO. But if the
Scheme spec didn't mention this, would it make a Scheme something different?

4\. Continuations

Continuations are very rare in Scheme, or any language for that matter. I
think someone would be hard-pressed to argue that it's fundamental to Scheme.

6\. S-expressions

There are many extensions to JavaScript, (JSGEN in Allegro, and various other
libraries in SBCL), that allows one to write Javascript in S-expression
syntax. So maybe while Javascript doesn't use S-expressions, one can very
trivially transpile S-exp JS to normal JS.

8\. Macros (he probably should have said hygenic macros)

It's pretty easy to implement a hygenic macro system on top of Javascript. In
fact, people have already done so, one example is sweet.js.

9\. Distaste for mutation

I'm not sure that distaste for something can be part of a language spec. But
it's a non-argument: if all JS programmers suddenly had a "distaste for
mutation" would it be anymore or less like Scheme? I don't think it matters at
all. This point is about culture, not language spec.

So, in conclusion, from the above points, one could trivially write a
Javascript to Scheme compiler that would have all the features of Scheme,
except maybe call/cc. If this is what you define as Scheme, sure, fine, you
win: Javascript isn't Scheme. But for the rest of us, it doesn't really matter
and JS is damn close to Scheme from a practical perspective.

Moreover, it wouldn't even be that hard to add call/cc to JS! So I don't think
that or any of the author's points are very valid.

~~~
Technetium_Hat
JavaScript does have lexical block scope since ES6, with the `let` and `const`
statements.

And continuations are implemented by convention in nodeJS (through callbacks)

It even is specified as including TCO, a classic scheme feature, but so far it
is implemented only in Safari.

Non mutation also seems to be a trend in web development, again thanks to
`const`. The only things that make JavaScript not scheme, then, are syntax and
the lack of macros. (Which is kind of a result of the more complex syntax)

------
yegle
Finally a public version of GCL/Borgcfg :-)

For people who doesn't know, GCL (Generic/Google Config Language) is a
language that uses dynamic scope. You can define a `template` object with a
variable defined as `external`. This way you can create similar objects by
providing a concrete value for those `external`s at the time of instantiating.

The following paper was referenced in Google's borg paper and gives a good
overview of the syntax.

[https://research.tue.nl/en/studentTheses/gcl-
viewer](https://research.tue.nl/en/studentTheses/gcl-viewer)

~~~
akhilcacharya
Maybe it’s because I’m not a top 1% intelligence that I don’t get this but
under what circumstances would configurations become complex enough to where
this becomes necessary?

~~~
snlacks
I don't think it's an intelligence thing just what problems one's come across.
Generating configs for various deployments where resources change? Generating
configs for developer set up? If you start having a lot of environmental
variables or private keys it might be more manageable to get them from another
secure source.

------
davidgould
The Lua language was originally a data templateing language [0], and is still
very good for that, but like all such things, it grew. Contrary to Greenspun's
Tenth Rule [1], Lua's implementation of Lisp is not at all ad-hoc, informally-
specified, bug-ridden, or slow.

[0] [https://www.lua.org/history.html](https://www.lua.org/history.html)

[1]
[https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule](https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule)

------
self_awareness
The name might come from a 'sonnet', but all I can see is a 'JSON parsing
library written for Microsoft .NET'.

------
rococode
AllenNLP uses Jsonnet for config files, for anyone ondering what one potential
use case is.

For example, here's a config for NER with ELMo:

[https://github.com/allenai/allennlp/blob/master/training_con...](https://github.com/allenai/allennlp/blob/master/training_config/ner_elmo.jsonnet)

I guess the only really jsonnet-specific thing here is std.extVar() call, but
generally speaking it's just a more flexible JSON file that doesn't cost much
more to use (brainpower or physical resources). I use it to mix and match
configs while debugging sometimes as you can do things like if statements and
adding dicts together.

It's also nice for modularization, you can do things like having a function
that returns a dict, so if your config needs two slightly different
datareaders you can just pull out all the common parts and call the function
twice instead of having two copies of the entire datareader config (with only
minor differences).

------
badtuple
There's a real need for JSON templating languages not targeting developers.

Tools like Zapier, Customer.io's webhooks, etc. allow integration with
arbitrary APIs and have become a standard part of many marketing/sales stacks.
These apps use things like Jinja, Mustache, or Liquid which are great for text
but not JSON templates.

Curly braces as primary delimiters, not allowing trailing commas on final
elements, and other features of JSON make generating it a real hassle.

Jsonnet is too complex to be that tool, but I'm glad people are working in the
space.

~~~
nfoz
What is it about "curly braces as primary delimiters" that you think is a
hassle? What would you suggest instead?

~~~
badtuple
It is easy to get confused whether a brace is part of the json or part of the
mustache/handlebars/etc.

    
    
      {
        "hello": {"{{place}}":{{{#items}}"{{thing}}":{{n}},{{/items}}}},  
        "urls": [{{#items}}"{{url}}",{{/items}}]
      }
    
    

It's unambigious and can be reasoned through, but it's certainly hard to tell
what structure the template is building at a glance. This is not a complex or
contrived example.

Syntax highlighting, linters, proper indentation, etc. all help...but when
talking about non-programmers who aren't used to this it's an incredible
hurdle. People are also lazy and error prone, so you can't rely on indentation
to make things clear, especially since it's not always clear how you should
indent nesting of the template vs nesting of the json datastructure you're
trying to build.

Most modern templating languages have a config option for changing the
delimeter. Setting it to something like (( or << might help differentiate
things a bit, but then you're throwing away easy use of examples,
documentation, stack overflow questions, and so much more that already exists
for that language. That documentation and copy-pastable community help is not
something to throw away lightly if you're designing a tool for this audience.

It's also worth pointing out that the above template generates subtly
incorrect json with trailing commas. In pure mustache it's not possible to get
rid of that trailing comma. You have to either change your data (add a
"last":true key to the last element of any arrays), or have some kind of post-
processing step.

All of the existing solutions are workable, they just aren't ideal.

~~~
UncleEntity
I hacked together a asdl parser in boost::spirit a while back that used
mustache to generate the C++ AST (which was then used to generate its own AST
to make sure it all worked) and can attest the curly braces in the templates
were a righteous PITA to get right.

If I had to do it all over again I'd probably use the config option and change
them to something like '(.', '.)' that doesn't naturally occur in the output
language.

------
zeliard
There's also another competing json templating project from google in the
works. Doesn't seem to be production ready yet, but it's open sourced already
- [https://github.com/cuelang/cue](https://github.com/cuelang/cue)

------
c_r_w
Some devs from my company just recently published
[https://jkcfg.github.io/](https://jkcfg.github.io/), building on lessons
learned from many years of configuring Kubernetes. Since we are discussing JS
for config management.

~~~
geezerjay
Love the concept, but the name is far from being the best.

There's already a relatively popular tool for JSON processing called jq, whose
name in some languages already reads like "jk".

[https://stedolan.github.io/jq/](https://stedolan.github.io/jq/)

~~~
c_r_w
I had that question and also the "just kidding" question when I heard the
name. What can you do, naming things is hard.

------
falaki
[https://databricks.com/blog/2018/10/12/writing-a-faster-
json...](https://databricks.com/blog/2018/10/12/writing-a-faster-jsonnet-
compiler.html)

------
wryun
Still plugging json-e over jsonnet, since it's valid JSON itself:
[https://taskcluster.github.io/json-e/](https://taskcluster.github.io/json-e/)

------
cwp
This is great. I've been looking for something like this and have been
considering nix, Starlark and Dhall. This seems to hit a sweet spot between
power and familiarity.

~~~
snthpy
I agree, Jsonnet looks really nice.

One thought I had though is that I wish they didn't use the % operator for
string interpolation though and rather the more recent f-string syntax. I
wonder whether it will be possible to add that or if there is a reason to
prefer one over the other in this context?

------
arthurz
We have just had an article hitting nearly the top on not to use JSON for
config [https://arp242.net/weblog/json_as_configuration_files-
_pleas...](https://arp242.net/weblog/json_as_configuration_files-_please_dont)

~~~
fhrow4484
That's the reason I posted this, since this is kind of using, without the
drawbacks (of not having comments etc)

I wonder if folks had had experience using it for kubernetes configs like
[https://jsonnet.org/articles/kubernetes.html](https://jsonnet.org/articles/kubernetes.html)

~~~
weberc2
Pretty sure stripe built skycfg for this very purpose, but it uses starlark
for the language, which is quite a lot nicer IMHO.

------
haxterstockman
No thanks I'll just write JavaScript

~~~
moltar
Exactly my thoughts. I don’t see how it’s better than JS or TypeScript. I can
render any object into JSON.

This is in fact how I manage Kubernetes manifests.

I use this package: [https://www.npmjs.com/package/kubernetes-
models](https://www.npmjs.com/package/kubernetes-models)

Then I write normal TypeScript code to create Kubernetes objects/resources.

Then I render it to JSON and feed to kubectl.

The beauty of this solution is that it’s OOP and DRY.

~~~
atombender
You're assuming that all code can be trusted. Jsonnet is much easier to make
secure than JS, since it can be statically guaranteed to be safe and side-
effect-free (e.g. can't access the file system). This permits evaluating
arbitrary untrusted data, which can be a boon to systems like CI servers,
databases, or even Kubernetes (think Helm charts).

~~~
CraftThatBlock
He is producing JSON, just as you said. Not sure you understood him.

He has a TS script that outputs JSON, which he is using as config.

~~~
atombender
The script to generate that JSON is code. Jsonnet also generates JSON. So his
script is filling the same role as Jsonnet would. The point stands.

He might trust _his_ code, but I was highlighting reasons why this is not true
everywhere, and indeed the ability to evaluate untrusted code can open up
previously unavailable avenues. For example, storing server-side Jsonnet that
can be safely evaluated by a multitenant backend.

------
netingle
We’ve been using jsonnet to combine Prometheus alerts and Grafana dashboards
into little packages called monitoring Mixins: [https://github.com/kubernetes-
monitoring/kubernetes-mixin](https://github.com/kubernetes-
monitoring/kubernetes-mixin)

A bunch more exist for things like Prometheus, consul, etc, vault, cluster,
ceph, memcached etc

We’ve also built a super simple package manager for this called jsonnet-
bundler (jb for short).

And of course we use jsonnet (with the ksonnet lib) at Grafana Labs for all
our Kubernetes jobs.

------
mafro
Why would I use this over, say, Jinja?

Do I really want imperative code in my config files? And if so, why not just
use vanilla JS?

~~~
cle
Why don't you take a look at the link? It has a page specifically addressing
these questions:
[https://jsonnet.org/articles/comparisons.html](https://jsonnet.org/articles/comparisons.html)

~~~
monkpit
The link doesn’t specifically address the question “why not use vanilla JS?”

~~~
cle
see "Comparison With Other Scripting Languages"

------
gfiorav
I needed a templated language recently (couldn’t do with Mustache methods).
Looked at jsonnet but it looked quite abandoned. Ended up finding Json-E [0].
Pretty powerful, and give you the option to pass in methods to the context as
to expose them. No support for Async methods, but I wrapped around it with
some Promises and did the trick.

[0]
[https://github.com/taskcluster/json-e](https://github.com/taskcluster/json-e)

------
beders
If your configuration files are so hard to write that you need to use a
templating language that is even harder to write, maybe you need to re-think
what a configuration is.

------
randomcoder1000
This looks like a public version of gcl/borgcfg... But with JavaScript mixed
in (so even worse, if that was possible)

~~~
atombender
Jsonnet does not have JavaScript, though. It's _JavaScript-like_ , but not
nearly as complex (e.g. no prototypes, classes, etc.).

~~~
randomcoder1000
Yeah, I guess this doesn't use JavaScript expressions so it wouldn't have the
weird stringiness and truthiness rules JS has, which is what I was worried
about. Still, there are a lot of other issues with BCL.. I think I would
rather see a piccolo style solution done with Skylark instead of BCL+json.

From reading the website I see it actually was inspired by gcl/bcl for
kubecfg, so hopefully they used some lessons learned!

------
nikolay
Jsonnet has been around for a while, but it looks like stale. Its stdlib is
limited and nonstandard and the whole lazy evaluation is something new
developers struggle with. I've been using it at work for 2 years and it's sad
that it hasn't moved much. I think it lost a huge opportunity if it had more
active and more open for innovation development. I'm sorry to say that, but
the lead is busy with stuff and is highly opinionated. Also, Jsonnet is slow.
We have a few hundred Jsonnet templates and it takes a very long time to
compile, which is a pity given it's written in C (now being slowly and
painfully rewritten in Go for whatever reasons).

------
okket
Previous discussion from 9 months ago:
[https://news.ycombinator.com/item?id=17363871](https://news.ycombinator.com/item?id=17363871)
(55 comments)

------
paultopia
I would be interested to hear what heavy lispers think of this. The whole
mixing code and data together feels very lispy, albeit with a heavier syntax
here... is this sort of a step off from s-expressions, or does it occupy a
fundamentally different place?

------
mafro
If there's anyone here who has used both dhall and jsonnet in anger, can you
comment on your experiences?

[https://news.ycombinator.com/item?id=17529722](https://news.ycombinator.com/item?id=17529722)

------
bboreham
I always want to know what the debugging experience is with any new language.
Being side-effect free is cool and all but if I can’t actually figure out why
it did A and not B then I’m going to get frustrated.

------
manigandham
Considering how easy it is to run Javascript anywhere, even small inline
statements interpreted dynamically, I can't think of a single reason to use
this instead.

------
mullikine
It's a nice tool to generate json, but then so is jq. I hope to start using
it.

jsonnet /dev/stdin <<< '{x: 1, y: self.x} + {x: 10}'

{"x":10,"y":10}

------
antoineMoPa
Configuration is nice, but good defaults are better, and this looks like an
invitation to make super complex configurations.

------
h1d
When do you need to use a complex configuration system like this?

Aren't they supposed to be predefined values?

~~~
perlgeek
[https://gocd.io/](https://gocd.io/) allows you store CI/CD pipeline
configuration in a git repository.

If you have many different components, you really want to factor out common
things into templates (like build, deploy), with the option to disable some
stages (some components might not have end-to-end tests, for example), plug in
some values etc.

I've long been on the lookout for a template system for data structures for
this use case, so I'll take a look at some of the suggestions in this thread.

------
peterwwillis
I have never used a language/tool like this at scale and enjoyed it.

For example, embedding a bash script in a templating language. This is such an
incredibly crap idea I don't know how it keeps getting perpetuated. Use of
this pattern works as follows: user A writes the script, tests it, shoves it
into template. user B modifies the template, commits it, it later breaks
silently because it was not tested, because it's in a template language, so
you can't lint it and it has no tests written for it (because how did you know
it would even get generated based on your input/function, let alone all the
places it would be used and how, because templates). user C finds the hidden
breakage after tearing out hair, removes the script from template, and writes
it as a normal script that takes options and writes a test for good measure.
(If you try to use normal dev patterns around the templates, like generate-
template-then-commit-to-vcs, relying on and testing the generated versions,
your tests have to be reworked whenever you regenerate, and later the
generator gradually begins to break, so you just modify the copy in VCS rather
than using the template)

A separate problem is that templating languages that give you advanced
functionality lead you to want to use that functionality, and the more you do,
the harder it is to maintain over time. It may not be readily apparent that
trying to make Ansible configs work declaratively requires an annoying amount
of complexity, or writing entire applications that cannot be easily reused and
calling them "modules" or "plugins". (Whatever you are trying to
template/generate, the only way to know for sure you're doing it validly is to
have a function that actually parses and outputs the format according to a
spec, because just spitting out a template and then externally validating it
becomes very difficult to debug, because you don't know why the template is
spitting out the output the way it is. So your templating language eventually
becomes a compiler)

Since this is a new language, you also have to re-implement everything you
normally have for a language, like a plug-in for your IDE, linter, debugger,
code analyzer, port the standard libraries to different systems, etc. And of
course all new devs have to learn this thing and all its quirks before they
can start doing work. An entire ecosystem eventually needs to exist to support
its continued use, and if you have problems, you hope this ecosystem continues
to flourish so you can get support for this unsupported language tool thing,
or you'll be hiring internal devs to do it.

Maybe the problem is these start out as tools for developers, and only when
they get used for operations do the anti-patterns surface. I've gone through
it several times and it always ends up a PITA.

~~~
zaphar
Tools like jsonnet and dhall exist to address the very problems you have
called out. You may want to give them a closer look.

~~~
peterwwillis
They appear to be designed for developers' convenience, not operational
excellence.

~~~
zaphar
As someone who does a fair amount of operational work I would much rather use
these tools than the other options so I disagree with your statement.

------
mshahi8210
Lua is also good for same purpose.

------
sheeshkebab
Ehh... this feels ugly and dirty (not that js code for generating similar json
files is typically any better)

