
Select Transform: JSON template over JSON - gliechtenstein
https://selecttransform.github.io/site/
======
arkh
Gotta love JSON. It's XML without the tools so you get to reinvent everything
to pad your resume.

    
    
        XML -> JSON
        WSDL -> JSON schema
        XPath -> JSON path
        XSLT -> JSON template?
        SOAP -> Swagger and friends
    

One or two decades from now there will be too many tools and things to learn
about JSON so another generation will reinvent a new "perfect format for
everything". And a new cycle will have started.

~~~
rtpg
I kinda get this criticism, but this is also a bit like lamenting that we
write the same libraries for different programming languages. I mean... yeah.
Some ideas are good across protocols. That's why they're copied.

On a merits-based thing, if you're just looking at XML, it has a lot of things
going against it .If someone sends me something "in JSON", I can have a good
guess what it'll look like. Not as much luck with XML. XML punishes you for
using attributes (because you can't place composed types within them), but the
awkward alternative is placing attributes in sub-elements.

I have a list of products with prices. I store the price as a quantity
attribute on a <product>. I decide later on to store currencies for all my
"money" objects. Do I now double the amount of attributes? It's all super
awkward, and JSON's mental model is more straightforward, even if you end up
with approximations.

~~~
jimktrains2
Why is it super awkward and how does JSON solve this problem any better?

~~~
tchaffee
JSON is a lot easier to read.

    
    
        {
          "employees": [
            {
              "name": "John Crichton",
              "gender": "male"
            },
            {
              "name": "Aeryn Sun",
              "gender": "female"
            }
          ]
        }
    

versus

    
    
        <employees>
          <employee>
            <name>John Crichton</name>
            <gender>male</gender>
          </employee>
          <employee>
            <name>Aeryn Sun</name>
            <gender>female</gender>
          </employee>
        </employees>

~~~
skocznymroczny
Or

    
    
        <employees>
          <employee name="John Crichton" gender="male" />
          <employee name="Aeryn Sun" gender=female" />
        </employees>

~~~
Matt3o12_
I believe that was OP's point. Using the annotation you cannot add an
attribute with subattributes. What if you wanted to add the employee's last
six salaries. Suddenly you need to restructure your xml:

    
    
        <employees>
          <employee>
            <name>John Crichton</name>
            <gender>male</gender>
            <salaries>
                <salary>10000</salary>
                <salary>10000</salary>
                <salary>10000</salary>
            <salaries>
          </employee>
          <employee>
            <name>Aeryn Sun</name>
            <gender>female</gender>
          </employee>
        </employees>
    

Using JSON, you just so: “salaries”: [10000,10000,10000]

JSON is not only less verbose, it is also more flexible and easier to read and
understand. You don’t have to worry about should I use tags or attributes for
that because I late might have to use sub tags, and that makes it far easier
to use (and honestly parse as well because many XML documents I have gotten
are very inconsistent).

~~~
bouke
Why do you need to restructure the xml? You can just keep the attributes. To
add the salaries there are quite a few options, as XML is just as flexible as
well.

Using a List (XSD List) type:

    
    
        <employees>
          <employee name="John Crichton" gender="male" salaries="10000 10000 1000" />
          <employee name="Aeryn Sun" gender="female" />
        </employees>
    

And using mixed content (complexType):

    
    
        <employees>
          <employee name="John Crichton" gender="male">
            <salary>10000</salary>
            <salary>10000</salary>
            <salary>10000</salary>
          </employee>
          <employee name="Aeryn Sun" gender="female" />
        </employees>

------
rtpg
I think there's a lot of useful unexplored territory in this sort of tool. So
much work nowadays is happening in maps/dictionaries/what-have-you, with a lot
of work being, essentially, data shape transformation.

Despite this, many languages aren't great at this! we have tools that work
alright on first-order operations but fall apart on higher level things.

Even really simple things like "conditionally include a key" are rarely
supported in a way that leads to concise code. How many times have we all
written

    
    
      foo = {...}
      if bar:
         foo['a'] = x
    

instead of something like

    
    
       foo = { ... , 'a': x if bar }
    

and then having bugs because of branches? Tools that solve this sort of
problem will be to modern programming what things like the Unix shell was to
data mungers in the past.

Haskell lenses kinda goes into this stuff, though Haskell itself is a bit hard
to use in a lightweight fashion. Clojure has Specter. But still looking for
some more stuff to fill this hole, especially in the "transitionally typed"
stuff like Typescript or Python + Mypy

~~~
sly010
IMHO any tool that converts JSON to JSON is pointing to the wrong direction.

Json should strictly be used as a serialization format and everyone who
touches it should immediately parse it into a typed data-structure that
represents your domain better.

In fact you really should represent your data in your code using whatever
native object hierarchy / algebraic data types / option types / etc. your
language supports. (Algebraic datatypes are awesome at this sort of thing, it
allows you to avoid the aforementioned conditionality)

The problem is the right way to do this is also the hard way, because people
hate writing "boilerplate" classes and serialization code and once you start
thinking about making it "easy", you end up using json as the in-application
data structure because "who needs types anyway".

~~~
earthboundkid
You are very correct. Boilerplate (de)serialization code is 100x better to
have to write than code with JSON-blobs pretending to be business objects.

------
mishoo
What's wrong with:

    
    
        return {
            labels: data.items.map(function(value){
                return {
                    type: "label",
                    value: value > 10 ? value : value * 100
                };
            })
        };
    

I hope your project does not get too popular.

~~~
addicted
Even cleaner with ES6.

return { labels: data.items.map(value => { type: "label", value: value > 10 ?
value : value * 100 }) };

------
turtlebits
Looks cool, but I can't help but think that you're just reproducing what map()
and filter() do, without any real benefits?

~~~
XaspR8d
I perceive it being middleware as the benefit. For when your client isn't set
up to manipulate the incoming data and you can't edit the backend resource
that generates it. Or vice versa.

------
DEinspanjer
This seems pretty interesting, and I will be happy to play around with it.

I am a bit concerned that there is no discussion in the docs of the potential
security risks of allowing direct native JS execution of arbitrary
instructions passed in by an untrusted source.

I use a project called JSONLogic (jsonlogic.com) which bears some similarity
to ST in terms of being able to select and transform values. The biggest
advantage I see with it is, unless you explicitly plug in a rule that parses
and executes user data, there is no way for the data to "escape the sandbox".
This means you can safely build a query syntax on top of it where you can
directly consume the arbitrarily complex query from an untrusted source and
execute it in a secure manner.

------
gliechtenstein
Hi, my name is Ethan. I'm the creator.

I thought I would provide some context on why I wrote this library, and how
I'm using this right now. So here it goes:

Other than ST.js, I also work on another open source project called Jasonette
([https://www.jasonette.com](https://www.jasonette.com)), which lets you write
an entire native iOS/Android app in nothing but JSON markup.

And when you can express the entire app logic--from model to view to
controller--in JSON, you can split them up whichever way you want and load
them from anywhere (from a file, from cache, from a remote server, or from
local memory).

But the biggest benefit of all is: you can load an entire ios/android native
app from the server in realtime, just like how web browsers load HTML/JS/CSS
in realtime.

When working on Jasonette, implementing model and view was relatively simple.
For model it's natural since JSON is all about describing data. For view i
just needed to come up with a syntax to describe layouts and all the standard
mobile UI components in JSON.

However the biggest challenge was how do i actually describe functions in
JSON. Without a function, it's just a mockup and won't really do anything
meaningful.

Which brings us to ST.js.

When you think about what a function is, it takes one value and turns it into
another. So basically what I needed to do was build something that will take
one JSON, and turn it into another JSON, but most importantly I would have to
do it using JSON. Basically I needed to implement a finite state machine in
purely JSON.

And this is what templates do. So I set out to build a JSON template engine
that turns one JSON into another using a template which itself is written in
JSON.

What's really cool about this is, since the template is JSON (As far as I know
there doesn't exist any prior art that uses JSON as a template, otherwise I
would have used it instead), it has all the benefits of the JSON format
itself:

1\. You can store it anywhere (Most DBMS nowadays support JSON natively)

2\. You can send it over the Internet

3\. You can compose them easily

4\. You can validate them using JSON schema

5\. etc.

To use a more specific example, I use ST.js in both Android and iOS versions
of Jasonette as the template engine. And a JSON template is absolutely
necessary in this case.

For example, if I want to take a piece of device-generated data and render it,
I need to be able to somehow parse it client-side (can't send it back to
server to re-generate a JSON) [http://docs.jasonette.com/templates/#3-device-
api-generated-...](http://docs.jasonette.com/templates/#3-device-api-
generated-data)

This also applies to many cases where the client-side data contains privacy-
sensitive data. The only way to dynamically parse something that happens on
the client side is by writing the template itself in JSON and sending it over
as data.

Anyway, I hope you guys take a look at the examples on the homepage to get a
glimpse of what makes ST.js powerful. Each example is almost its own library,
except that you don't need a separate library for those purposes since all
you're dealing with is JSON.

~~~
fiatjaf
So Jasonette functions are just about transforming the model, all in a
reactive way? Like view->events->transform model->view?

~~~
gliechtenstein
It can transform anything that's written in JSON, which means both model and
view and anything else.

One of the use cases for ST.js in Jasonette is dynamic client-side rendering,
where it takes dynamic data and renders it against the template, after which
it becomes native components.

Another use case is implementing actual functional programming. This one is
not as straight-forward since it involves maintaining another separate state
machine for function call stack. But this is also achieved using ST templates.
Here's a blog post that talks about it in detail
[http://blog.jasonette.com/2017/02/15/functional-
programming-...](http://blog.jasonette.com/2017/02/15/functional-programming-
in-json/) but it's a bit involved.

Also Jasonette has something called mixins, which you probably can guess based
on its name. It lets you mix in JSON objects into another, so that you can
make the app modular. That's also achieved internally using ST.

Overall, I believe even I'm just scratching the surface of what this can do,
which is why I decided to take some time to clean things up and prepare and
open it up as its own repo, because I think there's tons of other things that
can be done by taking advantage of the fact that all that's happening is:

JSON A + JSON B = JSON C

Hope this makes sense!

~~~
dualogy
> _Another use case is implementing actual functional programming_

I remember our chat 8 months earlier on Reddit about this:
[https://www.reddit.com/r/functionalprogramming/comments/5ufm...](https://www.reddit.com/r/functionalprogramming/comments/5ufmy2/designing_a_functional_programming_language_in/)

Yeah. You're still aiming for "functional programs encoded as (essentially)
s-expressions but written in JS-Object-Notation (JSON) instead of LISPy
parens". And that's still fine if you see that as a major leap forward. Not
sure about the audience, those who want to "functionally program a mobile
quasi-native APP that can self-update from a server" and know what the
"functionally program" part means, wouldn't they reach for JS via React
Native? Same selling points implicit already. Is it for those who never
programmed but are expected to declaratively express functional idioms in your
JSON notation without needing to install all sorts of dev tools and SDKs? Not
entirely implausible at all I guess. Curious to see who will end up as your
target audience. =)

------
skocznymroczny
XSLT for JSON?

~~~
athenot
That's the first thing that came to my mind too.

XSLT is very powerful but also incredibly messy because it has to be expressed
in XML.

Doing the same in JSON seems like it would result in the same problem.

~~~
masklinn
> XSLT is very powerful but also incredibly messy because it has to be
> expressed in XML.

Being expressed in XML does not help, but the language is just _bad_ with
implicit data flowing through and very odd constructs, there are XML template
languages which I found much more readable than XSLT e.g. I remember my
experience with genshi much more fondly than the years I spent with XLST.

------
Edmond
Have you considered reaching out to DevOps folks to consider something like
this for complex configurations?...There are related solutions such as
jsonnet/ksonette but I think something like this might be preferable.

I work on a similar problem though using a UI based approach, I gave a demo to
the Kubernetes SIG App group a while back which be found here:
[https://www.youtube.com/watch?v=alEzE8MSNaI&t=30m](https://www.youtube.com/watch?v=alEzE8MSNaI&t=30m)

Btw Ethan, maybe my email is landing in your Spam folder? I've tried reaching
out :)

~~~
gliechtenstein
Thanks! I actually built this library for my own usage (Jasonette) and decided
to open source it because I realized it can be powerful for all kinds of other
purposes.

In essence ST.js is a finite state machine implemented purely in JSON. So I
can imagine it being used for things I haven't thought of, which is why I open
sourced it. Would be really cool.

Please feel free to reach out at ethan.gliechtenstein[at]gmail thanks!

------
vvkmnn
Now that's a goddamn demo. No fussing about.

I can't wait to play around with this.

------
xfour
Currently I'm having issues trying to figure out a new stack where we can't
figure out where to put the backend for front-end logic in a React App over
the top of GraphQL. I've been thinking of implementing something like this as
the transforms or the formatters don't make a ton of sense to be on the client
or the API or in GraphQL and I'm loathe to make another API call in the chain
as it feels like I there's enough of those already?

Would that be a good use case for this?

~~~
gliechtenstein
Creator here. Yup, in fact you can check out something like that here
[https://github.com/SelectTransform/JSONQL](https://github.com/SelectTransform/JSONQL)

Basically you can send the JSON template over JSON to the server, and let the
server transform the raw data using the template before returning.

The most important benefit here is that, since everything can be expressed in
JSON, it can be stored or transmitted over the Internet, which enables
interesting cases like this.

------
fiatjaf
I like the fact that this exists, but I prefer to use jq.

I also would like to know in which situation this is preferable to jq.

~~~
astral303
There's also [http://jmespath.org](http://jmespath.org) \-- the syntax for
which is supported in AWS CLIs.

Not sure which one is better at this point. Using jq still often feels like
trying to decode a code golf contest entry.

~~~
fiatjaf
Really? I find jq syntax rather pleasant to read and write.

------
xer
Just saying learn ramda

------
dschuett
This is game changing.

