Hacker News new | past | comments | ask | show | jobs | submit login
Supdate – A Clojure library for transforming nested data structures (vvvvalvalval.github.io)
155 points by tosh on May 19, 2019 | hide | past | favorite | 16 comments

This is neat! When I first started reading and it described itself as "transforming nested data structures" I immediately thought of specter, which we use quite a bit at Latacora. We've published a library, eidolon[0], with common specter navigators.

To get a feel for the difference I translated a few of the examples to what they would look like in specter.

A few things I learned:

- supdate is very useful if the data you have is already regular and not too convoluted. By comparison, most of the time the data I pass to specter is a giant (>100MB serialized and compressed) rat's nest (data from every AWS service under the sun).

- supdate is very good at this "implicit projection/selection" (in the relational algebra sense). It feels a lot more like Datomic pull syntax that way than specter. By comparison, the equivalent specter thing made me write a pretty complex multi-path with submap to get the same behavior. By comparison, specter feels better at the logic programming equivalent of this: in specter, I express the data I want to address and specter figures out where it lives; in supdate, I tell you exactly where it lives, structurally.

- Implicit selection (per previous point) feels like something that's maybe most valuable for data representation, so, UIs? To quote Perlis, "It is better to have 100 functions operate on one data structure than 10 functions on 10 data structures.", so something I have done a few times with specter is write a navigator for commonly accessed things (like the actions in an AWS IAM statement) and then reuse that all over the place. It's harder for me to see how to do that in supdate, but I guess it's all data so it can't be that hard :)

In conclusion: I think it does what it says on the tin. There are things that it can't express (or at least I can't figure out how), but it's very clear at expressing the things it can. I probably won't end up using it, but that's not an indictment of the tool as much as it tells you what we use specter for :-)

[0]: https://github.com/latacora/eidolon

Libraries like this are an argument for dynamic types. In Clojure, all libraries work with associative or sequential collections, which is why you can compose supdate with literally anything.

We do lose the embedded proofs that statically typed APIs give. We regain some of the confidence with clojure.spec'ing them.

I wonder if there's a way to make a compile time spec - enforce contracts at API boundaries, but let stuff like supdate work in a highly generic way in between. C++ templates and Rust traits could work, but everyone would need to agree on a core set to use.

Regarding compile-time spec checking — Spectrum is working toward just that. https://github.com/arohner/spectrum

Spectrum would be great, but I think it's still way off from being usable. Taking a line from Specter's page, static analysis that catches idiotic bugs (es mis-typed keys) would really be "Clojure's missing piece". I have been trying it on and off for a bit of time now, but it still breaks on anything but the simplest cases.

This looks like a fantastic library for updating nested data structures -- something that's definitely been a pain point in writing Clojure.

I wrote a library that complements this nicely -- it's for transforming one deeply nested data structure into one that has a different shape, for example:

(f/transform {:a [1 2 3] :b [2 3 4] :c [5 6]} {k [vs]} {vs #{k}}) => {1 #{:a}, 2 #{:a, :b}, 3 #{:a, :b}, 4 #{:b}, 5 #{:c}, 6 #{:c}}


Love it! This looks like it solves a problem I have for an community project (parsing a 2mb convoluted json with status data for 100+ devices) in an elegant way!

What skills does one need to have to be able to write such libraries for yourself? I'm just feeling like a mediocre copy&understand&adapt&paste programmer but rarely I'm able to create unique concepts/software.

Exposure to several different ways of solving similar problem sets could help you think outside of the box (or in many boxes at once, at least). Clojure is a great choice for one box. Maybe look into what Haskell or ML do with type systems, or what Smalltalk does with message passing. I picked these to complement Clojure's strenghts.

Algorithms and Data Structures.

Some related work to look at, if you're interested in approaches like this, are the pattern-based syntax transformers from Scheme (e.g., early `syntax-rules`) and Racket, and the XML transformation languages.

What a coincidence. I've been working on a macro to do exactly this yesterday and think I might have a working version tonight or tomorrow. In Clojure too.

The only difference is in the way I decided to treat arrays

    (xxx {:a [1 2 3]}
         {:a [_ _ inc]})
    ;; {:a [1 2 4]}
This is because I intend to use it on arg vectors rather than proper data. So I'm fine with having to access everything step by step from the left and having a one to one relation between leafs in the data tree and leafs in the transformer tree. Actually I wrote a cowalk function (as well as coprewalk, copostwalk, etc), to allow for traversal and editing of multiple data structures at the same time provided they have the same shape.

Interesting! In supdate the transform would be:

{:a {2 inc}}

This looks like it enables functionality very similar to specter. https://github.com/nathanmarz/specter


1. Much easier to grok syntax/easier learning curve

2. Not as powerful/generic

As the author, I fully agree with this description!

The cost of abstraction should not be ignored, and that's why a lib like supdate sometimes makes sense before reaching out to Specter, lenses etc. (although I love those too!)

    (require '[clojure.string :as str])
Infix, really? I love both Lisp and Haskell. Code like this makes me wonder why I ever strayed from Haskell.

It’s not infix, but keyword args which have been part of lisps since at least the cltl2

What do you mean by infix? This is prefix, the function is first, and the argument comes after.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact