
Clojure's missing piece (2017) - tosh
http://nathanmarz.com/blog/clojures-missing-piece.html
======
vemv
As one can see in two other top-level comments in this thread, the choice of
an all-caps DSL is off-putting.

It would be more idiomatic to have a hiccup-like API, where data is passed -
not code, not syntax. And the keywords should be namespaced, so meaning is
self-documented (and one gets superior IDE support).

~~~
yogthos
A vendor my team works with actually open sourced something along those lines.
They provide a FHIR (medical data format) repository, and we need to do
mappings between this format and data models used in our other systems. They
worked with us to make a data driven DSL on top of Specter.

[https://github.com/HealthSamurai/ironhide](https://github.com/HealthSamurai/ironhide)

------
fmjrey
Funny this is being posted as I just started to use specter for extracting
entities from XML documents we receive. I never used it before but now I see
the value of:

\- paths as data

\- navigators as transducers

First I started using plain clojure code, then played with using xml-in [1],
then created my own transducers and channels to build a dataflow-like pipeline
manually, and finally started using specter because I wanted paths to be a
data structure I can use to build the pipeline programmatically instead of
manually. For that I needed a way for paths to become tranducers (which
specter traverse-all does [2]), and I also needed paths to be a sequence of
literals or "interned" elements [3] such as keywords or symbols, but not
functions created on the fly. That way I can create a tree from a sequence of
paths, using something similar to this SO answer [4], and build the pipeline.
I'm almost there, I should end up in a situation where I can specify a list of
paths to extract from an XML document, and the dataflow pipeline needed will
be built automatically, with an input channel for the parsed XML, and output
channels for each given path. If you use clojure.data.xml instead of
clojure.xml, my hope is to have a one pass XML parsing of larger-than-RAM XML
docs.

[1] [https://github.com/tolitius/xml-in](https://github.com/tolitius/xml-in)

[2] [https://github.com/nathanmarz/specter/wiki/List-of-
Macros#tr...](https://github.com/nathanmarz/specter/wiki/List-of-
Macros#traverse-all)

[3]
[http://www.yourdictionary.com/interning](http://www.yourdictionary.com/interning)

[4] [https://stackoverflow.com/questions/49515858/clojure-
convert...](https://stackoverflow.com/questions/49515858/clojure-convert-file-
path-to-tree)

------
Expez
Specter is a fantastic solution to a problem you should aim to avoid.

~~~
islon
How do you avoid having complicated, nested data structures in your software
if your business is complicated?

~~~
augustl
One answer to this is to have your raw "source" data stored as flat as
possible (Datascript, RDF, ...). Then you generate nested structures (you
could call these "views") as much as you want from this data. That way, you
don't have to write to nested data, because that's the pain point imo.

~~~
fnordsensei
If you control the source, that's fine. I get externally generated documents
containing an annoying mishmash of XML representing metadata and XML
representing rendering logic. Sometimes the same stuff even represents both of
these things.

I take them apart to something flatter as early as possible, but the
complexity is nevertheless unavoidable.

(changing the upstream representation would involve changing an entire
industry as well as a standard)

------
js8
Is this something like Lens in Haskell?

~~~
thaumasiotes
> If you have a Haskell background, I'm sure you're screaming to yourself
> "Lenses! Lenses!" I actually didn't know about lenses before I made Specter,
> but they are certainly very similar. I'm not on expert on Haskell, but what
> I do know is it explicitly distinguishes between navigators that go to one
> element (Lens) vs. zero or more (Traversal). I fail to see how that
> complication adds any sort of expressive or performance benefit, but perhaps
> a Haskeller out there can educate me.

~~~
involans
It is useful to be able to speak of Lens and Traversal to distinguish intent.
All lenses are Traversals, but not vice versa. What you have here is of course
the Traversal system without the other optics.

~~~
Tarean
It's also useful for stuff like 'get' to look differently.

    
    
        user ^. id
        txt ^? _JSON . key "foo"
        txt ^?! _JSON . key "foo"
    

First one gives exactly one result. The second one is the equivalent of
returning null on failure. The third one throws an exception on failure.

And because of the somewhat peculiar implementation of van laarhoven lenses we
get subtyping so it doesn't get in the way.

------
lbj
Calling this the missing piece is no understatement. I'd love to see the
design challenged by rhickey and integrated into core.

~~~
nickik
Its has already been rejected from the core

~~~
corporateguy6
Mind elaborating as to why?

~~~
cbcoutinho
Because Rich wants to keep core small

[https://groups.google.com/d/topic/clojure/qN1UPMVQmaM/discus...](https://groups.google.com/d/topic/clojure/qN1UPMVQmaM/discussion)

~~~
zackmorris
I've never used Clojure but am extremely interested in ClojureScript and
Elixir (or some hybrid of the best of both) running under a JavaScript-like
syntax so that developers can build from their existing contextual knowledge.
So far the closest language that I've found is Skip:

[https://news.ycombinator.com/item?id=18077612](https://news.ycombinator.com/item?id=18077612)

My comments:

[https://news.ycombinator.com/item?id=18080968](https://news.ycombinator.com/item?id=18080968)

[https://news.ycombinator.com/item?id=18086937](https://news.ycombinator.com/item?id=18086937)

I'm not familiar with several of the functional programming terms brought up
so far, but I think the gist of them is that all data structures need
deterministic iteration.

So for example, I can understand how returning a copy of an immutable map
might result in a new map whose values are in a different order than the
original. But that isn't acceptable. Either the map's order is determined by
aspects of its values (it's deterministic), or it's not. This might even need
to be true for structures like sets that aren't thought of as having order.

This is a pretty serious situation for not just Clojure but a whole host of FP
languages. Maybe this determinism is as important as immutability or process
isolation. It certainly should rank higher than the need to minimize bloat. My
vote is for Clojure to address it, even if it doesn't incorporate Specter.

------
thom
My last project relied very heavily on zippers. I generally find that even
something like Specter wouldn't be helpful here because quite often the
branching is highly conditional on data (e.g. find all Foos and give each one
a reference to the next Bar in the sequence etc). Of course it can quickly
become complicated to rewind when making a complicated excursion in a zipper -
I'd love to hear what the state of the art in this stuff was (presumably from
a Haskeller).

~~~
lmm
Maybe you want some kind of histomorphism? (Perhaps even a zygohistomorphism
if you're alternating between Foos and Bars)

~~~
whitten
For those, like me, who might not be familiar with those terms, this
StackOverflow question strives to answer it:

[https://stackoverflow.com/questions/36851766/histomorphisms-...](https://stackoverflow.com/questions/36851766/histomorphisms-
zygomorphisms-and-futumorphisms-specialised-to-lists)

Oh, and what is a zipper as a data structure?

~~~
cstrahan
I'll preface: this is more of an interesting connection, rather than a direct
answer to your question.

Zippers can be seen as (or at least isomorphic to) the derivative of a type
with respect to one of its type parameters. This is described in Conor
McBride's paper "The Derivative of a Regular Type is its Type of One-Hole
Contexts."

[http://strictlypositive.org/diff.pdf](http://strictlypositive.org/diff.pdf)

If a video is more your speed, Kenneth Foner gave a great talk titled
"`choose` your own derivative" that takes the concept of derivatives of types
and takes it a step further. You can skip to the 11 minute mark to watch his
explanation of zippers as derivatives here:

[https://www.youtube.com/watch?v=79zzgL75K8Q&t=11m](https://www.youtube.com/watch?v=79zzgL75K8Q&t=11m)

With that said, the standard go-to paper for Zippers would probably be "The
Zipper" by Gerard Huet:

[http://gallium.inria.fr/~huet/PUBLIC/zip.pdf](http://gallium.inria.fr/~huet/PUBLIC/zip.pdf)

------
vbuwivbiu
I only wish it didn't use shouty CAPS for the navigation

------
purple_ducks
Always found it odd that a data oriented language didn't have XPath
equivalent.

~~~
amelius
Perhaps a stupid question, but what language isn't data oriented?

~~~
zimablue
The meaning of data oriented in this context is that it's idiomatic for
functions to receive and return plain data (primitives like map, list). In
most languages it's idiomatic to receive and return mostly your own defined
classes.

~~~
amelius
Ok, thanks for the explanation, but I don't see why languages where you use
more heavy machinery to define and check the types of your data would be less
"data oriented" than languages where you use mostly primitives.

~~~
aasasd
If you have a limited set of types, you can make extensive use of higher-level
functions that compose frequently-needed operations on those types, and add
your closures in the mix. If you have extensible types instead (OOP and such),
you either have to program to these types, recreating the generic
transformations for each type, or you'll need functions that can do those
transformations on arbitrary types―presumably via internal mechanisms―thus
circumventing strict type checking.

~~~
lmm
Or you need to be able to produce appropriately parameterized transformations
on user-defined types without compromising safety, such as typeclass
derivation?

------
agumonkey
The birth of specter. Very interesting article.

------
knubie
Is it possible to use these functions on their own like 'map-values' without
the traversal DSL?

