
Spec-ulation – Rich Hickey [video] - Moocar
https://www.youtube.com/watch?v=oyLBGkS5ICk
======
programnature
This is a very thick talk, one Rich's best ever IMHO.

The first point is how we talk about 'change' in software, to center around
what things 'provide' and 'require'.

Breaking changes are changes that cause code to require more or provide less.
Never do that, never need to do that. Good changes are in the realm of
providing more or requiring less.

There is a detailed discussion about the different 'levels' \- from functions,
to packages, artifacts, and runtimes, which he views as multiple instances of
the same problem. Even though we now have spec, theres a lot of work to
leverage it across all those different layers to make specific statements
about what is provided and required.

~~~
mshenfield
I found value in dissecting the different levels of change. For the sake of
sanity though, we should do breaking changes. Breaking changes exist because
we have limited capacity as individuals and an industry to maintain software.
This is especially true for infrastructure that is supported by (limited)
corporate sponsorship and volunteers. Breaking changes limit our window of
focus to two or three snapshots of code, instead of having our window of focus
grow without bound. Our limited capacity can still be effective as a library
changes over time.

The most important point of this talk is here: "You cannot ignore
[compatibility] and have something that is going to endure, and people are
going to value" [0]. Breaking changes provide a benefit for library
developers, but it is usually damage done to end users. As consumers we should
weigh the cost of keeping up with breaking changes against the quality of a
tool, and the extra capacity its developers are likely to have.

[0] [https://youtu.be/oyLBGkS5ICk?t=4177](https://youtu.be/oyLBGkS5ICk?t=4177)

~~~
blueprint
Who says you have to maintain old code? We're talking about simply not
deleting it and establishing a discrete semantic for the new version as
truthfully, a new version is new content which demands a new name to
accurately and precisely describe it. If it didn't it would be like saying
different content doesn't produce a different hash.

~~~
mshenfield
You're right, there is no obligation to maintain it. I think that misses the
point though. The value in keeping the code is to allow the end user to
continue to enjoy improvements in parts of the library that don't have
breaking changes without upgrading those that do. You could continue to have
security patches installed, for example. That value is much less when you
don't do basic maintenance implement bug fixes and security patches.

~~~
blueprint
Unless I'm missing something… the answer to that problem is to (a) factor the
code sufficiently to then (b) create an abstraction (interface) that backs out
the concrete implementation to the specifically desired version/functionality.

------
DEADB17
I don't understand why he says semantic versioning does not work. In my
experience (with NPM, not maven) it is very useful, adding meaning of intent
by convention:

Given a version number MAJOR.MINOR.PATCH, increment the: MAJOR version when
you make incompatible API changes, MINOR version when you add functionality in
a backwards-compatible manner, and PATCH version when you make backwards-
compatible bug fixes.

I got the impression that the issue was maven not being able to handle
multiple versions of the same package/artifact, not in the convention.

~~~
sheepmullet
Like Rich mentioned from the point of view of a library consumer it's:

PATCH: Don't care MINOR: Don't care MAJOR: You're screwed

MAJOR is simply not granular enough and MINOR and PATCH are pointless.

Sometimes when I update to a new major version of a dependency it all just
works. Other times I've got to spend weeks fixing up all the little problems.

Did you break one thing I didn't even use? Update the MAJOR version. Did you
completely change the library requiring all consumers to rewrite Update the
major version.

~~~
DEADB17
Being screwed would be the case if the consumer of the artifact is forced to
upgrade. i.e.: if versions cannot coexist in a code base.

Otherwise I think the information that they convey is useful:

PATCH: improvement or correction that does not affect the consumers
expectation (safe improvement)

MINOR: additional features that may be useful directly to the consumer or its
transitive dependencies (safe to upgrade)

MAJOR: No longer safe to upgrade automatically. The consumer may need to
investigate further or stay with the previous MAJOR.

In any case it is useful information being conveyed. The consumer decides how
to act on it.

~~~
sheepmullet
> Being screwed would be the case if the consumer of the artifact is forced to
> upgrade. i.e.: if versions cannot coexist in a code base.

In theory multiple versions can co-exist in a codebase in js land.

In practice though the vast majority of js libs don't compose well with
different versions.

At best I've found it can work as long as you consider them seperate and
unrelated libraries (basically the version in js land is part of the
namespace).

Edit: I definetly don't think it's as big of an issue in js land as in Java
land because of transitive dependency handling.

------
gniquil
Haven't watched through the entire video, but the first part, analyzing
dependency bottom up, and claiming version doesn't work, really reminds me of
graphql. We've been doing traditional rest with /v1/..., /v2/.... This sucks
for so many different ways. Graphql's solution of providing contract at the
lowest level and just evolve the "graph" really feels a bit like what he was
talking about in this video. And note that Facebook by not "provide less" or
"require more" in their API is how they made their API still compatible with
older clients. This talk is very interesting! (note I could be spewing BS here
as I've not finished the video)

~~~
opvasger
exactly my thought, having dug into GraphQL recently and seen a number of
talks about it.

Getting over the annoyance of looking at some initial functionality that is
misplaced can be quite hard... It's really tempting to just get rid of those
as you're bumping up the MAJOR.

------
anonfunction
Rich Hickey gave one of my favorite talks that I recommend to all programmers
no matter which language they code in:

[https://www.infoq.com/presentations/Simple-Made-
Easy](https://www.infoq.com/presentations/Simple-Made-Easy)

------
taeric
The whole section on "just change the name" fits interestingly in with the way
that the kernel is developed. They have been rather strict on this for a long
time. A function released to the wild stays. Changes to it require a new
function.

~~~
Rapzid
[http://man7.org/linux/man-
pages/man2/syscalls.2.html](http://man7.org/linux/man-
pages/man2/syscalls.2.html)

------
tbatchelli
This is a worthwhile talk, albeit a bit too long. In this talk RH provides a
framework to think about libraries, how they change over time and how we
should be communicating those changes. He proposes a way forward in which the
changes are communicated in a way that can be dealt with programmatically, a
much better framework than versioning schemes (semver gets a beating in this
talk!).

~~~
adwhit
Video is a high-bandwidth way of delivering information at a low bandwidth.
But try watching tech videos at 1.5 speed, it makes them much more engaging.
Also works for ponderous anime.

~~~
asymmetric
Got any examples of ponderous anime? Lain? Akira? GITS?

~~~
Garlef
Dragonball. They sometimes charge up one attack over several episodes.

~~~
cygned
The planet will explode in five minutes.

------
atemerev
I can't get any useful information from videos, it is so slowwwww. Even on 2x
(the fastest I can watch), it is still 3 to 5 times slower than I read. Also,
I can control my reading, skip the minutiae, or reread important parts or look
up some reference — but I can't do that with videos or audiobooks or podcasts.
I wonder how people even find it convenient.

Unfortunately, Rich Hickey really loves videos for some reason, so everything
he says is lost to me.

(Yes, there are transcripts, but those are linear and unstructured.)

~~~
greenyouse
Hey, I usually take notes when reading or watching stuff. If you're willing to
read through my semi-coherent org-mode notes, it summarizes the talk better
than a full transcript.[0]

[0][http://pastebin.com/bmF0xc7y](http://pastebin.com/bmF0xc7y)

~~~
atemerev
Thanks!

------
qwtel
I've revisited this post so many times, I know the number by now:
[http://hintjens.com/blog:85](http://hintjens.com/blog:85), "The End of
Software Versions". I think it is related to, and similar to what Rich is
saying.

------
j-pb
> Logic systems don't have "And nothing else will ever be true!".

Uuh. Closed world assumption? Everything that is not true is false. Most logic
systems do have this. Prologs (not a logic system I know) cut operator even
turns this statement into an operation.

I feel like Rich really gets it wrong this time. His request to support all
the abominations that you have ever written and keep them compatible with
recent changes, might work if you have people pay for using your library and a
company behind it. But doesn't fly if you maintain these things out of
goodwill in your anyhow limited spare time.

The best example of this style going horribly wrong are the linux kernel file
system modules. Different api versions all in use at the same time by the same
code with no clear documentation on what to use when.

It's also ironic that the examples he uses to make his point namely, Unix
APIs, Java and HTML, are horrible to work with especially because they either
never developed good API's (looking at you unix poll), or they, like browsers,
have become so bloated that nobody want to touch them with a ten foot pole.
One of the reasons why it takes so long for browser standards to be adopted is
that they have to be integrated with all the cruft that is accumulating there
for almost three decades now.

"Man browsers are so reliable and bug free and it's great that the new
standards like flexbox get widespread adoption quickly, but I just wish the
website I made for my dog in 1992 was supported better." -no one ever.

Combinatorial complexity is not your friend.

I'd rather have people throw away stuff in a new major release, maybe give me
some time to update like sqlite or python do, and then have me migrate to a
new system where they have less maintenance cost and I profit from more
consistency and reliability.

I think that Joe Armstrong has a better take on this.
[https://www.youtube.com/watch?v=lKXe3HUG2l4](https://www.youtube.com/watch?v=lKXe3HUG2l4)

Also even though I'm a fulltime Clojure dev, I would take Elms semantic
versioning that is guaranteed through static type analysis anytime over spec's
"we probably grew it in a consistent way" handwaving.﻿

~~~
freshhawk
> His request to support all the abominations that you have ever written and
> keep them compatible with recent changes, might work if you have people pay
> for using your library and a company behind it

This is generally his focus, on big professional software. I'm also a Clojure
dev and I'm on the other side of the fence on this one as well so sometimes
I'm disappointed in the enterprise focus but I knew what I was getting into
and it is still worth it. Am I crazy to think that maybe other lisps would
have done better if they had demanded less purity?

Same with the examples of Unix APIs, Java and HTML. Sure, they are all bloated
and horrible to work with. They are also massively, insanely successful. I
think they are great examples because at that scale it's impressive that they
work _at all_.

This is part of the pragmatism that makes Clojure great, they generally stay
away from trying to solve the problem of huge projects being unwieldy and ugly
and painful and instead they accept it as a given and work on tools to
mitigate the problem. For a lot of people backwards compatibility isn't a
choice, it's a requirement set in stone. Even though it always causes everyone
to pull their hair out in frustration.

One day maybe one of these other research languages or experiments will find
an answer to this, and prove that it works at scale. I will celebrate more
than most.

~~~
DigitalJack
The primary advantage of clojure to my mind are the lovely data structures and
the interaction with them.

Common Lisp and scheme can implementation them, and through reader macros
interact in probably the same ways, but it would always be second or third
class.

Second big deal for clojure is the ecosystem.

I'd love a native language like clojure that was driven by a language
specification.

------
ozten
As I understand it, one of the original problems Go was designed to solve was
long compile times at Google.

This talk is fascinating and points out a possible solution space for real
world problems in giant codebases and build infrastructures.

What if 80% of your dependency graph can be identified as dead code paths at
build time and not require you to actually take dependencies on all that dead
code?

~~~
_halgari
Sadly then Go went and shot themselves in the foot. Originally (I assumed this
hasn't changed), you specified dependencies via git URLs. Sounds great, but
then they went and said every dependency always used the head of master. So
now you're in the horrific situation of your deps suddenly changing whenever
your build decides to pull the latest from the git repos.

(Note: I don't use go, but this was the situation about 3 years ago when I
last looked).

~~~
NightMKoder
One of the things Rich talks about in this talk is semantic versioning is
fundamentally broken. In fact any versioning scheme that allows removal is
broken. In some sense this model encourages Rich's ideas better - since you
can't version directly, you either make a new library or maintain backwards
compatibility.

Of course the current 'go get' model has a lot of downsides, e.g. Non
deterministic builds. I still think it's worth considering building on, rather
than trying to fix semver. All that's really missing is a stable monotonic
identifier - something as simple as git tree height may be enough.

~~~
_halgari
Absolutely, I think the "perfect world" would be adopting some of the ideas
Rich proposes but adding (git url + sha) as the dependency management system.

Of course for that to work well you'd also need to protect yourself from
people deleting their git hub repos, or rewriting git history.

~~~
robto
Which is a great use case for something like ipfs, I think.

------
eternalban
I was surprised by this. (Rich is rigorously perceptive and characteristically
peels conflations apart.) Here, he seems to be conflating _linker_
responsibilities with _(unit-module) dependency management_ concerns.

If deployment-unit-A depends on du-B, but system-language level dependencies
have additional subtleties, then that is fine, since someone like RH could be
looking into code stripping of unreachable/dead-code on linking, and I'd be
completely surprised if that hasn't been already looked at extensively to
date.

Yes, you would sacrifice some transfer bandwidth/time on artifact acquisition
but keep conceptual models local and their implementations simpler. Or,
looking at it from another point of view, OP should consider that a
'dependency' is maximally the provider of objects in context of its namespace.

------
greenyouse
When I was listening to the part about breakage vs accretion, I kept thinking
back to the Nix pkg manager from NixOS. It does a pretty good job of
separating out the artifacts by indexing them by version number and a hash of
the username (i.e. artifact-name = sha($user) + $project-name + $semver).
There have to be other examples of pkg managers that do this kind of thing
too...

------
sgentle
I think there is a missing piece to this conversation, which is about the
definition of namespaces. It's easy to conflate "problems with semver" and
"problems with semver+the way my package manager handles versioning".

In npm, or with any package manager that allows multiple independent versions
to coexist, the _version becomes part of the namespace_. Multiple different
versions of a package can be installed at the same time and referenced from
each other. It's no problem if you depend on B1.0 and C1.0, and they depend on
D1.0 and D2.0 respectively; you just get all four installed.

In Rich Hickey's vision, we would move the compatibility contract into the
package name, so if we want to make a new version 2 of helloworld that does
not obey the contract of version 1, we call it helloworld2. In practical
terms, this is no different from the above, it's just that we add the version
number to the namespace through the package name rather than the package
version.

Not surprisingly, this latter strategy is already used by most systems where
versions aren't part of the namespace. Debian has python2 and python3. Python
has urllib and urllib2 (and urllib3). Unix has mmap and mmap2. It works fine,
but it's a bit of a hack.

If your version is part of the package namespace, semver works fine and solves
the problem it was intended to solve. If, like in Java, every version has to
share the same name, then I agree that semver doesn't really buy you that
much. After all, it doesn't help you much to know that your unresolvable
dependency tree is semantic.

It's not a coincidence that npm handles transitive dependencies in the way it
does; it's the whole point. I'm sure it makes npm much more unwieldy to allow
multiple different package versions, but having used it it is exactly one of
those "solves a problem you didn't realise you had" moments.

With that said, I definitely agree that more careful API design would allow
far fewer backwards-incompatible (ie semver-major) changes, and more tightly
defining compatibility (like with this requires/provides concept) is a good
way to do that. Many web APIs already say things like "we will return an
object with at least these fields, and maybe more if we feel like it", which I
think is a good example to follow.

As far as changing package names, there's also an interesting question, not
about how computers deal with those package names, but how we do. After all,
why helloworld and helloworld2? Why not give it another name entirely? Part of
that is because you want to make the connection between these two packages
explicit. But if you change your package dramatically, perhaps that connection
is not doing a service to your users.

One of the biggest complaints about major rewrites is "why does it have to be
the new way? I liked it the old way!" If you genuinely believe the new way is
better, why not let it compete on an even playing field under its own name?
(And, if you know your new package can't compete without backwards
compatibility, maybe don't make a new package.)

