
Why we’re no longer using Core.typed - dankohn1
http://blog.circleci.com/why-were-no-longer-using-core-typed/
======
arohner
I introduced core.typed to Circle, and I was responsible for most of the early
type annotations in the Circle codebase. Because of me, Circle was also the
first large contributor to core.typed's kickstarter campaign.

Other commenters are speculating that this kind of tool needs to be built into
the language, and planned ahead of time. IMO, this is completely wrong.
core.typed fits surprisingly well into the language, and there are no
fundamental reasons in Clojure or core.typed preventing it from working well.

I largely agree with the criticisms raised in the post, but it's worth
pointing out that core.typed is developed by one guy, part-time while working
on his phd. The problems are all related to maturity of the project, and
bandwidth. core.typed's version number is currently 0.3.11, which gives you an
idea of its state.

Ambrose has shown what is possible; I'm confident that with more help,
core.typed can become an amazing tool for finding bugs in production code.

~~~
dopamean
> I'm confident that with more help, core.typed can become an amazing tool for
> finding bugs in production code.

This is true but I wonder how much Clojurists care about what core.typed
offers. If not enough do then many there will never be enough contributors to
really make it work. LightTable is a project that seems to have had something
similar happen.

------
wpietri
Over and over I see reports of iteration speed being critical to real-world
projects, and that's certainly my experience.

But it's something I rarely see featured in tool design. So many of our tools
are still batch oriented: they start from scratch, read everything, process
everything, and then exit. That made sense in an era where RAM and CPU were
expensive. But now that they're cheap, I want tools that I start when I arrive
in the morning and kill when I go home. In between, they should work hard to
make my life easy.

If I'm going to use a type checker, I want it to load the AST once,
continuously monitor for changes, and be optimized for incremental changes.
Ditto compilers, syntax checkers, linters, test runners, debugging tools.
Everything that matters to me in my core feedback loop of writing a bit of
code and seeing that it works.

For 4% of my annual salary, I can get a machine with 128GB of RAM and 8 3.1
GHz cores. Please god somebody build tools that take advantage of that.

~~~
jrochkind1
The problems I've seen with tools that try to load everything once and monitor
for incremental changes -- is it's hard to get RIGHT. These tools tend to be
buggy, or worse tell you it's not them that's buggy, it's you that's "doing it
wrong" somehow, trying to work the way you want.

The beauty of simple tools is it's easier to make simple tools that work
really really well. From loading everything at once, to considering text
editors vs "IDE", which is really part of what you're talking about too. When
we have new languages and platforms all the time these days... it's important
to be able to get high-quality tools that don't take forever to develop.

~~~
wereHamster
The problem is that our editors are at the core still "text" editors. That
forces the tools (IDEs) to continuously synchronize the text with the parsed
AST.

It's trivial to get an editor right if it edits the AST directly. You skip the
whole tokenization/parsing step which is where most complexity is. However,
such an editor will need specific plugins for each language, and these plugins
will be easier to write for languages which have a standardized AST (Haskell
comes to mind). Also, users would have to give up part of the formatting
(whitespace etc), and some people are very religious about that :(

~~~
the_af
The counterpoint to this is that if you're not using text editors then you
lose the universal property that most source code is plain text which can be
edited with your preferred text editor. Plain text seems to be the most tool-
agnostic representation, which for many people is a big plus. Notice that in
this day and age there are still people complaining about "bloated" IDEs...

~~~
wpietri
I think there's a useful distinction between serialization format and working
format (that is, what it looks like in RAM). For a plain text editor, they're
the same. But that's pretty unusual.

There's reason we couldn't have an editor that works with the AST while saving
everything out in plain text for interchange purposes.

~~~
the_af
True. But the property you mention -- that for plain text editors the
serialization and working formats are the same -- is actually a pretty useful
and convenient property! I think this is why it's extremely unlikely there
will ever be widespread adoption of a different format for source code. Niche
tools will remain niche forever, in my opinion.

------
jerf
That... is pretty much exactly the list of problems I've expected out of
optional typing.

I think other dynamic-type language communities should read this and think
really hard. I've said before, I'd rather see a language become the best
$LANGUAGE it can possibly be, for whatever that $LANGUAGE is, than try to bolt
things on that pull it away from its core focus. Dynamically-typed languages
are dynamically typed. Become the best dynamically-typed language you can be,
don't bolt less-than-half-assed static typing on to the side. The sum ends up
being less than the parts.

And if that means the next ten years is the language slowly fading into
history... that is the destiny for all of us anyhow. Let it be with a
fantastic dynamic language that was the best it could be, let it be a language
that people speak wistfully of twenty years later. Don't make it be remembered
as a Frankensteinien monstrosity as it tried to stay current.

~~~
fernandezpablo
I disagree. Typescript is an excellent example of an optional static type
system that works great. It has none of the 3 problems the OP describes: it's
fast, it nicely encompasses the whole language and the Definitely Typed
repository includes a TON of 3rd party library type definitions.

~~~
softawre
How big is your Typescript project? Because when I read the first paragraph of
this I cringed and knew where they were going.

We have a medium sized Typescript app (~100k lines), and it takes 7-10 seconds
to transpile to JS.

Whenever a change is made, that is 7-10 seconds the watch command is running
and I'm not testing my changes in the webapp.

In fact, a slow running grunt watch is why I'm on HN right now in the first
place.

~~~
pjmlp
> Whenever a change is made, that is 7-10 seconds the watch command is running
> and I'm not testing my changes in the webapp.

Compared with C and C++ build times which are measured in minutes or even
hours, 10s is nothing.

~~~
JoeAltmaier
Huh? My 100,000 lines of code takes seconds.

~~~
Matthias247
That also depends massively on the used libraries and the structure of the
code. A 100kloc c file could build superfast. Whereas multiple c++ totalling
about 10kloc that include use use lots of Boost stuff can take ages compared
to that.

------
numlocked
I don't have nearly the breadth of experience in Clojure as the circleci guys,
but I found that Prismatic's Schema [0] works well for me vs. core.typed. I
can selectively annotate functions with their return types, and test runs will
validate that they are returning as expected. It's runtime evaluated, so you
lose a lot of static analysis benefits that type checking gives you, but it
still provides some good code coverage, forces you to think about types a bit,
and (the biggest benefit for me) it serves as documentation for functions that
might return somewhat complex types. I can return to code months later and
have a good idea of what's going on.

I've found the syntax a little arcane, particularly for coercion (which is a
secondary function of Schema) but the documentation benefits have been worth
the effort of learning it.

A nice middle ground between nothing, and full-blown type annotations.

[0] [https://github.com/Prismatic/schema](https://github.com/Prismatic/schema)

------
DougBTX
Interesting to compare with TypeScript, which runs into similar issues.

Before TypeScript 1.1, the intelligence was quite slow, but is now fine even
for large projects. Third party libraries often have type definitions in the
DefinityTyped collection.

That just leaves constructs that work in JavaScript, but which can't be type
checked in TypeScript. This is often double-edged. TypeScript's types are not
amazingly flexible, so it encourages writing less magical code, improving
readability, but it also discourages some patterns that are quite useful, such
as passing references to properties using string key names, which are hard to
type-check.

~~~
aikah
> TypeScript's types are not amazingly flexible, so it encourages writing less
> magical code (...)

you can "cast" anywhere so it's not really an issue :

(<(string)=>string>baz['baz'])('foo')

Otherwise typescript support union types and such ... Even without .d.ts
headers everywhere it is still useful. I prefer it to babel and co as it is
easier to track ES6/7 integration. Never used typed clojure though ,but I
don't know any example of a language that didn't have types before and
retrofitted type safety without any issue and breaking backward compatibility.

------
muhuk
I gave up on core.typed because I've found adding type annotations to most of
the forms is significantly harder compared to a statically typed language. (In
hindsight this is obvious, I know.)

Not having type-checking doesn't necessarily make Clojure any less awesome for
me though. I write unit tests, add (some form of) contracts at least to the
public functions and pay attention to documenting public stuff. And not doing
excessively clever things like having unnecessary mini-DSL's within the code
also help.

------
TheAceOfHearts
I feel like I can somewhat relate, I've been adding flowtype [0] annotations
to my JS code, which is transpiled with babel. However, since I'm using
unsupported ES2015 features and experimental ES2016+ features, it fails to run
over my code (which is completely understandable).

The workaround I've found is to use a plugin [1], which converts the type
annotations into runtime checks. It's not perfect, but I've found it helps
catch some bugs quickly. It's also really helpful for documentation purposes:
you know that this function param takes a number param, if you're not passing
in a number you've made a mistake. Also, having the inline type is nicer than
having it in a jsdoc-style comment above.

I think there's diminishing returns as you add more annotations to your code
and you make them stricter, but having basic sanity checks has been helpful in
catching stupid mistakes quickly.

[0] [http://flowtype.org/](http://flowtype.org/) [1]
[https://github.com/codemix/babel-plugin-
typecheck](https://github.com/codemix/babel-plugin-typecheck)

~~~
cwyers
Why Flow instead of Typescript?

~~~
TheAceOfHearts
Mainly because I was already using babel (which was 6to5 at the time).
Typescript didn't support JSX at that time either. I don't know if TypeScript
has full ES6 support yet? (And async/await!)

Overall babel seems to be moving in a direction which I'd consider really
interesting. With babel 6 you'll be able to write a plugin to add runtime
checks that aren't limited to module boundaries.

------
j_m_b
>In September 2013 we blogged about why we’re supporting Typed Clojure, and
you should too!

I wonder how many other premature "you should too"s have been issued over the
years. At least they had the guts to retract theirs.

~~~
pbiggar
I wrote that post, and it was very much about why you should support Typed
Clojure, not why you should use it. You should support it for the potential,
you should decide whether to use it on practical concerns.

------
lisa_henderson
This bit raises the question, is #1 being caused by #2 and #3?

\--------------------------------------------

The problems that we have hit with using Typed Closure are

1.) slower iteration time when programming,

2.) core.typed not supporting the entire Clojure programming language

3.) using third-party libraries that have not been annotated with core.typed
annotations

\--------------------------------------------

If #1 is caused by #2 and #3 then presumably #1 will get better with time. But
if #1 is caused only by typing itself, then it sounds like, for them, using
types was a bit of premature optimization.

It does raise the question, is it possible to create a type system that does
not slow a project down?

------
Gonzih
But what about then gradual typing in core.typed? Will it make core.typed more
useful? Or is problem not just in gradual typing but more in type oriented
system vs data oriented system?

~~~
Ovid
The issue I have with gradual typing in most languages is that it's added
after the fact and not native. As a result, the integration isn't native and
you start seeing issues like this.

This, incidentally, is one of the reasons I like Perl 6
([http://perl6.org/](http://perl6.org/)). It also has gradual typing, but if a
third-party library doesn't have a type annotation, that's OK: it will simply
fail at run time instead of compile time.

Another reason to like native gradual typing instead of third party gradual
typing is that it can be integrated natively into the system instead of being
used as an afterthought. Perl 6 has native type infererence to allow for
compile time failures:

    
    
        $ perl6 -e 'sub foo(Int $x) { say $x }; foo("bob")'
        ===SORRY!=== Error while compiling -e
        Calling foo(str) will never work with declared signature (Int $x)
        at -e:1
        ------> sub foo(Int $x) { say $x }; ⏏foo("bob")
    

(Yes, that's compile time instead of run time)

What happened above is that the optimizer saw that the call to foo() requires
an Int and checks the argument type to see if the call is allowed.

And here's the untyped version:

    
    
        $ perl6 -e 'sub foo($x) { say $x }; foo("bob"); foo(3)'
        bob
        3
    

Interestingly, the Perl 6 core team could allow for powerful type inference
with this, but one of the core issues has always been that many languages
using type inference have rather opaque error messages because they
essentially dump a "proof" of type failure and many struggle to understand
them. Thus, the the Perl 6 devs are taking more careful approach, though they
know they can do more. For example, this falls back to a runtime failure
instead of a compile time failure:

    
    
        $ perl6 -e 'sub foo(Int $x) { say $x }; my $name = "Bob"; foo($name)'
        Type check failed in binding $x; expected 'Int' but got 'Str'
          in sub foo at -e:1
          in block <unit> at -e:1
    

In other words, because it's not sure of the type of $name at compile time, it
falls back to run time (though in the simplistic example above, that's clearly
not the case). Annotate the $name variable and you get a compile time failure
again:

    
    
        $ perl6 -e 'sub foo(Int $x) { say $x }; my Str $name = "Bob"; foo($name)'
        ===SORRY!=== Error while compiling -e
        Calling foo(Str) will never work with declared signature (Int $x)
        at -e:1
        ------> t $x) { say $x }; my Str $name = "Bob"; ⏏foo($name)
    

With this, third-party libraries which don't declare types work just fine, but
you get type-safe run time failures. If they do declare types, you often get
compile time failures (though it will still fall back to run time, and won't
check at run time if it knows at compile time that it works).

~~~
bjr-
I believe that's where core.typed is going:
[http://frenchy64.github.io/2015/06/19/gradual-
typing.html](http://frenchy64.github.io/2015/06/19/gradual-typing.html)

Typed and untyped code must (and will soon) be able to cooperate in providing
compile-time and run-time type checking in Clojure.

~~~
mattrepl
There are also tentative plans to integrate core.typed with the Clojure
compiler so type annotations can be used during compilation (e.g., type hints
through inference).

There was a post about it, but the best I could find is a recent paper:
[http://frenchy64.github.io/papers/typed-clojure-
draft.pdf](http://frenchy64.github.io/papers/typed-clojure-draft.pdf)

------
Confusion
Does anyone know how the situation is with typed Racket? Similar problems with
speed, annotations being hard to write and tooling being insufficient?

~~~
samth
Typed Racket (due to different integration into the compiler) is much more
incremental -- the speed issues in the blog post would be much reduced. The
typechecker still isn't as fast as we would like, though, although we have
plans to improve it.

I think the tooling integration is better in Typed Racket, although Ambrose
has made some improvements in Typed Clojure as well. As for typed versions of
existing libraries, I think that things are bit better in Typed Racket, but
also Racket has many fewer libraries than Clojure at the moment, so the scale
of the problem is smaller.

------
MBlume
It seems like something like the dependency tracking and selective reloading
in Stuart Sierra's tools.namespace could be used to mitigate their first
problem.

------
agumonkey
The code ahead types strongly remind me of the state of UML model driven
trends a few years back. I wonder if it ended the same way.

