
True Scala complexity  - plinkplonk
http://yz.mit.edu/wp/true-scala-complexity/
======
jashkenas
Fantastic post. The most salient excerpt for me:

    
    
        def filterMap[B,D](f: A => Option[B])(implicit b: CanBuildFrom[?,B,D]): D
        def filterMap[B,D <: GenTraversableOnce[B]](f: A => Option[B])(implicit b: CanBuildFrom[?,B,D]): D
        def filterMap[B,D <% GenTraversableOnce[B]](f: A => Option[B])(implicit b: CanBuildFrom[?,B,D]): D
        def filterMap[B,D[B]](f: A => Option[B])(implicit b: CanBuildFrom[?,B,D[B]]): D[B]
        def filterMap[B,D[B] <: GenTraversableOnce[B]](f: A => Option[B])(implicit b: CanBuildFrom[?,B,D[B]]): D[B]
        def filterMap[B,D[B] <% GenTraversableOnce[B]](f: A => Option[B])(implicit b: CanBuildFrom[?,B,D[B]]): D[B]
        def filterMap[B,D[_]](f: A => Option[B])(implicit b: CanBuildFrom[?,B,D[B]]): D[B]
        def filterMap[B,D[_]](f: A => Option[B])(implicit b: CanBuildFrom[?,B,D[B]], ev: D[B] <:< GenTraversableOnce[B]): D[B]
        def filterMap[B,D[_]](f: A => Option[B])(implicit b: CanBuildFrom[?,B,D[B]], ev: D[B] => GenTraversableOnce[B]): D[B]
    
        ...
    
        > The answer to our original question? It turns out none 
        > of these are correct. In fact, *it is impossible to insert 
        > a new method that behaves like a normal collection method.* 
        > This, despite the heavy advertising of enrich my library.
    

Stuff like this makes think about how, despite all of the problems with using
it in libraries, it's lovely that many dynamic languages can be extended in
your application with little fuss.

    
    
        # Ruby.
    
        class Array
          def filter_map
            ...
          end
        end
    
    
        // JavaScript.
    
        Array.prototype.filterMap = function() {
          ...
        };

~~~
hp
To me, a big advantage of Scala's "enrichment" over monkey-patching in Ruby or
JS is that it isn't global. That is, you have to import the enrichment.
Another code module in the program won't be unexpectedly affected by it.

In practice, I almost never use monkey-patching in dynamic languages because
it's too dangerous. While in Scala there are cases where enrichment won't
work, you can always just write a regular function in those cases, and there
are lots of cases where enrichment _does_ work...

~~~
rue
Adding and/or altering functionality at runtime isn't dangerous.
Monkeypatching may be, so avoid that.

Also, an entirely too-little used idiom (blame Rails programmers):

    
    
        module OverrideSomeMethod
          def some_method
            …
          end
        end
    
        s = SomeClass.new
        s.extend OverrideSomeMethod
        s.some_method

~~~
draegtun
This is an idiom I use regularly (in Perl, Ruby, Io & Javascript) and come
across it often in the Perl world where Moose _roles_ are used.

The only downside of this idiom is the extra runtime cost which maybe an issue
for Rails?

------
dkhenry
After reading a bunch of these articles and really walking away unimpressed
this one is fantastic. It does a great job of showing how cryptic you can make
Scala code. It also shows that the same topic on which half the article is
spent explaining can be boiled down to four lines of less "Scala Like" code.
As a pragmatic user of Scala I don't see why the fact that you _can_ do some
amazing but cryptic things with the language is a detriment. Its like pointers
in C or templates and multiple inheritance in C++. Some people say that the
developer shouldn't be given such capabilities since it can lead to cryptic
and possible dangerous code. I am of the opinion that the additional
flexibility the features provides is nothing but a benefit and that its up to
the developer/organization to make sane limits on what can be used.

~~~
jerf
"As a pragmatic user of Scala I don't see why the fact that you _can_ do some
amazing but cryptic things with the language is a detriment."

Because you hit the cryptic too soon, and you can't really avoid it. And I
generalize this to any such language that has this characteristic, not just
Scala. You can't use C++ for very long without having to know huge swathes of
it if you want to use the libraries created by others, and you want to be able
to debug why they don't work perfectly. (Bearing in mind "working perfectly"
also includes you having a correct mental model of how they work, which is
tricky if you only have a subset of the language in your head!) Same for
Haskell. Compare with Python, where you basically _can_ learn a reduced subset
of the language, yet still use libraries fairly effectively. You need to know
the basics of having objects and calling methods. You don't need to know how
to write your own iterator, any of the double-underscore methods or the
resulting protocols, you _probably_ don't need to know decorators (and even if
you do, probably just how to use them, because they are unlikely to bite you
in odd ways if you just copy-paste instances of them), you don't need to know
metaclasses, etc.

Type systems of almost any kind are very hard to satisfy without
understanding. In some sense, this is one of the very constraints the stronger
ones are trying to enforce.

~~~
soc88
This is not my experience.

I don't give a dime about higher-kinded types, type bounds, type views, type
constraints, CanBuildFrom ... and I'm happily using the language.

~~~
Yeroc
You must be one of those lucky people that only write new code all the time
and never have to maintain code written by someone else. For the rest of us
poor suckers who have to do a mix of writing new code and maintaining existing
code bases written by other sadistic (!) developers then it's a very valid
concern.

------
harryh
A lot of the items the post author describes as complex you can't do at all in
other languages.

"Simple things should be simple, complex things should be possible." - Alan
Kay

I don't think the blog author gives particularly great examples of simple
things that are made complex by the language. He simply gives examples of
things that are inherently complex that scala at least makes possible.

~~~
akeefer
I have to respectfully disagree. First of all, the things he shows aren't
inherently complex: adding an additional function to the existing collections
library is something that's possible in other languages in less confusing ways
(see other examples in this post). The various _concepts_ required to solve
the problem in Scala like implicits and higher kinds might be inherently
complex, but the _problem_ he's trying to solve isn't inherently complex. And
while the Scala solution might not be 100% identical to, say, just adding the
method directly to a class in Ruby (since in Scala it's type-safe, etc. etc.)
that's still missing the point. The point is that if you sit down and say "I
want to add a method to all my collections that works like the existing
methods," in Scala it requires an understanding of a huge number of
complicated intersecting concepts, whereas in other languages it doesn't.

Secondly, excusing complexity by saying "it makes things possible that
wouldn't otherwise be possible" isn't really enough of a justification. The
question is: do those exact things _need_ to be possible? Or is there some way
that gets me 90% of the way there without the complexity, and that's good
enough? More power isn't always better, which was a big part of the point of
the article. Just dismissing it by saying "well, the complexity lets you do
powerful things" doesn't really refute the point of the article, it totally
misses it.

There's a tradeoff to be made. You might make those tradeoffs differently than
I would, which is totally understandable, but we should at least be able to
have a conversation about the fact that there is such a tradeoff without
people dimissing statements like "Scala is complex" out-of-hand.

~~~
harryh
"adding an additional function to the existing collections library is
something that's possible in other languages"

Is it possible in other statically compiled strongly typed languages? I don't
think so?

The advantages/disadvantages of static vs. dynamic languages seem out of scope
for the "is scala too complex?" question.

~~~
akeefer
See my examples below for how to do it in Gosu (via enhancements). In C# it's
pretty much the same (via extension methods). Both are statically typed
languages. Again, not 100% the same as Scala, and each comes with a different
set of tradeoffs than the Scala approach (i.e. they're statically dispatched
in Gosu), but they're certainly both reasonable, and much "simpler", solutions
to the problem of "how do I add a filterMap function to all arrays or
collections that works pretty much how I want it to."

~~~
harryh
OK I read up some on this and neither Gosu enhancement nor C# extended methods
come close to what the post author is trying to attempt (adding methods to Seq
and having them automatically get attached to Array, String, and CharSequence)
_with a single method_.

So it doesn't really seem like they solve the problem "how do I add a
filterMap function to all arrays or collections that works pretty much how I
want it to" any simpler than Scala. All the complexity that the author brings
up in his post is because he was trying to add this functionality with a
single method (and a bunch of implicits).

~~~
Dn_Ab
Up top <http://news.ycombinator.com/item?id=3444688> I gave working code that
comes close. To be fair the reason why it gets closer (works with both F# and
C# lists,arrays, sequences,dictionaries,maps,strings etc) is clerical. From
this thread, I understand Scala and Java arrays are divorced.

But the problem is: put in a string and it gives back a list of characters. So
I use the term it is "topologically correct" heh. To be honest I do not think
there is anything functional about what he is attempting but perhaps the
idioms in Scala are different? My knowledge of scala is mostly horizontal
(from Ocaml,F#, haskell).

~~~
soc88
Returning the most precise result types (both collection _and_ element type)
is a requirement. If this wouldn't be required, the problem would be much
easier.

------
adriaanm
I think it would be great if we could replace implicits by:

1) a simple mechanism for "enrichment" (aka. retro-active extension, virtual
classes)

2) functional type-level computation (as opposed to the mini-prolog engine
that is implicit search + type inference).

Reducing complexity is complicated, unfortunately. We [have been|are] thinking
about both of these alternative features, though.

\---

ps: The following part of the article is inaccurate: "Turns out that Scala
will search up to one level up the type hierarchy for a matching shape for the
implicit."

Check out the "implicits without the import tax" part of
<http://eed3si9n.com/implicit-parameter-precedence-again>. The implicit scope
includes all the superclasses (and their companion objects) of all the parts
of the type of the implicit value that's being resolved.

------
jberryman
Site is unresponsive for me:

[http://webcache.googleusercontent.com/search?q=cache:http://...](http://webcache.googleusercontent.com/search?q=cache:http://yz.mit.edu/wp/true-
scala-complexity/&hl=en&strip=1)

------
dxbydt
Regarding some of the "brainteasers" posted in the article, I am assuming that
most of them are simply bugs that will get ironed out in version 2.9.2 or
thereabouts ?

eg. The following fails:

Set(1,2,3).toIndexedSeq sortBy (-_)

But doing the same in 2 steps, ie. after assignment, works

val xs = Set(1,2,3).toIndexedSeq; xs sortBy (-_)

I have been bitten by this several times now, so I don't unnecessarily chain >
2 functions even if it does compile ( which also solves one other brainteaser
posed in that article ) Have also seen the problem with the add function, when
I wrote a matrix manipulation library.

def add(x: Int, y: Int) = x + y

add(1,_) fails, but add(_,_) works. Even though the error message " missing
parameter type for expanded function" seems reasonable, and providing the
parameter type ie. add(1,_:Int), does compile, the bahavior is hard to explain
to newbies.

His point about hanging your hat on asInstanceOf[...] and the resulting code
being clumsy...ok, guilty as charged, but then you only have so many hours in
a day, and mgmt is paying $$ to solve boring business problems ( compute the
asset quality of five million loans in thirty lines of business over twelve
quarters using scala ) and not mucking around ( add a filterMap to the scala
collection library to get an idiomatic Scala implementation )

I enjoyed the programming snippets in the article very much, but I still don't
get what the point of the rant was. He goes on and on about "lack of
acknowledgement of complexity", but what does that really mean ? Does he want
like a gold star ? Even if everybody acknowledges that scala is complex, what
then ? Eventually some of it will get fixed & the rest will not & life will go
on. Why be a downer at such an early phase of growth of the language ? All
this talk of excess complexity will simply scare off the early adopters. Java
has been around for 15 years and we still don't have decent generics. Scala is
so far ahead in such a short time. Patience, etc.

~~~
eropple
> Even if everybody acknowledges that scala is complex, what then ? Eventually
> some of it will get fixed & the rest will not & life will go on. Why be a
> downer at such an early phase of growth of the language ?

Because things don't get fixed if people won't talk about them. Irrational
defenses (like calling someone who points out legitimate problems a "downer")
are a hindrance toward making Scala better than it is right now.

------
modersky
First, I think a blog post ending with "bring on the flames" should have
comments enabled, otherwise it's not really fair. It comes across as whining
without wanting to listen to advice or a response.

Second, I think the blog post is useful because it shows what's wrong with
some (fortunately very small) part of the Scala ecosystem, and because it
points to a way to fix it.

Here's a quick recap: The author tries to add arbitrary operations to Scala's
Seq abstraction without changing its source code and wants them to work also
on arrays (which are plain old Java arrays), without any extra work. Arrays in
Java support: length, index, and update; that's it. There is as far as I know
no language in existence that allows the precise thing the author wants to
achieve. And there are many variations, such as adding only to Seq or only to
Array that would be really easy in Scala but still impossible in most other
mainstream languages. The author then throws all the machinery he can think of
at the problem to still achieve the same non-result. Well, tough luck. He
might have hit a thing that's simply impossible to do in a generic way, given
the tools we currently have. In fact, I have not checked whether there would
be a way to achieve the result that he wants because that's beside the point.
There are always limits to a generic formulation that will force you at some
point to treat things on a case by case basis.

The problem is that, in trying to achieve his impossible goal, the author
(mis-)uses a lot of the most powerful features of Scala, and concludes that
Scala is simply too complex for helping him achieve the result. I believe he
wrote this blog post to prompt the maintainers of Scala to add even more power
to the language so that he can achieve his goal (the only other motivation I
can think of is that he's trying to actively damage the ecosystem he writes
code in, but that would make no sense to me).

My response will probably not please him. I think that we need to take away
sharp knifes from people who have a tendency to cut themselves. I was always
proud that in Scala you could do in a library where in other languages you had
to change the compiler. Inevitably, some of this is cutting edge stuff. We
have tried many times to clarify the boundaries, for instance when I defined
the Scala levels

<http://www.scala-lang.org/node/8610>

But we can't prevent a developer who prides himself to "stroll right through
level L3" to get hurt.

So, I believe here is what we need do: Truly advanced, and dangerously
powerful, features such as implicit conversions and higher-kinded types will
in the future be enabled only under a special compiler flag. The flag will
come with documentation that if you enable it, you take the responsibility.
Even with the flag disabled, Scala will be a more powerful language than any
of the alternatives I can think of. And enabling the flag to do advanced stuff
is in any case much easier than hacking your own compiler.

I would be interested to read your comments on this proposal.

~~~
raganwald
_First, I think a blog post ending with "bring on the flames" should have
comments enabled, otherwise it's not really fair. It comes across as whining
without wanting to listen to advice or a response._

Speaking as someone who sometimes writes controversial things and never
enables comments on my posts...

I view collections of comments as discussions. Readers often form into
communities, like Hacker News. If there are comments on a blog post, are you
supposed to write your comment here in your community? Or there on the post?
Well, a good question is, are you a member of the author’s community? Probably
not! Communities have all sorts of standards for participation. What are the
author’s standards? Why bother figuring them out for one comment?

Communities also have other tools like seeing a person's history of posts to
get a sense of their viewpoint and biases. How do you do that on an author’s
blog?

Ultimately, authors either have to get rid of comments or buy into a plug-in
system like Disqus. In my own case, I usually just provide a link to HN for
posts that I think are of interest to the folks here. Everyone else can use
twitter, reddit, or their own blogs to continue the discussion.

That doesn’t mean I won’t read all the flames, there are no end of analytics
tools for discovering what people are saying about my writing.

I don’t speak for this author, but considering he has a link to this
discussion right at the top of his post, I’m guessing he’s listening to your
thoughts and welcomes your feedback right here or wherever people congregate
to discuss the news of the day.

~~~
barrkel
Sometimes comments are just comments. If you need a link to figure out whether
to take a person seriously (i.e. their comment has no merit free-standing -
I'm not sure this is true), a link to a blog ought to be sufficient.

I know I frequently click on the links people provide when they make blog
comments, but I don't think I've ever looked at someone's Disqus history.
Disqus cuts across multiple sites in potentially drastically different arenas;
it's too horizontal. But when someone provides a link, they usually mean it to
be relevant.

------
hp
One source of Scala's design is Java interoperability, much as C++ has to live
with its C legacy. This compromise may also be why people are able to use
Scala in practice, though; they need to talk to Java libraries or need the
performance of a language that maps straightforwardly to the JVM.

Scala is a functional/object-oriented hybrid, making it more complex than a
purist language in either mold. It always lets you do things in a Java-like
way - you can use it as "Java with less boilerplate" and touch almost nothing
Scala-specific. Then it adds functional programming alongside.

To me this feels very natural; I like objects for the big-picture structure,
but I like to write algorithms and manipulate data in functional style. If you
need raw performance in some hotspot, write a Java-like while loop with
mutable state; otherwise, write something nice and high-level (and the JVM
will still be much faster than a "scripting language" however you define it,
e.g. [http://blog.j15r.com/2011/12/for-those-unfamiliar-with-it-
bo...](http://blog.j15r.com/2011/12/for-those-unfamiliar-with-it-
box2d-is.html)).

Scala does fix some Java warts that are legitimately complex or confusing in
their own right. For example, primitive and boxed types are less strongly
separated; there's no "static", just nice syntax for singleton objects;
collections are 100x nicer with far less noise; a nice multiple inheritance
design; covariance eliminates a bunch of nasty hacks; better ways to specify
access controls; there's a decent way to factor out exception handling; case
matching is _awesome_; and _so_ much less boilerplate in general.

I'm not sure the static vs. dynamic religious war can ever be resolved, but
static types feel less broken and less verbose in Scala than in Java. Java
makes you lie to the type system, or do something unnatural, much too often.
Scala hasn't cured every such situation, but it's cured a lot of the most
common ones, and greatly reduced the need for manual type annotations.

Scala gives you the conciseness of Ruby, but with static type checking, higher
runtime performance, and interoperability with existing Java code.

Some tradeoffs of static type checking remain, such as compilation times.

People do go on wild goose chases trying to push the language farther than
it's ready to go. I've done it myself. I agree with the article that there are
lots of areas to improve and appreciate the constructive write-up.

But on the other hand, the perfect shouldn't be the enemy of the good. I
certainly would not choose to go back to Java, even as I'd love to keep seeing
Scala get even better.

------
jonshea
If anyone is able, would you please explain to me these two questions from the
quiz? I’m stumped.

Why does `toSeq` compile, but not `toIndexedSeq`?

    
    
        Set(1,2,3).toIndexedSeq sortBy (-_)
        Set(1,2,3).toSeq sortBy (-_)
    

Why does `h` compile, but `f` does not?

    
    
        def add(x: Int, y: Int) = x + y
        val f = add(1,_)
        val h = add(_,_)

~~~
extempore
toIndexedSeq takes a type parameter (I'm not sure why - it appears unncessary)
which means the result of toIndexedSeq is not known when it is attempting to
infer the Ordering. toSeq doesn't take one, it's known to be Seq[Int].

The other looks like some quirk of partial application. It's unlikely there's
any fundamental reason, only an implementation imperfection.

~~~
jonshea
Good to know. Thanks Paul!

------
j_baker
I have a confession to make. I voted this article up simply based on the
title. Even if it turned out to be incredible link bait (which it didn't), I'm
just happy that someone out there is at least recognizing the desire to have a
reasonable discussion of this issue.

------
oacgnol
I wonder how much of an impact it would be if Scala simply had _better
documentation_. I love the terseness of the code and the complex things you
can do, but I'll be damned if I could ever glean ideas from looking at method
definitions rather than code examples.

~~~
hp
I found it essential to read Martin Odersky's Programming in Scala book - if
you haven't then I recommend it. It makes many things clear that can be hard
to pick up using only the free online resources.

There's also a new docs site and it's getting better all the time:
<http://docs.scala-lang.org/>

~~~
gtani
There's a _lot_ of good books on the language, this is 1st edition of
staircase book, which covers 2.7, so probably the majority was written for 2.8

<http://www.artima.com/pins1ed/>

There's also a book manuscript (downloadable from typesafe.com, and
Manning.com has 3 books in the pipeline, tho the 3rd, on FP by Tony Morris
doesn't show there.

------
reinhardt
Just wow. My eyes started glazing over at "views don’t chain. Instead, use
view bounds" and he completely lost me on the next slide with generics. I had
Scala in my mental _TODO/Maybe_ list but after skimming over this post I think
I'll pass.

~~~
hythloday
I'd have the opposite reaction, I think: I'd want to learn a language
_because_ it introduced new concepts (and thus, terminology). It seems to me
that it's a bit pointless to learn a language if you already intimately
understand its theoretical underpinnings.

------
pron
I think that the source of the problem is that Scala is simply not an
opinionated language. You can go functional, you can go OO; you can go
immutable, you can go with locks etc. And because it tries to do everything it
falls on its face, but worse than that - its users don't really know HOW
you're supposed to use it. I always like to contrast Scala with Clojure and
Erlang. Both are very opinionated: they have a philosophy of tackling
problems, not just a set of tools, and that's why they are so elegant and
beautiful (well, at least Clojure is. Erlang is showing its age at times).
Even languages more similar to Scala like Groovy or Kotlin are more
opinionated and more focused (and thus more elegant and simpler) than Scala.
They are trying to be a more modern Java, a blue-collar OO language with some
modern fun. But it seems like Scala is trying to push not only tools but
concepts, only it hasn't decided which concepts are best so it's pushing all
of them at the same time. The result is not only frustration but harmful
education as to the best way go forward.

~~~
soc88
Both "more opinionated" languages are considerably slower than Scala, so
sometimes it is necessary to fall back to "ugly, imperative" code and those
languages make sure you will hate that experience.

The difference imho is that Scala doesn't punish you for trying to be fast
where necessary.

Apart from that I would really like to where Groovy or Kotlin are "more
elegant or simpler". I would have probably looked into the specification but
something like that doesn't even exist for Groovy. From my last journey into
Groovy I learned that this language is substantially underdocumented and buggy
as hell. I prefer not touching it anymore.

~~~
pron
Actually falling back to ugly imperative and super-fast code in Clojure is
extremely easy, and its integration with Java is a pleasure. But I don't want
to pit a specific language against Scala. My point is that it isn't clear (to
anyone possibly), what Scala IS, other than a kitchen-sink language for
programming concepts.

Some things in Scala are easy and some things are hard, but it's not always
clear why. I'd expect the "good" things (whatever is considered good by the
language designers, their expert opinion, that is) to be easy and the "bad"
things hard. But in Scala it seems that whether something is easy or not
depends on how well it fits with the language's algebraic model, and not how
well it fits with recommended practice. Immutable is easy (that must be good,
no?), but mutable is just as easy (so, wait, I'm supposed to... what
exactly?); functional is easy, but so is imperative; implicits are easy but
extension methods, as the blog post shows, can be really hard; type inference
is easy except when it's not. See the problem?

~~~
soc88
> See the problem?

No. Must be because I actually use the language instead of trying to bash it
to advertise another language. :-)

I'm still excited to see another language solving the problem mentioned in the
article. At the moment it really looks like as Scala gets bashed for the
complexity of something not possible in any other language out there.

~~~
pron
Yeah, I'm sorry about that. That was not my intention. But you're right, I
have grown disillusioned with Scala, partly because of stuff like the examples
in the pop-quiz section of the post, but mostly because of the other things I
mentioned.

But I don't agree with your analysis. I won't go into the question of whether
or not the feat mentioned in the post is possible in other languages or not,
but I do know that if I came to a rather experienced Scala programmer and
asked him off-hand if the tasks attempted in the post are easy, I suspect that
the answer would be "yes". Scala is just... surprising like that.

------
OlegYch
Very good post. One point with which i cannot agree is "In fact, _it is
impossible to insert a new method that behaves like a normal collection
method._ "

Please see <http://ideone.com/ePUHG> An excerpt: import MyEnhancements._
println("qwe".quickSort) println(Array(2,0).quickSort)
println(Seq(2,0).quickSort)

~~~
etorreborre
Miles Sabin's solution is also worth a look:
<https://gist.github.com/f83892f65f63b14a1f75>

It uses dependent types which will be included by default in Scala 2.10. I
don't consider this as "simple" but my point of view is that Computer Science
is not "simple" :-). And having a language supporting that level of genericity
is really helpful for type-safety and code reuse.

~~~
milessabin
@OlegYch's solution also represents a good trade off of complexity vs.
convenience: his solution is simpler but doesn't get along so nicely with type
inference; mine gets on just fine with type inference, but is more complex and
depends (no pun intended) on dependent method types. As ever YMMV.

