Twitter seems like the poster child for Scala. I've played with Scala a little and it definitely has a lot of things going for it yet I wonder:
- Is Scala used exclusively for systems programming in Twitter or are there other languages in the mix?
- The majority of other big software players seem to be not using it for various reasons. Is that a concern?
- How is the learning curve for new hires? How long before they become productive? How long before they write great code in Scala (as opposed to still figuring things out but maybe being productive)?
- Isn't the JVM too big of a performance compromise for large scale software where every bit of performance matters?
- Is it easier to hire people for Scala positions? Harder? The same?
- How do engineers in Twitter with a lot of experience in other platforms and languages feel about Scala after having used it for a while? E.g. vs. C++, Go, Java, C# just to throw a few names into the hat there. Do people tend to just write C++ in Scala or Java in Scala or whatever or do they adopt to new paradigms with ease?
I know we always say that a good engineer can pick up any new language and to some extent that's true but I think expertise in a specific language builds up over time. Maybe an analogy would be a musical instrument. It seems some of the other languages have larger pools of talent, more mature tools, etc. I guess it's hard to be a new language in this world...
May I ask what you think Scala has going for it? All of your questions undermine rather than point out the strengths of the language and the platform on which it runs.
> The majority of other big software players seem to be not using it for various reasons. Is that a concern?
Twitter, LinkedIn, Sony, FourSquare, Tumblr, Amazon, UBS, NASA, The Guardian, etc., is that a concern? On the flipside it's not likely that Facebook uses F# as its primary language, and that Apple does not use Haskell; in neither case is this a cause for concern ;-)
> How is the learning curve for new hires? How long before they become productive? How long before they write great code in Scala (as opposed to still figuring things out but maybe being productive)?
So, your asking 2 questions in one: how long before new hires become productive in Scala, and how long before they develop mastery of the language. The former, very quickly, write Java without semicolons, and the latter, depends on the abilities of the new employee, could be in as little as 3 months to learn the ropes, corner cases, etc., but for average programmers I'd say at the very least 6 months, and more likely a year or more.
The biggest hindrance in learning Scala, IMO, is not the language itself, but the build system, SBT. God help you, there are some hard yards to wade through at first, especially if you have trouble getting the build *.scala files recognized by the IDE (i.e. blind coding plain text files with no idea what <+= is vs. <<=, or % vs. %%, or that these types of symbolic method pointers are even available to begin with).
> Isn't the JVM too big of a performance compromise for large scale software where every bit of performance matters?
What? The JVM is a performance beast, perhaps if you drop down to straight C or go with C++ you'll get better performance, but you'll do so at a price (i.e. reduced maintainability and flexability).
> Is it easier to hire people for Scala positions? Harder? The same?
In terms of what I think is going for it: There's a nice IDE from JetBrains and Eclipse support is good. The REPL/worksheets. It does perform well on the JVM and running on the JVM means it's portable and you can also leverage a lot of the Java eco-system.
It seems that where performance really matters a lot of companies do choose to "drop down" to C or C++. I think there's still a gap vs. the JVM.
I guess I still need to be "sold" on it hence all the questions. It does seem to work for Twitter as far a large application and scale but I was hoping to get a little inside information on that. I thought what differentiated Twitter is how comitted they are to the language and the scale it's used so it seems there's a lot to learn from their story beyond what's in the presentation.
In a large system like Twitter you're not going to micro-optimise every line of code. You want a good performance baseline and the ability of quickly write correct code. For many Scala and the JVM is a better tradeoff than C or C++ in this regard. See Martin Thompson's blog if you want to know more about low-level performance optimisation on the JVM.
You're also overlooking the benefit of a modern type system. If you haven't used one before it's difficult to appreciate what they bring. Twitter's Summingbird project is a great example of how a few powerful abstractions, enabled by Scala's type system, can simplify a very complex problem.
What I like about Scala and the JVM is they have a large expressive width (http://noelwelsh.com/programming/2013/07/10/expressive-width...). I can write very terse, expressive code when productivity is the main focus, but also reach down to grub around with low-level code if I need to squeeze out performance. Very few platforms offer this.
Isn't the JVM too big of a performance compromise for large scale software where every bit of performance matters?
Actually no - having a good concurrent garbage collector actually helps a lot.
Consider any message passing system. Messages come into the system, you create an immutable object, and send a reference to that object to other subsystems. Eventually, when each subsystem has processed that object, it is garbage collected.
In C++ you need to manually handle tracking the object and deleting it when it's no longer needed. This dramatically hinders the sort of systems you can reliably build.
As for low level performance, the JVM is actually very fast - i.e., nearly as fast as C. The main places where it becomes an issue is that you can't always control the cache locality of objects.
class Foo {
val bar: Bar
val y: Int
}
In C/C++ you can demand that foo.bar be stored in the same page. On the JVM you can't.
> Is Scala used exclusively for systems programming in Twitter or are there other languages in the mix?
Java and C++ were also used.
> The majority of other big software players seem to be not using it for various reasons. Is that a concern?
Not really.
> - How is the learning curve for new hires? How long before they become productive? How long before they write great code in Scala (as opposed to still figuring things out but maybe being productive)?
Took me about four months. It would be easier these days because the related tools (mostly IDE and build) suck less.
> - Isn't the JVM too big of a performance compromise for large scale software where every bit of performance matters?
It's mostly fine. It's just the very highest throughput systems (load balancers where you expect to saturate the pipe and not pause for 5ms) where you wish you had C.
>- Is it easier to hire people for Scala positions? Harder? The same?
You mostly train into it.
>- How do engineers in Twitter with a lot of experience in other platforms and languages feel about Scala after having used it for a while?
I've heard many complaints about Scala being too much of a "kitchen sink" language; too many paradigms and too many features from each paradigm. It still seems like the best language that uses the JVM, though.
There's also Kotlin, which is largely a better Java. And Java 8 is an enormous improvement too, since it supports lambdas, default methods, streams, etc.
- Is Scala used exclusively for systems programming in Twitter or are there other languages in the mix?
- The majority of other big software players seem to be not using it for various reasons. Is that a concern?
- How is the learning curve for new hires? How long before they become productive? How long before they write great code in Scala (as opposed to still figuring things out but maybe being productive)?
- Isn't the JVM too big of a performance compromise for large scale software where every bit of performance matters?
- Is it easier to hire people for Scala positions? Harder? The same?
- How do engineers in Twitter with a lot of experience in other platforms and languages feel about Scala after having used it for a while? E.g. vs. C++, Go, Java, C# just to throw a few names into the hat there. Do people tend to just write C++ in Scala or Java in Scala or whatever or do they adopt to new paradigms with ease?
I know we always say that a good engineer can pick up any new language and to some extent that's true but I think expertise in a specific language builds up over time. Maybe an analogy would be a musical instrument. It seems some of the other languages have larger pools of talent, more mature tools, etc. I guess it's hard to be a new language in this world...