I'm very puzzled about the Swift for Tensorflow investment. I understand that's primarily because of Chris lattner's personal investment...but given Google's massive mindshare in Kotlin/Dart/JavaScript, this is a very puzzling investment.
Kotlin is successful because Google was able to drive the direction of the language. For example AndroidX Compose is being built with compiler extensions in Kotlin with Jetbrains.
Dart is obviously driven by Google.
Swift is going to be controlled and driven by Apple. From a strategic perspective, I don't understand why Google would make an investment like this when Swift-for-Tensorflow has a fundamental need to drive changes in the language itself.
Now, I understand if Google didn't have an alternative. Are you saying Kotlin is soooooo worse than Swift that you can't fundamentally take a similar direction ? Kotlin is currently one of the world's most popular languages, born out of sheer LOVE from the community (without being pushed by the giants).
Presumably Kotlin wouldn't work for the same reasons they excluded Scala and Java and C#.
(I think that either of C++ or Julia would have probably made better choices, given what they spell out here, but they put a lot more thought into what language to use than most HN commenters seem to give them credit for.)
Swift was chosen because Chris Lattner is leading this project (Chris Lattner designed Swift). This explanation is clearly just a token justification for having chosen Swift.
The charitable way of looking at that is Chris Lattner is leading this project because the project requires certain things Lattner has experience with, and therefore the language he designed is one that meets all their criteria.
Now, it's possible that they were less objective about the language choice than they believe themselves to be, but they certainly have thought about it an awful lot, and they deserve some benefit of the doubt, unless you can demonstrate what of this justification is untrue.
They full on brushed off Julia with hardly a justification. It’s the strongest contender and got hardy a mention. They just dismiss it on the grounds of “oh uuuh there are...aren’t many libraries or something”, nevermind the fact that Julia’s data science and ML ecosystem is far more developed than Swift’s is.
Moving TF to Julia would have killed 2 birds with one stone: community size and outright number of packages would have both been fixed as a community grew up around TF in Julia.
Julia compares very unfavourably to both Python and Swift: Slow startup time (unpredictable JIT compiler lag), terrible error messages and stack traces, duck typing misrepresented as type checking. I've tried using it for what it is meant for (numerical code / ODE solving etc.) multiple times now. It performs admirably in that domain.
But no ten horses would make me use it for anything else. Machine learning needs to be fully integrated into the larger language ecosystem. You can easily imagine developing a large scale software product in Swift, not so much in Julia (even less so than in Python). Julia is probably a Nightmare to develop in once you have more than 10 people working on the project.
The internal structure of Julia's compiler is also a mess compared to the Swift compiler. You could easily imagine building a compile service / large scale build system / tooling for swift, not so much for Julia. Plus they were able to easily extend the Swift intermediate language in the way they wanted to, while there is no comparable abstraction in Julia. The Swift compiler also fits neatly into the C++ based ecosystem that Google has.
First, julia has it's own IR (comparable to swift IR as mentioned by Chris), which is used to great effect in the autodiff packages to allow for compile time zero overhead gradients. The flexibility is actually the other way around, Julia's compiler allows these sort of compile time initiatives to be built in Julia rather than hacked into C++ like swift.
Regarding the duck typing vs type checking bit, that's a matter of tooling. It's much harder to replicate Julia's combination of speed and dynamism (for example to do flexible multistage programming) than to capitalize on Julia's the compile time information to build type checking (which are planned)
The compiler lag is going to be gone soon from two ends: more dynamic (interpreting code that doesn't need to be compiled, this is pretty much there already) and more static: Compiling and caching entire Julia programs. There are already fantastic improvements to this situation in 1.2
That leaves the error messages and stack traces, which ...well what's the last version you have tried? I find them to be ok, but they can and will be improved.
Julia's multimethods are awesome and a clear advantage over swift's methods. A protocol abstraction is similiar to julia's facility for traits, will at some point will be likely baked into the language (but even now allows for compile time resolution of methods just through multi dispatch abstraction , which again, is only one step away from building type check abstractions!).
I don't think static type checking is a matter of tooling,
getting Julia to perform well amounts to either a lot of
trust in the compiler (is static type inference for the Julia type system even
possible?) or manual type annotations as far as
I can tell. And static types are not there in a large program
to make them necesarily faster, but to automatically enforce
contracts. For this to work they have to be mandatory, not
added on as an after thought. I don't want to be able to run
a program only to discover it crashes because a method was
not overloaded with this particular combination of parameter
types.
Swift has clearly an advantage in that domain, as
has C++, C#. In an enterprise setting such as Google no one
actually wants speed and dynamism out of a language (C++
programmers are not allowed to use rtti for example), rather
consistently boring, predictable, statically guaranteed results
with as minimal fuzz as possible (Go, Java, C++) and enough
competent programmers in that language.
I agree that it is neat that you can hack the Julia
compiler in a library to do compile time
automatic differentiation, but my point was that if you
build it into the language like the Swift for Tensorflow
authors are doing this will result in a more conservative,
if inflexible, but potentially more performant solution.
In other words precisely because you need a team of people
to implement and integrate this feature in C++ you will
get a stable "single source of truth" implementation
in the language, not several (Flux, Zygote)
constantly changing libraries.
Well I've tried DiffEqFlux.jl with Julia 1.1, it works
well enough, but the stack traces are ridiculously long
if something goes wrong, a ~200 LOC program takes seconds
to compile and the library and package management story seems
immature compared to rust/swift/python/c++.
>Swift has clearly an advantage in that domain, as has C++, C#. In an enterprise setting such as Google no one actually wants speed and dynamism out of a language (C++ programmers are not allowed to use rtti for example), rather consistently boring, predictable, statically guaranteed results with as minimal fuzz as possible (Go, Java, C++) and enough competent programmers in that language.
That's a pretty weird statement, considering how hard it is to write a complex C++ program error free. I suppose Swift is much better in this regard - But compared to C++, Julia is a bliss and one can easily write huge applications error free, while I had my absolute worst debugging experiences in C++, where it's very easy to create obscure and hard to find bugs.
If you really need to get predictable results and actually know those results - you won't get away with a static type checker anyways, but will have to write & run tests, which then pretty much make it irrelevant whether your language is dynamic or not.
Btw, the argument "If it compiles, I don't need to be afraid about run time errors anymore" is also pretty foreign to me.
I just run my code in small batches while developing it, so at every point I already know that it works. This helps writing code that nicely separates into small chunks of functionality, and you basically automatically write tests for it while at it.
This is especially nice when refactoring a large code base. I can already run many small and usable tests in the middle of a refactor (while the program wouldn't even compile in a static language).
In my experience, this greatly helps to improve the quality of the refactor, and makes it much easier to see problems in your refactoring approach very early on.
In a static language, you'd only get these crucial insights after you're done with fixing all compilation errors - which is usually when you're already done with the whole refactor... and after that, you might still have lots of errors in it that you can only find with real tests ;)
It's possible that it's less effort for a top notch compiler developer to come in and fix Julia's compiler than it is to give Swift the data science ecosystem that Julia is building. I have little experience with it, but it's a question of tradeoffs.
Well from my point of view, the largest advantage that Swift has over Julia is that it has a pretty nice static type system. If you look at what they did with the Differentiable protocol, then this is precisely what you want from a type system perspective without going all in on dependent types. Historically there are very few examples where a dynamically typed language was retrofitted with a static type system successfully (Typescript and Javascript being one example).
Julia's multi methods are really nice but produce horrible errors when they fail at runtime, I don't see an obvious way to fix that.
There is a bigger chance that another contender arises within the C#/F# ecosystem as an evolution of the ML.Net work, then that a company invests significant resources in Julia to make it suitable for large scale machine learning in a non scientific "blue collar" setting.
> Julia's multi methods are really nice but produce horrible errors when they fail at runtime, I don't see an obvious way to fix that.
What? I have had no trouble with the errors produced from multiple dispatch! More often than not they've helped me get straight to the fix, which is more than I can ever say for Python.
> There is a bigger chance that another contender arises within the C#/F# ecosystem as an evolution of the ML.Net work
I would rather claw my eyes out than do significant data work in .Net. C# is our language of choice at work, and it's pretty great for them, but my god is it the most verbose, confusing and convoluted language I've ever had to deal with. F# isn't too bad, but once again, lacks any serious data science community and package support. They're also not terribly fast languages, which is a pretty important factor in ML stuff.
One reason for the choice may be encoded in the date of that post: April 2018. That was before the release of Julia 1.0. Were they making that choice today, I can't help but think the economics might be rather different.
The fact that it didn't even make the list should give you a clue that Kotlin is not a particularly relevant language outside Android apps, where it's poised to become important.
Who knows, maybe in 5 years it will make the list.
To those that are upvoting this and previous Swift + TF announcements: What are you excited about, specifically? Why Swift? Why not Julia? Is it the syntax? Types? Compilation? Performance? Community?
I like Swift and all but our ML/DL/RL/DS tools and libraries are in Python (and occasionally R). Most are missing for Swift without an awkward Python compatibility layer and I don't see a compelling reason to adopt it.
I upvoted it not because because I use Swift (or intend to use it) but because I find it interesting that the concept of differentiable programming is pushed further and further. Having a host language being used to write and compile a second language (which is implemented in a third language) just feels restrictive in many ways.
I do think what is being done with Julia, Cassette, Flux and Zygote more interesting since it's Julia all the way down (while Tensorflow's backend is still C++) and the compiler work is focusing on not being specific to one implementation or technique, but allowing any such language extensions (such as auto-parallelization and other forms of source to source transformations) to be done by any 100% Julia library. So if Tensorflow for Swift (regardless of the actual reasons behind the choice of the language) proves that the technique is a significant upgrade over what currently exists, it could spark interest in the competing approaches, and I think Julia can help pushing the concept even further.
In the perspective of someone who writes Julia, which can also call directly any Python method (and R, Fortran, C and C++), that's a nice stopgap, but you really want a true native ecosystem. Not only there is more mental effort dealing with two languages at the same time (which might lead to people just use Python in the first place), the whole purpose of Swift for Tensorflow is having a language with first class differentiation support, which is pointless when the ecosystem is fragmented in multiple languages.
And there is the risk that the community simply ends up considering that good enough and just make wrappers (since it needs a lot of work to create something nearly as good from scratch). Thankfully that didn't happen with the Julia community, and the key is probably making the creation of the tools much easier so they can catch up to mature but constantly evolving environments.
As far as i know, Swift is the only one embedding the graph flow of the TensorFlow directly in its compiler, so those idioms can translante from IR to machine code with paralelism and a fine-tuned machine code.
As its very common to target this sort of code to GPU´s, and LLVM can target them as output, i think is mostly because you can design and shape the langage right on its high level representation and tune for high performant code in the low level.
The why, i think, its about opportunity.. Chris Lattner going to Google, and people from the language and the compiler side of the fence being open about bake this right into the compiler when necessary.
”To those that are upvoting this and previous Swift + TF announcements […] Why not Julia?”
You seem to imply that people who upvote both Swift/TF articles won’t upvote Julia ones.
For me personally, I have upvoted earlier Swift/TF articles because they were well-written, and integrating differentiation fairly deeply into the compiler seemed novel to me.
I think I also have upvoted some Julia articles in the past, not because they were about Julia, but because they were interesting and well-written.
”tools and libraries are in Python (and occasionally R). Most are missing for Swift without an awkward Python compatibility layer”
An upvote need not mean “that’s immediately useful for me”. It could also mean “I like Swift, and this looks quite an improvement for it, making it more competitive with the leading technologies for machine learning” (you like Swift, so you _could_ upvote articles like these for that reason, too)
As a stopgap while all the data science tooling is being built out for Swift you can use anything from the Python ecosystem (including np and pd) by using the Python interop:
i just tried this out and i'm blown away that it works. even matplotlib. does anyone know how to point it at a different interpreter? a venv for example?
My top four reasons: 1) the eventual MLIR for LLVM will empower new algorithms; 2) Lattner and team have a track record; 3) vast community of swift application developers; 4) fastai.
Is swift a common language for kind of thing tensorflow is used for? Given that swift does not work on windows and Linux support is sub par, is this only because of Chris Lattner?
Swift support for Windows has apparently been making great strides lately; see, for example, <https://forums.swift.org/t/swift-windows/22458>. They're not quite at the point of announcing "official" support yet, but the test suite was "effectively ... fully passing ... (pending patches)" as of April 1 of this year.
Disclaimer: I don't use Windows but read the Swift development forums from time to time.
The way I see it. The technical points have been explained on their GitHub page. One might not agree with it. From an organisation's perspective it makes total sense. You need in house expertise and you need a free hand. Not many languages would have fit both of these things. I mean if it were you or me, wouldn't you pick what you are most comfortable with given it fit the constraints?
Linux support for this can't be sub par since the whole stuff will run on linux like 99.99% of the time.
Glad to see the update. I have experimented with both the macOS and Linux drops before and they were rough. v0.3.1 is focused on providing a productive platform. Anyway I am on travel without a laptop so I need to wait to try it.
The idea of LLVM/Swift ‘turtles all the way down’ might really pay off in the long run. In TensorFlow there is an un-understandable barrier (for me) between the Python layer and the lower level C++ code. Good to see this Swift effort along side Julia + Flux, which offers the same implementation language for the whole system advantage.
I took the recent fastai course that had two lessons about S4TF taught by Chris Lattner and walked away excited.
It’s not ready for prime time yet but the roadmap looks great. Building auto-differentiation into the language and having an easy way for anyone to experiment with core architecture changes (without killing performance) will be a huge win for everyone.
I expect lots of prototyping will be done on S4TF (because it will be easy for developers to keep things on the GPU with XLA and MLIR) and then the things that work will be back-ported to other ecosystems.
If you missed it, Chris gave a talk a little while back, and went into detail regarding the motivation behind the project, what Swift provides, etc. It answers a lot of the questions being asked here:
Kotlin is successful because Google was able to drive the direction of the language. For example AndroidX Compose is being built with compiler extensions in Kotlin with Jetbrains. Dart is obviously driven by Google.
Swift is going to be controlled and driven by Apple. From a strategic perspective, I don't understand why Google would make an investment like this when Swift-for-Tensorflow has a fundamental need to drive changes in the language itself.
Now, I understand if Google didn't have an alternative. Are you saying Kotlin is soooooo worse than Swift that you can't fundamentally take a similar direction ? Kotlin is currently one of the world's most popular languages, born out of sheer LOVE from the community (without being pushed by the giants).