Last time I looked the automatic differentiation was in a compiler branch with no immediate plans to merge in master.
But overall it is promising. I even installed Swift on Linux to play with it, didn't get to ML as I have an AMD GPU and this is a can of worms. Hope it's finished one day.
I would prefer for Julia ml libraries to become mainstream. But, it is what it is.
Also, the ideal for me would be Rust for tensorflow, but the slower compile times (didn't compare with Swift) are an impediment for an iterative workflow such as tweaking models.
A Rust for TensorFlow (and/or a "RustTorch") would be awesome.
I hope all the work being done on improving incremental compilation[a] and developing interactive Rust REPLs like evcxr[b] makes using Rust for AI a practical reality.
tch-rs is really nice. I have built a Rust sequence labeler + dependency parser + lemmatizer on top of it. It supports multi-task learning, finetuning of BERT-like models, model distillation, etc.:
Unfortunately, when I started this guillaume-be's rust-bert crate wasn't around yet (I think), so I also ported some of the models from Huggingface transformers:
There are torch bindings that are used by some users but what I personaly would like is a JAX clone built on top of Rust.
I see a way to do it [0] but... I already have a PhD to finish.
[0]: a macro compiling functions into an intermediate representations that are transformed with const functions (gradient computation) at compile time and jitted into XLA at runtime.
I recently spend some days playing with differentiable programming in Swift on Linux:
* as you said, auto-diff is in a branch or the Google fork of the project
* the only pre-built images are for Ubuntu 18.04
* on Linux, the REPL seems to somewhat broken
* many libraries are assuming OSX or iOS
I don't feel a lot of hope for adoption of Swift on Linux. Apple obviously is not working on that (fair, they have no reason to do so) and the Swift community also has no focus on Linux, since ... they are in the Apple ecosystem.
Meanwhile, the Open Source community is much more interested in Rust than Swift.
For the differentiable programming - this is what got me excited, but after trying, I was a bit underwhelmed. Not that it isn't great technology, its just not figured out yet. I tried to come up with a use case outside of ML and the one I tried wasn't really applicable.
I do feel however that someone will come up with something and that it will have quite some impact.
Apple I believe has a reason to work on Swift for Linux, and that reason is their considerable cloud infrastructure and various backend services.
I’m sure being able to share domain specific Swift code between client apps and backend would be pretty high on their list of wants.
Also, little clues like the way Xcode generates SwiftPM packages in a Linux-ready fashion out of the box shows that they care at least a bit.
Having a lot of interest in the programming languages, my opinion is that Swift is a damn good one. It’s very high level, supports FP deep enough, has a great type system (that is getting better with every release), great OOP support, native performance characteristics and it still lets you get to a really low level when you need it.
I also like how they took great ideas from Haskell, Scala, Smalltalk, C# and others. I code daily in Scala and Swift, previously had done Erlang, Clojure, Common LISP, TypeScript, Ruby, Python, Haskell, OCaml, Java, PHP, C, Smalltalk and some others. In this list, Swift is now almost at the top.
They need to get Higher-Kinded Types and then it’s going to win the world (just kidding, JavaScript gets to win the world, unfortunately) :)
I can't speak for Apple, of course, but some indications of their seriousness are there.
SwiftNIO is a cross-platform asynchronous event-driven network application framework for rapid development of maintainable high performance protocol servers & clients. (https://github.com/apple/swift-nio)
Also, on official https://swift.org/download all releases and snapshots are automatically available for: Ubuntu 16.04, Ubuntu 18.04, Ubuntu 20.04, CentOS 7, CentOS 8, Amazon Linux 2
Official Swift Server work group is a steering team that promotes the use of Swift for developing and deploying server applications: https://swift.org/server/
Official Swift 5.3 release goals have stated a major goal of "expanding the number of platforms where Swift is available and supported, notably adding support for Windows and additional Linux distributions."
(https://swift.org/blog/5-3-release-process/)
Yep. It's difficult to find usages for autodiff, and it looks like a very niche thing to add to a language. But, it's still cool. In a language with extensible syntax(ex: proc macros), this would sit in a library.
And I think you're right, there is a low chance that this gets traction. Especially since Chris Lattner is at SiFive, they probably will have some kind ofAI accelerators, but TF is for training rather then execution. So not sure they'll find a reason to push it.
Jeremy Howard from fast ai might be able to convince people to give it a try. But without people working on it full time, the chances are not great. Especially with a compiler fork that requires constant merges/rebases. But, who knows.
I guess there are lots of uses in optimisation problems, and in sampling algorithms for statistics. I don't know how easy it will be to sell Swift to people who now use Stan or Stata (or R) and don't think of themselves as programmers.
> In a language with extensible syntax(ex: proc macros), this would sit in a library.
And this would allow easier iteration of different designs.
Does autodiff exist in R? Then why not use R since it seems it also has much ML algo support due to its focus on statistics - its used by many statisticians.
Is the idea here that Swift is a more approachable language and thus this is to lower the barrier of entry to TF?
I think the main selling points of Swift to tensorflow are:
* Speed, ML learning pipelines are often bottlenecked by the data load and transformation. TF had new mechanisms (the last I've seen was TF.data). But a language compiled to native is much more flexible in that regard.
* Type safety. Sometimes issues with the models can pop long after they have been running. The hope is that typed API's will show simple errors at compile times
* Auto differentiation built into the language. If I'm not mistaken, this is more powerful than backpropagation in TF, wich also has autodiff. The idea is that this would allow for more custom models without a performance penalty. My knowledge here is limited, since it's been over 2 years since I've implemented back propagation. I've successfully forgotten most of the things I knew about ML/DL.
I don't have any experience with R, but from what I've heard, it was known to be slow. But that might have changed or I may have misunderstood the situation.
You cannot really do autodiff in a slow language. I mean you can but nobody wants to run large machine learning training algorithms on a slow language like Python or R.
You could write autodiff in say C++ but it is a user unfriendly language not well suited for machine learning and scientific computing.
Swift is a nicer high level language you can do autodiff in. But honestly I don’t see the point with Swift either.
Julia already does AutoDiff extremely well and outperforms Swift and pretty much everybody else.
> Apple obviously is not working on that (fair, they have no reason to do so) and the Swift community also has no focus on Linux, since ... they are in the Apple ecosystem.
Neither of these things are true, fwiw. Strong Linux support is an explicit current goal for Swift, though it is admittedly not all the way there yet. See this post regarding Swift 6 and its goals for wider support, for example: https://forums.swift.org/t/on-the-road-to-swift-6/32862
But overall it is promising. I even installed Swift on Linux to play with it, didn't get to ML as I have an AMD GPU and this is a can of worms. Hope it's finished one day.
I would prefer for Julia ml libraries to become mainstream. But, it is what it is.
Also, the ideal for me would be Rust for tensorflow, but the slower compile times (didn't compare with Swift) are an impediment for an iterative workflow such as tweaking models.