
Why Swift for TensorFlow? - nnd
https://github.com/tensorflow/swift/blob/master/docs/WhySwiftForTensorFlow.md
======
3jckd
It's interesting how it's going to play out. On one hand side, Swift is a
pleasant language to work with (despite its infancy). But on the other, having
a Tensorflow API doesn't suddenly give it a bunch of libraries for statistics,
comp. vision, modeling, visualisation, etc. that Python/R/Julia _cough_ MATLAB
_cough_ have.

Nowadays, it's difficult enough to convince people to drop e.g. MATLAB for R
or Python for Julia (let's assume that there's some merit to it), despite them
having excellent counterparts for almost everything. Swift's success in this
domain depends solely on the adoption by developers/researchers/engineers.
Unless they're just going to mostly use it internally (as Google is known to).

Which brings me to the last point - why on Earth would they pick Swift (apart
from Christ Lattner being involved) when Julia was on the table? It ticks all
their boxes and has more mature ecosystem for all things "data". Provided
rationale is hardly convincing.

~~~
marmaduke
Julia doesn't tick the "compile to .o/.h" box. As far as I can tell, the use
case for AOT Julia is avoiding package compilation overhead, not delivery of
standalone code objects.

_edit_ seems JuliaC does support this sort of thing:

[https://juliacomputing.com/blog/2016/02/09/static-
julia.html](https://juliacomputing.com/blog/2016/02/09/static-julia.html)

~~~
StefanKarpinski
Julia does already support this kind of thing. Moreover with a minuscule
fraction of the money that’s being poured into making Swift usable for data
science and machine learning, truly top notch support for generating
standalone binaries from Julia could readily be developed. Which is kind of
frustrating but what can you do?

------
elpakal
I also wonder how much of this coincidentally lines up with Chris Lattner
landing at Google. As Chris will admit, and as was left out of this analysis,
Swift has also been given the humble goal of achieving world domination. All
joking aside I'm very thrilled about this and have enjoyed tremendously
watching the Swift language mature since its launch due in large part to the
open source community and the Swift team's admirable commitment. Onward!

~~~
DannyBee
What do you mean?

This was not in process before Chris came, it was a project he suggested and
started pushing on?

What is there to be coincidental?

~~~
insulanus
I think they mean the exact opposite of "coincidental", in the sense of
"hmm... is this a coincidence" (no, it isn't)

~~~
DannyBee
I get that, but i guess i don't understand what there is to be guessing at at
all.

I think it's been pretty straightforward that Chris joined Google and started
this project. You don't have to wonder if that's a coincidence, that's what
he's happy to say happened.

------
tanilama
Fail to see the point of this project.

Swift for Tensorflow might work if the scope is to create a client side model
definition loader natively for various TF models.

Nobody use Swift seriously for server side training, there is no point in
doing so except to add swift to the list of language that claim to do deep
learning but in reality nobody will consider them.

------
kodablah
Strange to see a requirement for choosing one language over another is
supposed ease of adoption and then they choose the one not easily adopted
across every platform. That easy to write syntax takes precedence over general
easy to write/run is unfortunate.

------
igotsideas
Well written explanation! I really enjoy Swift but it's not as accessible as
some of the other languages mentioned. I have a 2011 Macbook Pro and wanted to
use the latest and greatest new Swift features. Unfortunately, my machine is
too old to upgrade to Mojave which means I can't download the latest version
of xcode, which means no new version of Swift. I'm not mad at Apple in the
least bit. I just wish I could use Swift 4 on my machine.

~~~
yoz-y
If you don't mind stranding from the 100% stable roads, you can install Mojave
on your macbook using this patcher:
[http://dosdude1.com/mojave/](http://dosdude1.com/mojave/)

Personally I am running Mojave on a late 2009 macbook pro and it still works
amazingly well. Transition from Mojave and especially the new XCode are also
way faster than previous iterations. There are caveats though, as the
processor in my computer is too old, I had to hack homebrew to compile
everything from source.

(Also, using the patcher does not hinder my ability to push updates to the App
Store or use iMessage, if that is a concern)

~~~
igotsideas
I'm gonna try this, thank you!

------
byt143
Julia would have been a much better and more cost effective choice in my
opinion.

It's a superior platform to on which develop this sort of thing, and further
along at that. Also easier to use.

~~~
siproprio
Julia doesn't have a debugger... They specifically claimed this is a very
important thing.

In my experience, Julia has also inscrutable scoping rules, a slow REPL, and
it's only fast if you don't count the "startup time" of having to precompile
everything.

~~~
byt143
Re: Debugger, fair enough but it's in the works

Re: Scoping rules, these are being evaled.

Re; startup time, already better in 1.1, and will soon be marginalized from
two ends: Better static compilation and better tiered compilation.

------
m0zg
Is there a "standard" way of running Swift on Ubuntu LTS nowadays? A while
back I looked into it, and ran into some hokey and unsatisfying solutions. I
used Swift on iOS, and I like it a lot, but if they care about adoption,
someone needs to reduce friction of getting up and running to approximately
zero. A snap package a-la Go or per-user script based installation a-la Rust
would be quite OK, as long as it's just one, easy to discover command.

~~~
rudedogg
It looks pretty straightforward, see the Linux section of
[https://swift.org/download/#using-
downloads](https://swift.org/download/#using-downloads)

~~~
m0zg
Yeah, going through two screenfuls of text every time I want to upgrade is not
"straightforward".

~~~
rudedogg
Sorry, I just didn't want to give people the impression that it's more
difficult than other languages. To upgrade you'd just have to remove the
original install directory and untar the new release.

I'm not familiar with Snap, but I did find
[https://snapcraft.io/swift](https://snapcraft.io/swift)

Also, upgrading will be less common than Rust since almost everyone uses the
latest release/toolchain. There's not really a reason to use the daily builds
unless you're contributing to the Swift project.

~~~
m0zg
For comparison, for Go, it's "sudo snap refresh go". For Rust it's "rustup
update stable". I mean, how hard would it be to properly package this stuff,
and why should tens of thousands of users deal with all this manual
downloading and unpacking? Assuming, of course, that Swift folks don't
deliberately want to make the language unpolular, like Haskell.

~~~
Redoubts
I’m kinda annoyed there’s no PPA for this, but I suspect they won’t bother
shipping this in the standard repo until there’s a stable ABI

~~~
m0zg
And that's fine: ship an official snap or use the Rust solution. Not doing
this very directly impacts adoption. Most people won't even try to set it up.

------
gthippo
Swift is also now supported in Colab (Google-hosted Jupyter notebooks) and
there's a nice tutorial on some of the features of Swift for Tensorflow at
[https://colab.research.google.com/github/tensorflow/swift-
tu...](https://colab.research.google.com/github/tensorflow/swift-
tutorials/blob/master/iris/swift_tensorflow_tutorial.ipynb)

------
sjwright
I have no idea what TensorFlow is (other than the basics) but I enjoyed
reading that entire document because it did such a wonderful job of explaining
a complex and potentially contentious decision. It’s fascinating to see Swift
feature so strongly in a pragmatic analysis that doesn’t explicitly favour
Apple platform interop.

~~~
skwb
I am a bit ignorant on the topic, but is swift available for Windows/Ubuntu?
Most of the deep learning scientists I know and work with use either of the
two setups. I know there technically exists CUDA GPU support for Apple, but I
have frankly never even attempted to mess with it.

~~~
rudedogg
Ubuntu is supported (see
[https://swift.org/download/](https://swift.org/download/)), but Windows is
pretty early I think.

See
[https://github.com/apple/swift/blob/master/docs/Windows.md](https://github.com/apple/swift/blob/master/docs/Windows.md)
and [https://forums.swift.org/t/windows-
nightlies/19174](https://forums.swift.org/t/windows-nightlies/19174) for more
info.

~~~
pjmlp
Ubuntu is "supported". The compiler might be available, but there are hardly
many libraries available that would compile outside Apple platforms.

~~~
skohan
That's no longer true. The Foundation framework is basically complete on
Linux, and the vast majority 3rd party libraries which are not iOS specific
will work on Ubuntu. Even many of Apple's own libraries (i.e. SwiftNIO, a low-
level, high performance networking library for things like implementing web-
servers) are cross-platform.

~~~
pjmlp
Last time I checked, the "the vast majority 3rd party libraries which are not
iOS specific" was actually quite tiny.

~~~
skohan
You'd be surprised. There are a few reasonably well developed server-side
frameworks, some of which are already used in production various places.

Also a lot of the libraries which are _mostly_ used in iOS don't have any
dependancies on the iOS platform: for instance promise or event emitter
implementations etc.

IBM is actually supporting a number of open-source swift projects as well:
[https://github.com/IBM-Swift](https://github.com/IBM-Swift).

Between that and painless interop with C/C++, Swift does not feel under-
supported on Linux in the least.

------
tylerwhipple
I am surprised Dart is not mentioned at all (maybe implied under the OOP
languages?). While Flutter and Tensorflow are very different usecases, I am
surprised there is nothing in the document on why Dart specifically would be a
good choice. I believe if they used Dart for Tensorflow as well, the community
would be able to get behind the idea that will not be an abandoned language.

------
mamcx
I vaguely know tensor flow as the most(?) popular lib of his kind, but I
wonder how is the history of swift on non-apple platforms and its impact of
the actual users.

Is TensorFlow "huge" in linux, windows, android? Because I also evaluate swift
for my use case
([https://www.reddit.com/r/swift/comments/8zb9y1/state_of_swif...](https://www.reddit.com/r/swift/comments/8zb9y1/state_of_swift_for_androidlinuxwindows_on_2018/))
and decide instead on use rust mainly because the lack of solid support on
non-apple platforms. However, after use rust for months now I still consider
swift a better contender between performance/ergonomics than rust (rust is
damm hard sometimes, and suddenly you could hit a trouble with above-average
complications. I don't see how put this burden in a library to be used for
more "regular" folks could work)

~~~
skohan
The story of Swift on Linux is now quite good.

Windows is less far along, but recently a contributor got nightly builds
started on Azure, and it appears there is serious work on this front.

In any case, it's already possible to run Swift for Tesorflow on Windows using
WSL and Docker.

~~~
mamcx
Ok, that is for swift...

But is not tensor flow popular on windows? Because then build on top of swift
will mean:

\- Put swift on a fast track to be decent on windows, linux, android(?)

\- Ignore the windows users and let them battle a bad dependency?

------
solidsnack9000
Setting aside for a moment the appropriateness of Swift for TensorFlow, this
is a very impressive example of using an embedded DSL to work with a component
that is a full programming system in its own right.

On the one hand, we do want full access to the programming model exposed by
the component -- its control structures, abstractions, everything else. One
the other hand, these are mostly duplicated by our host programming language:
it's going to have variable bindings, operators, iteration, conditionals and
everything else. Doing an embedding like this is a way to expose most of the
component's facilities without introducing a ton of "new syntax" in the form
of combinators or having programs where a lot of the critical code is escaped
in strings.

This same problem shows up in programming interfaces to RDBMSes. LINQ is a
good example of the same embedding technique.

------
victor106
You might agree/disagree with their decision but this is one of the most
honest and comprehensive evaluation for using a language I've read.

------
skwb
Forgive me for my ignorance, but does swift have any good plotting and
interactive "notebook" ability? Specifically the ability to plot images such
as matplotlib.

I ask this because the number 1 reason my deep learning research group chose
python was because of the extensive and interactive scientific plotting
ability that's built into python jupyter notebooks. While our volume of
analysis isn't on the scale of say a google/fb (primarily biomedical image
analysis), the ability to easily visually debug the results is much more
important for developing robust models.

~~~
dynamicwebpaige
Yes! Swift is supported in Google Colab, and as a Jupyter kernel:
[https://github.com/google/swift-jupyter](https://github.com/google/swift-
jupyter).

~~~
skwb
What is the plotting experience like though? As I previously mentioned,
plotting is one of the main reasons our group uses python.

Another reason now that I think about it, is the number of scientific
libraries that I can just "pip install" without much thought (such as
scipy/opencv).

~~~
marcrasi
You can call out to matplotlib (or any other python libraries installed on
your system), using the python interop feature
([https://github.com/tensorflow/swift/blob/master/docs/PythonI...](https://github.com/tensorflow/swift/blob/master/docs/PythonInteroperability.md))!

[https://github.com/google/swift-jupyter#rich-
output](https://github.com/google/swift-jupyter#rich-output) has an example
with screenshots.

~~~
Redoubts
Oof

------
moocowtruck
why? because i made the language, thats why...no real good reason

~~~
FridgeSeal
And here's 3 pages of very vague, half-justifications as to why we didn't
choose anything else to head off any complaints.

------
lovasoa
Reading the document really gives a feeling the author is not being honest on
why they chose Swift.

The lack of windows support is addressed in just two lines. Julia being an
already established language in the domain of data science does not seem to be
especially important to them.

I think the most honest part of the document is:

> because we were more familiar with its [Swift's] internal implementation
> details

------
sometimesijust
[https://github.com/malmaud/TensorFlow.jl/blob/master/docs/sr...](https://github.com/malmaud/TensorFlow.jl/blob/master/docs/src/why_julia.md)

------
mikkelam
As as python machine learning practitioner and previous iOS engineer I have
for while come to miss using swift and type safety for that matter. I really
like the language and wish great success for the TF team with swift.

Side note, does anyone know the effort required to get various python based
libraries running on swift? i.e. numpy, scipy, pandas and so on?

~~~
physicsyogi
The Swift for Tensorflow team has added some python interop to Swift. So
you’ll be able to, for instance, do an “import numpy” in your Swift code.

------
adamnemecek
I can imagine swift really taking off in this space. It’s going to be a battle
between Julia and Swift for who does the best automatic differentiation.

~~~
pjmlp
Google would need to make Swift a first class citzen on Windows, currently
Julia is winning.

~~~
skohan
That's one of the outcomes I am hoping for in this. I would love for Swift to
be a first-class language.

I believe the Swift for TensorFlow team is currently hiring for this.

------
midgetjones
I wonder what this means for iOS apps themselves

~~~
skohan
At the moment not much. Swift for TensorFlow is a fork of the language, with
language-level support for some features which are useful for data science,
for instance automatic differentiation and dynamically-callable objects.

Some of those features are making their way into the main branch, but at the
moment you could not import the TensorFlow library into an iOS project and use
it. Swift for TensorFlow needs to be built using a separate toolchain.

------
masha_sb
I'll wait for the PyTorch version.

------
ramoz
Is there an official release roadmap?

~~~
skohan
Last I heard the goal for "initial adoption" is set for Spring 2019.

------
xiaodai
Haven't we seen this before?

~~~
skohan
It's been in development for about a year, and I have seen several posts about
it here. It's still not quite feature-complete or ready for real use.

------
hooloovoo_zoo
[https://xkcd.com/927/](https://xkcd.com/927/)

~~~
adamnemecek
This can be used as an argument against just about anything.

~~~
shuoli84
noop, some solution will dominate. Which will make 14 => 1 or 2.

------
aaaaaaaaaaab
Why not Rust?

Edit: I wonder if Swift could be replaced with Rust for iOS development?

~~~
criddell
From the article:

We believe that Rust supports all the ingredients necessary to implement the
techniques in this paper: it has a strong static side, and its traits system
supports zero-cost abstractions which can be provably eliminated by the
compiler. It has a great pointer aliasing model, a suitable mid-level IR, a
vibrant and engaging community, and a great open language evolution process.

A concern with using Rust is that a strong goal of this project is to appeal
to the entire TensorFlow community, which is currently pervasively Python
based. We love Rust, but it has a steep learning curve that may exclude data
scientists and other non-expert programmers who frequently use TensorFlow. The
ownership model is really great, but mostly irrelevant to the problems faced
by today’s machine learning code implemented in Python.

~~~
coder543
As I pointed out in two lengthy comments on day one[1][2], that reasoning is
nonsense. If Chris wants to use the language he created in this new endeavor
for machine learning simply _because he made it_ , that's totally fine and
completely his prerogative, but he should just say so, rather than trying (and
failing) to convince people that other languages aren't better suited for this
task.

From my point of view, a weak justification is worse than no justification in
cases like this.

Rust is much better suited to this task than Swift from a technical point of
view. The _far_ superior platform support for Windows and Linux is ample
reasoning to say Rust is better suited for this task, since very few data
scientists will be training models on macOS. However, that's only one of
several areas where Swift has shortcomings for a project like this. Swift is
great for iOS and macOS development, of course, since it was designed for
_that_. I don't think Swift is a bad language by any means, and with enough
effort, it _can_ be reshaped to be good for Tensorflow... the GitHub document
just provides zero useful justification for the work required to make it good
for Tensorflow.

EDIT: to some of the replies talking about Rust's learning curve, that mostly
applies when you start trying to design efficient, interlinked data structures
involving ownership. For most applications of machine learning, this simply
wouldn't be a problem. The library would provide the data structures, you just
have to use them. Rust can provide simple interfaces to complicated things.[3]
The compiler's error messages are usually incredibly helpful.

The learning curve of Rust should not be relevant here, compared to Swift,
which is also full of idiosyncrasies. Swift and Rust _both_ have a large
learning curve for someone coming from Python. This is because they're
statically typed languages that are just _different_ from a scripting
language. For an application like this, I would say those learning curves are
roughly equal at the language level, but as I pointed out in my comments,
Swift has an _enormous_ learning curve of requiring many data scientists to
either install and learn Linux, or throw out their current computer, buy a
Mac, and learn macOS.

My point here is _not_ that Rust is _the most_ suitable language for
Tensorflow (although it _could_ be), but rather I'm making the point that Rust
is more suitable than Swift for a project like this, and therefore this
document is just annoying. It would be better for them to delete this document
and just say "we're using Swift because our team has a lot of experience with
it and because the creator of Swift is leading this project, so we would lack
enthusiasm and momentum if we were using something else, even if it were more
suitable."

Julia would be really interesting to see explored further, since it would
appeal much better to many existing data scientists who would be transitioning
from Python. The times that I've played with Julia, I was amazed at how slow
the JIT is for even tiny scripts. LLVM is powerful stuff, but it is painfully
slow at _everything_. It would be nice if Julia offered an alternative backend
for rapid development.

[1]:
[https://github.com/tensorflow/swift/issues/3#issuecomment-38...](https://github.com/tensorflow/swift/issues/3#issuecomment-384823808)

[2]:
[https://github.com/tensorflow/swift/issues/3#issuecomment-38...](https://github.com/tensorflow/swift/issues/3#issuecomment-384828003)

[3]: [http://kiss3d.org/](http://kiss3d.org/)

~~~
phillipcarter
I personally find Rust to have quite a learning curve (which I guess is also
an opinion shared by others). The language is great though.

I do agree with your criticism of the document here, though. It feels very
much like Swift happens to check many boxes, but the lack of Windows support
is baffling. It's simply table stakes to be able to run, fully supported, on
Windows, macOS, and major Linux distributions. That should be the very first
thing anyone considers.

But beyond that, I think even with Rust's macro system it could be difficult
to make it work for Tensorflow in a way that feels appropriate for Rust
programmers _and_ for TensorFlow. This was explored in F# for Tensorflow
research[0] and a completely different approach[1] was taken because making a
type system suitable for tensorflow got too unweildy.

[0]:
[https://github.com/fsprojects/TensorFlow.FSharp](https://github.com/fsprojects/TensorFlow.FSharp)

[1]: [https://github.com/fsprojects/TensorFlow.FSharp#live-
checkin...](https://github.com/fsprojects/TensorFlow.FSharp#live-checking-
tooling-for-ai-models)

~~~
surrealize
> But beyond that, I think even with Rust's macro system it could be difficult
> to make it work for Tensorflow in a way that feels appropriate for Rust
> programmers _and_ for TensorFlow.

If you're talking about matrix shape compatibility (matching up rows from one
with columns from another) I'm hopeful about const generics here:
[https://github.com/rust-
lang/rfcs/blob/master/text/2000-cons...](https://github.com/rust-
lang/rfcs/blob/master/text/2000-const-generics.md)

------
5stospace
I still don't understand why they would choose Swift over C#?

They complain about C#/Java having "highly dynamic constructs" but correct me
if I'm wrong but isn't swift also a GC/OOP like Java and C#?

I don't think Swift has any inherent objective advantages over c#.

I think it would have been a better decision to go with C# over Swift as
Microsoft has a clear roadmap with the language and it is already supported on
linux/mac/windows.

~~~
FridgeSeal
I would rather claw my eyes out than write any ML stuff in C#.

It's a fine enterprise language, but good lord writing data science and
machine learning stuff in it would be an right pain. It's also not super high
performance, and when you're doing a lot of maths heavy operations, high
performance is absolutely crucial. I had great difficulty establishing whether
SIMD/vectorisation was even supported, and then even more difficulty getting
it to work.

Julia would have been a far, far superior choice than Swift.

~~~
pjmlp
SIMD is supported already for quite some time on RyuJIT. Quite easy to find
out when searching the MSDN .NET Blog.

Its performance is good enough for doing medical digital imagining as
presented by Siemens at FOSDEM 2019.

It is a matter to properly use the features that the language gives us.

~~~
FridgeSeal
Yeah I found the blog posts, but then had the problem of “what compiler am I
using now?” Was it the Roslyn one or RyuJIT? Does RyuJIT support .NetCore or
is it in Standard or Framework or one of other seemingly limitless versions of
.Net that exists for some reason.

Apparently I could use a library called Vectors, buried deep inside some
numerical library, but then the runtime wouldn’t recognise the libraries
existence despite being a dependency and installed (and linked and every other
thing you have to do to get .Net to do anything). After I fixed that issue it
wouldn’t let me construct any arrays or anything.

Suffice to say, on top of C#/F# being painful to deal with at the best of
times, attempting to do anything numerical was an absolute shit fight. I’m
sure if you’ve got a whole team, you can make anything work, but for me it was
not at all worth the effort.

When you consider I can get fully guaranteed (not just hoping the compiler
chooses to optimise it right) in Julia practically for free along with nicer
syntax, 100% less namespacing hell, equal or greater performance, and far more
data science and numerical packages and it’s hard to see what the draws of C#
would be.

~~~
pjmlp
It appears to me that the issue was not being confortable with the .NET eco-
system.

Roslyn and RyuJIT aren't the same thing. Roslyn is the new compiler
infrastructure for MSIL generation, where the original C++ compiler got
replaced by bootstraped VB.NET and C# compilers.

RyuJIT is the new JIT compiler introduced in .NET 4.6, replacing the former
JIT64.

I don't disagree that Julia is better suited for data science given the eco-
system, as proven by my other posts in this thread, just that the performace
is also there when one wants it.

