
Embracing Swift for Deep Learning - zeristor
https://www.fast.ai/2019/03/06/fastai-swift/
======
zeristor
If I'm correct the key issue is the need for a deeper level of control, and
that Python as as scripting language gluing things together doesn't afford
that.

They talked about using Swift for differentiable programming, is there
something unique in Swift that enables that? From what research I've done it
seems the techniques of Deep Learning are much more adaptable than the
standard sklearn fit/predict template.

I was a bit surprised since fast.ai seemed to be big on PyTorch, it seems a
paradigm switch to TensorFlow, but that's what research is all about.

In addition to the TensorFlow / Pytorch dichtomy there seems to be Julia's
Zygote, still in early stages. Deep Learning is like The Thing, you look away
from it for a few minutes and it's sprouted legs out of its head and walked
off into a different room:
[https://en.wikipedia.org/wiki/Differentiable_programming](https://en.wikipedia.org/wiki/Differentiable_programming)

"Some day this Singularity's goin' end..."

~~~
solidasparagus
PyTorch and Tensorflow are not that different, particularly with TF eager.

------
cs702
Question for Jeremy:

See links below. Did you look at or consider Julia?

\--

[https://julialang.org/blog/2018/12/ml-language-
compiler](https://julialang.org/blog/2018/12/ml-language-compiler)

[https://arxiv.org/abs/1810.07951](https://arxiv.org/abs/1810.07951)

~~~
sytelus
Julia is GC based language. If you are ok with then there is whole world of
other languages with similar disadvantages. For high performance numerical
computing most GC based languages won’t cut it and that fact is typically
swept under the rug by creating C++ implementation wrapped by your GC based
language bindings.

~~~
pjmlp
Just like Swift is one.

~~~
ajconway
One can argue that automatic reference counting is a form of garbage
collection. It is, however, deterministic and that’s why some engineers tend
to prefer it.

~~~
pjmlp
It is not deterministic in the presence of deep nested data structures like
acyclic graphs, where the amount of time running cascading deletes canny be
fully calculated. Likewise the amount of stack space cannot be fully
determined. Finally, sharing pointers across threads introduces non-
deterministic cache locks and delays.

Herb Sutter has a quite good CppCon talk about it.

Many engineers prefer it due to cargo cult and anti-tracing GC bias, despite
years of CS papers proving the contrary since the early 80's.

~~~
0815test
It is deterministic wrt. the _object lifetime_ which is exactly what you need
if your "delete" is actually controlling access to some shared resource that
must be freed promptly when it goes out of use. The drawbacks you mentioned
are largely-unavoidable consequences of that requirement, and they mostly
matter when the allocation graph is large and complex. It's why _obligate_ RC
as found in Swift is likely a bad idea, and even inferior to obligate tracing
GC - but other languages can do better than that, e.g. introducing arenas that
can be used to allocate or free objects as a group, in a single operation.

~~~
pjmlp
Plenty of tracing GC also offer the same control over arenas and deterministic
destruction, one just needs to actually use them, e.g. D, Modula-3,
Mesa/Cedar, Active Oberon, Eiffel, System C#, C# 7.3/8...

------
amimetic
There is a proper explanation of why Swift makes sense for TensorFlow at
[https://github.com/tensorflow/swift/blob/master/docs/WhySwif...](https://github.com/tensorflow/swift/blob/master/docs/WhySwiftForTensorFlow.md)

~~~
lordnaikon
Which is in part written by the author and creator of Swift. Keep this in mind
while reading this. Not to sound judgmental but I would be curious while Bill
Gates explains to me why windows makes sense for X.

~~~
yodon
Given that he explains his thinking and reasoning in that post, one can either
accept or challenge his logic and discuss it accordingly. There is no need for
unprovoked attacks on his credibility. Doing so just suggests you don't feel
qualified to evaluate the arguments he makes, in which case it's probably
better to ask questions than attack.

~~~
shard
It does not seem like an attack to me, rather informing readers of possible
blindspots or conscious/unconscious bias in the author's point of view. I
think it's only human nature to cheerlead something that one has put a
significant amount of effort into making. I, as someone who does not know
Swift and who its author is, would be more inclined to seek out alternate
points of view after knowing that the author of this article is also the
author of Swift.

Just as one would take any marketing/promotional material for why a company's
products are better than competitors with a grain of salt, and would want such
material to be clearly marked as marketing material, I think bringing to light
the author's relationship to the Swift helps, and does not hinder, further
discussion.

~~~
yodon
If presented in the context of a substantive comment about the assertions, I'd
agree with you. Without any such substantive comments it's simply assuming the
worst about others. The HN guidelines [0] are actually pretty darn good
guidance for encouraging intelligent discussion of complex topics, and by my
read advocate for more thoughtful approaches to commenting on multiple fronts
here, including not posting shallow dismissals or assumptions about
astroturfing and shillage. Again, if there is a concrete objection to the
logic the author took time to detail in their post, by all means that's a
great topic for discussion, and that's where we would all actually learn
something positive instead of just insinuating something negative.

[0]
[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

~~~
sytelus
To be fair author should have disclosed this fact as disclaimer in the
article.

------
echelon
We should _NOT_ do this. Swift is not cross-platform! As nice a language as it
is, Swift support for Linux and Windows seriously lags behind Mac.

If we choose to elect Swift as an ML language, we're handing keys to a certain
platform vendor. I don't want to be tied down like that.

Keep using Swift for app development--it's great at that. But keep it far away
from research and development.

~~~
pharaohgeek
Swift on Linux is great! I use it for a ton of projects, both standalone as
well as for microservices development. Aside from cryptographic operations
(provided by CommonCrypto on the Mac) I haven't really encountered any
scenarios where it can't get the job done. And, for those, I could
theoretically make use of OpenSSL (if I felt like going through that
exercise).

------
mark_l_watson
When this article came out in March, I tried Swift and TensorFlow on both
macOS and Linux - no problems but it seemed like early days, something to
check in on occasionally.

EDIT: there is a new drop for macOS from April 18 which I am now downloading.

I really like modeling with TensorFlow but I have a like-sometimes/hate
relationship with Python. If Swift support gets stable and well supported then
that would float my boat.

~~~
ahurmazda
Likewise, I tried _swift_ back when it was first announced. I had trouble
finding tools in standard library to handle simple FILE operations. There are
many resources on using PythonInterops to get the job done. But kinda defeats
the purpose. Maybe these are early days.

That said, I remain optimistic given Jeremy's involvement —
he/Rachael/fastai's been nothing but a source of good for ML in general.

~~~
saagarjha
You have access to all of libc and Foundation; was this not enough for you?

~~~
ahurmazda
I am familiar with libc (I will have to look up Foundation). My opinions were
based on cursory browsing at best.

To be clear, coming from mostly python/c++, I was looking for something like
`import pathlib` or `#include <filesystem>`. Lot of modeling work I do
involves boring things like moving files around, plotting them etc.

Again, its likely that I did not look around enough.

~~~
saagarjha
Yeah, you are probably looking for Foundation.

------
pjmlp
At least Python and Julia embrace Windows.

------
m0zg
I just hope Fast.ai adopts some kind of style guide for Swift at least. Python
version of fast.ai is an unreadable mess, and it doesn't need to be.

~~~
fredguth
It has a style. It is ok if you don't like it. But it is not ok saying it
hasn't.

By the way, Jeremy Howard spends several minutes on one of his videos
explaining the style and why despite seeming strange it is the right choice
for fast.ai. I just can't remember which video, but I would guess it is in dl
2 2019 first video.

------
rahulrrixe
Swift large projects takes huge time to compile whether as interpreter wins
here. Though there are model building layer on compiler level which Swift
offers in Tensorflow which is unique but apart from that I don’t think it is
going to hit nail on Python community.

------
bertomartin
Sounds really interesting, but it seems like this is all due to the graph
extraction algorithm needing to use static analysis. Is this such an important
requirement that it requires switching languages? I'm not too familiar with
how PyTorch works under the hood, but I suspect it doesn't builds a graph
(correct me if I'm wrong) and they seem to get good perf. I'm all for new
tech, but just wanna get a sense of how impactful this change would be.

------
maxaf
> But Python is not designed to be fast, and it is not designed to be safe.

When decisions made by an algorithm can result in real monetary costs, one
must always err on the side of caution. Only an irresponsible person would
choose Python or other unsafe, dynamically typed language when implementing a
crucial piece of business logic.

~~~
ToBeBannedSoon
Python 3.6+ does have type annotations and multiple static analysis tools for
type checking. Only an ignorant or biased person would yell at Python for
flaws it doesn't have. For crucial business logic, one also writes unit tests.

~~~
petilon
Unit tests are not an adequate replacement for compile-time checks performed
by a compiler. The former is lax and expensive, the latter is thorough and
inexpensive.

~~~
soVeryTired
Language flamewars are silly.

Sometimes python is appropriate for production code, weak type system or no.
Sometimes it isn't. As a project gets larger, more complex, and more
interconnected with other things, static type systems become more useful.

End of story, surely? Didn't we all know that already?

