
A Look at How Traders and Economists Are Using the Julia Programming Language - ViralBShah
http://waterstechnology.com/waters/feature/2476518/the-infancy-of-julia-an-inside-look-at-how-traders-and-economists-are-using-the-julia-programming-language
======
VodkaHaze
Always fun to point out is that the NY federal reserve switched their full
scale model of the economy to Julia last year[1]. It's also open source, so
data scientists interested in macroeconomics can feel free to play around with
it.

Moreover, quant-econ, an academic entity dedicated to scientific programming
in the social sciences has a few notebooks in python and Julia [2]

[1][https://github.com/FRBNY-DSGE/DSGE.jl](https://github.com/FRBNY-
DSGE/DSGE.jl)

[2] [http://quantecon.org/notebooks.html](http://quantecon.org/notebooks.html)

~~~
amelius
Interesting. What quantities do economic DSGE models measure for example? And
how accurate are these models?

Is there some way to play with these models? Or do you need lots of input
data, which isn't freely available?

~~~
economist42
The foundation DSGE models are nothing like regular statistical or machine
learning models. They are general equilibrium models, a type of mathematical
model the falls under the microeconomics area of economics. DSGE try to
predict how an economical system will evolve over time, given agents
preferences, technologies and institutions, how much will be produced,
consumed, traded, prices and how those behave over time, taking into account
stochastic impacts, like idk oil price fluctuation. They are beautiful
theoretically, since they are micro constructed. See, economists don't like to
use aggregates only to make predictions and infer policy impacts due to
something called Lucas critique (
[https://en.wikipedia.org/wiki/Lucas_critique](https://en.wikipedia.org/wiki/Lucas_critique)
) which basically means that taking an inflation series create an ARIMA(x,y,z)
to predict next weeks CPI is theoretically. invalid.

The estimation however, is usually based on Bayesian hierarchical models,
taking the constraints imposed by the theoretical model, I believe.

Here is a good source for that
[https://www.newyorkfed.org/medialibrary/media/research/staff...](https://www.newyorkfed.org/medialibrary/media/research/staff_reports/sr554.pdf)

DSGE have been criticized for their low predictive power, but it can be said
that in that for mid to long term predictions they are more robust for
monetary and fiscal policy use.

~~~
conjectures
Wasn't aware DSGEs were often formulated in a Bayesian framework. Learn
something every day.

------
s_q_b
Most financial firms need their quants to worry less about the new programming
language hotness, and more about moving entire systems off unbelievably
complicated Excel spreadsheets.

~~~
valarauca1
The reason for this is simple. Excel is

1\. An incredibly powerful tool.

2\. No bar of entry (cost aside, true in corporate environment).

3\. Very gradual learning curve.

4\. The efficiency gain vs time invested is exponential.

Power Excel users, much like their VIM/Emacs counter parts don't use a mouse.
It is just keyboard short cuts [1][2]. This makes them insanely productive.

[1] [https://youtu.be/jFSf5YhYQbw](https://youtu.be/jFSf5YhYQbw)

[2] [https://youtu.be/0nbkaYsR94c](https://youtu.be/0nbkaYsR94c)

~~~
s_q_b
This is digital Stockhold syndrome.

Excel is something managers and executives can understand, so it became the
default language for data analysis. Now technologists trapped using it have to
create ex post facto justifications for why it's really "just misunderstood."

Excel is massively slow, makes it easy for beginners to make massive mistakes,
computes lots of types in very odd manners, performs floating point operations
wrong, and leads to spaghetti code that is a rat's nest of incomprehensible
cross-references.

Worst of all, the lack of code path visibility usually leads to a bus factor
of 1.

Sure, one _can_ learn to operate Excel for data analysis with a decent level
of efficiency, in the same manner one _can_ cross the Pacific in a canoe, but
both are still terrible tools for the job.

~~~
jerf
Everybody's right. Excel is a powerful, flexible tool that also has almost no
guard rails and all but begs people to make profound mistakes and huge messes.
There's too many people who sneer at spreadsheets when they should be using
them, and there's too many people who use them when they shouldn't.

No contradictions.

It would be interesting to see if anyone could get some power Excel users
together and construct a next-generation spreadsheet that encouraged better
practices and worked to prevent huge messes. Spreadsheets are like SQL, where
the initial release was so far ahead of its time that it managed to entrench
itself into the very fabric of computing, even though it's long overdue for a
reimagining.

~~~
claytonjy
Could you say a bit more about what you'd like to see in a "reimagined" SQL?
Are there any serious efforts to replace it?

There was a comment thread around here a week or two ago where someone pointed
out it's kind of insane SQL has stuck around so long, and no one could point
to any worthy potential replacements.

~~~
gregw134
Visual query tools like Tableau don't seem to be going away. I'd love to see
an effective open-source alternative to Tableau that doesn't require scripting
your own D3 website.

~~~
theoh
Butler Lampson makes the point (in a recent set of slides) that relations are
a good base for DSLs: they have enough complexity to model graphs, functions,
sets etc.

That thought has made me wonder if logic programmming has something to offer
in the design of new APIs.

------
jernfrost
I really love Julia and see a bit potential outside of the scientific
computing. I used to do all sorts of automations, smaller throw away programs
etc in Python and Ruby, but I find Julia much more effective to work with.

It also feels like a smaller language in many ways. You don't have to keep as
much in your head, which makes it more effective to keep up with.

The killer features for me, automating a lot of tasks is the awesome
integration with the shell environment. Running and combining Unix commands is
very elegant and getting hold of the output is done really smoothly.

Secondly what I think saves a lot of time is that it really has ALL BATTERIES
included. With Python that feels more theoretical. With Julia you just start
writing the name of a function you think should exist in the REPL and it pops
up. With Python you got to start guessing which package it might be located
in. And that goes for stuff you use all the time like string manipulations,
regular expressions, file reading and writing, process input and output etc.

With Python functions often end up having unnatural names because you can't
have naming conflicts. Julia handles functions with the same name but
different argument types elegantly.

Also there is no schizophrenia wondering if something is a function or method
on an object. Everything is a function so that is easy.

~~~
leethargo
I would also like to use Julia more for "throw away" scripts and general
automation. Unfortunately, startup time (and sometimes compilation) make this
a less fun endeavor than with Python etc.

------
sebg
If you want to learn more about Julia, here are Julia Conf videos / slides
from past years:

* [http://juliacon.org/2014/](http://juliacon.org/2014/)

* [http://juliacon.org/2015/](http://juliacon.org/2015/)

* Julia Con 2015 India - [https://www.youtube.com/playlist?list=PL279M8GbNseuhXmZk9rWM...](https://www.youtube.com/playlist?list=PL279M8GbNseuhXmZk9rWMqY8BGJr_UOxK)

~~~
optimali
2016 talks are also up - [http://juliacon.org](http://juliacon.org)

Overview -
[http://julialang.org/blog/2016/09/juliacon2016](http://julialang.org/blog/2016/09/juliacon2016)

~~~
e12e
Speaking of talks on Julia, I'll again recommend:

[https://www.youtube.com/watch?v=dK3zRXhrFZY](https://www.youtube.com/watch?v=dK3zRXhrFZY)

"Julia: to Lisp or not to Lisp?" from the European Lisp Symposium in May 2016.
I found it was rather illuminating on Julia in particular and lisps and
programming languages in general.

------
agounaris
Julia is a very interesting language but big data processing is not a matter
of the programming language itself, but how your platform and your
architecture is put together.

And nearly always your bottleneck is not the language itself..

~~~
staticassertion
I think that if this were true, the data science libraries in Python wouldn't
all be written in C under the hood, linking to Fortran, etc. Language can have
a huge impact on performance with big data.

Julia is just a small step forward. It still has performance problems where
the JIT can't help much - like the DataFrame structure, which is effectively a
black box and therefor hard to optimize.

~~~
johnmyleswhite
This is not quite accurate as stated: the problem is the combination of a data
structure that is not amenable to type inference with an API that would only
be efficient if type inference had perfect knowledge of the contents of a
DataFrame. There is a lot of work being done to develop a new API that has
none of these problems and provides very high performance, which demonstrates
that Julia's JIT is up to the task so long as you choose an appropriate API.
Julia, like any language, has intrinsic limitations, but this is not a good
example: it's an example of how good API design for Julia differs from good
API design for other systems.

~~~
staticassertion
> This is not quite accurate as stated: the problem is the combination of a
> data structure that is not amenable to type inference with an API that would
> only be efficient if type inference had perfect knowledge of the contents of
> a DataFrame.

Sorry if I implied otherwise - what I meant is that the current DF structure
is hard for the JIT to optimize, not that this is a fundamental limitation of
Julia. Just that JIT's are limited. I shouldn't have said Julia is only a
small step forward, I think. A JIT gets you a long way.

> There is a lot of work being done to develop a new API that has none of
> these problems and provides very high performance, which demonstrates that
> Julia's JIT is up to the task so long as you choose an appropriate API.

Can you link to information on this? I'd love to see information on the design
around this.

~~~
mandelken
>Can you link to information on this?

This blog post is a good starting point on the current shortcomings and design
solutions of dataframes in Julia:
[http://julialang.org/blog/2016/10/StructuredQueries](http://julialang.org/blog/2016/10/StructuredQueries)

~~~
staticassertion
Thanks a lot.

------
Ftuuky
I'm really interested in learning more about data science. Currently attending
moocs to learn R but I don't have much free time and it's a slow progress.
Currently I work as analyst for an environmental consulting firm but I always
wanted to work in something related to quants. I know I'm probably being naive
but I want to change careers. Do you guys recommend that I keep learning R or
should I take a closer look to Julia? Or both? I don't have a programming
background but I do know statistics and Excel. Edit: fast, concise and useful
comments in a matter of minutes, I love HN. Thanks!

~~~
mi100hael
Become fully confident & proficient in a single programming language before
hopping around. If you get to the point where you can build a full application
from scratch and answer other peoples' questions on SO or IRC with one, you'll
be able to easily get up to speed with another should the need arise.

------
blitzd
My beef with Julia when I last tried it out was the lack of flexibility within
the 'data munging' sphere. The assumption appeared to be that all of your data
would come in to as UTF-8, whereas in my org we need strong support for
Latin1/cp1252, and UTF-16LE BOM. Is this something that has changed recently,
or are these traders are economists just working with more sane data?

~~~
jernfrost
Not a language issue per say then. It shouldn't a problem to add support for
this. Julia is even designed from the start to work with Strings of different
types. Something not common in most other languages I've worked with.

Most string functions operate on AbstractString types. You can create whatever
subtype you like implementing alternative string encodings.

~~~
elcritch
Building on this comment, it's actually surprisingly straightforward to add
your own encodings into the core language. Just add the appropriate types and
extend the necessary base methods.

In this case, it appears Julia 0.5 has a transcode method supporting
UTF8/16/32\. [1]

1:
[http://docs.julialang.org/en/release-0.5/stdlib/strings/](http://docs.julialang.org/en/release-0.5/stdlib/strings/)

------
mark_l_watson
I am not a quant programmer, but last summer I did a ten day job for an old
customer in Julia and I had to learn the language as I worked. I thought the
expressiveness and conciseness of the language was good and I liked the way
the package manager used github projects; easy learning curve and REPL based
development.

------
hodder
As a quant, I can't say I've heard of a single quant I know who uses Julia. It
very well could be promising, but firms will be very hesitant to switch from
Matlab, R, Python, C++, and VBA

~~~
sndean
I can't find the talk that Stefan Karpinski gave where he said this, but it
was something like:

"We've been spending a lot of time working with firms to replace K code."

Maybe the small percentage that uses K/Q/KDB+ are moving to something more
readable?

~~~
qewbie
Do you have a link to the talk? It would not make business sense for Stefan to
spend time replacing the small percentage, if it were indeed a small
percentage. Actually, kdb+ is dominant everywhere on wall st, and the company
mentioned in that article are retiring an old, discontinued version of k, not
kdb+; perhaps that is the firm Stefan refers to? The debate about readability
of k code has become boring - if you learn the language, you'll find it
perfectly readable, elegant even.

~~~
sndean
> Do you have a link to the talk?

[https://www.youtube.com/watch?v=QRWZBWwBVR0](https://www.youtube.com/watch?v=QRWZBWwBVR0)
(I can't listen to the audio now so I'm not sure how much I misparaphrased.)
And, you're right, I shouldn't have conflated KDB+ with K.

> if you learn the language, you'll find it perfectly readable, elegant even.

Just comparing K to something like Python.. Though, personally, it only became
(somewhat) readable after playing with APL and J for a year.

~~~
qewbie
thank you for the link. It's at 12:28 in the talk where he mentions the firm
is Conning, who used k for financial simulations for insurance. In this talk,
he doesn't suggest he is working with other firms to replace k/q/kdb+.

------
sgt101
When Julia has an environment as good as Rstudio and a dashboard as good as
Shiny, and a Spark interface that is as good as Sparkler and R-Spark then I'll
be up for it.

Writing good code in R is an effort, it's possible but the language doesn't
make it easy. Julia makes me write good code off the bat and facilitates
writing excellent, elegant code.

~~~
claytonjy
This is how I tend to feel about all non-R things; I've been using R for long
enough to dislike a lot about the language, but the environment and tooling is
too good to give up.

Letting R be the first language I've gotten pretty good at is really spoiling
me.

~~~
pitaj
Yep. In my short time working on an R project, I really enjoyed how everything
was right there. I'd love to see more REPLs include package managers.

------
sdegutis
Why isn't LuaJIT instead? If they both get their speed from JIT, isn't LuaJIT
faster than Julia?

~~~
cschmidt
I like Julia as well, but there is an interesting scientific library for
LuaJIT:

[http://torch.ch/](http://torch.ch/)

Anyone have experience with it?

~~~
ulucs
having worked with it, torch is amazing for neural network stuff (thanks to
the nn rock); however it raises the same problems there are with Python. Lua
is basically a wrapper around C/C++ and whenever you need to do complex stuff,
you find yourself writing C/C++ rather than Lua.

------
wruza
I really liked Julia (and Juno) when tried it last year, but it was too math-
aware as all [actually good] math-originated programming environments. Hope it
will break that circle and get some general-purpose attention in nearest
future. For me, it is very promising even without that HPC or whatever.

------
arcanus
Anyone used Julia for something approaching HPC, e.g. in a distributed
environment instead of MPI?

I've heard plenty of buzz about it in the HPC community but haven't had a
moment to play with it yet.

~~~
KenoFischer
Yes, there's been some scientific workloads that used Julia/MPI to great
effect. I don't think the paper is out of review yet, so probably can't link
to details, but expect news on that front very soon.

------
fredcy
"The importance of data and speed to information can't be understated in the
capital markets ..."

That says, it's less important than you could possibly state.

------
williamstein
"Lovas adds that currently, the de facto standard numerical programming
language when talking to banks and other finance firms is Python/NumPy."

~~~
dx034
I doubt that. It's certainly the trend and a lot of new code is produced in
Python. But most models will run on R/Matlab/C++/C#. I'd guess that most quant
departments have less than 10% of their code in Python.

But Python is the modern way to go, so every shop stresses that they use it
and they expect every new hire to master it.

~~~
thefastlane
... master it instead of C++ or in addition to C++?

~~~
dx034
In addition, although that's often not possible. You need people who can write
C++ to maintain code. But it's hard to find because few people with a finance
background have any experience with C++. So companies often end up developing
in python and then passing it to developers who port the code to c++ for speed
and to fit it in the existing framework. If they could be replaced with Julia,
it would potentially save a lot of money.

------
tekknik
So how come not golang instead of Julia? I don't have anything against Julia
myself but it seems like go with its maturity, easy cross compilation, static
compilation and easy IPC/multithreading it would fit the bill quite well.
Granted the GC is a bit slow but should be improved with the next release.

~~~
whyrt12
Golang is far far inferior in many ways, for scientific programming. For
starters, it doesn't have generics.

------
cutler
Given Julia's pre-1.0 status, general immaturity and good Python interop
wouldn't a saner approach be to use Python as your primary language and Julia
for whatever needs to be fast?

------
jefferson666
just like pandas in python, I want to give a quick impl in go lang for my
backtest strategy system. now it is open sourced in github:
[https://github.com/qingtiandalaoye/GoDataframe](https://github.com/qingtiandalaoye/GoDataframe)

could anyone give a performance test of it? many thanks

------
dekhn
it's almost the perfect language. The breaking issue for me was the choice to
count from 1 (yes, I know there are pro arguments for counting from 1, but
every other language I use doesn't do that).

~~~
rrock
Why limit yourself to zero or one? Julia now lets you use any start point
you'd like. A fairly recent change, so you might have missed it.

[http://docs.julialang.org/en/latest/devdocs/offset-
arrays/](http://docs.julialang.org/en/latest/devdocs/offset-arrays/)

~~~
dekhn
The problem is that I maintain other people's code, and I don't want the
language to add all those extra things. It's a big cognitive load to maintain,
and leads to bugs in production software that is hard to debug.

The julia docs mention that Fortran arrays can be indexed with various
starting numbers. I've programmed and maintained millions of lines of Fortran
over 20 years and I've never seen anybody use anything other than 0 as the
starting index.

~~~
ffriend
Fortran, Mathematica, Matlab, R, Lua - all start indexing from 1 (you can find
an incomplete list in [1]). For users coming from these languages (which is a
large part of the Julia community) the cognitive load would be in the opposite
direction.

[1]:
[https://en.wikipedia.org/wiki/Comparison_of_programming_lang...](https://en.wikipedia.org/wiki/Comparison_of_programming_languages_%28array%29#Array%5Fsystem%5Fcross-
reference%5Flist)

~~~
dekhn
I suspect the number of C++/Python or Python/Fortran (where users use 0-based
indexing) programmers is larger than Fortran +Math +Matlab + R + Lua.

Anyway, I'm not defending my decision as a globally correct one- it only
matter to me.

------
farright
I've never used Julia but it reminds me of Go in that it attempts to replace
Frankenstein systems that mix low level languages and high level
languages/scripts, with a single modern, high level language with sufficient
low level features and performance.

However, in order to do so, they have to give up the generality of either
language, and focus on a particular domain (numerical computation and web
servers respectively). And because of this, the language never develops a big
enough ecosystem. Because of this, I think Python combined with native C
libraries is ultimately the better way forward and will be more successful.

------
edblarney
Someone explain to me why we need a new language here?

If they are trying to express ideas - wouldn't any language do just fine?

They can 'compile' in an 'optimized manner' using something else ...

But in terms of algorithmic expression?

Unless they are writing true 'real time' stuff right for the chip ... why a
new syntax?

~~~
awm
> They can 'compile' in an 'optimized manner' using something else ...

Thats actually the issue Julia attempts to solve as highlighted in the
article. There are many languages that you can use to express ideas, but when
you need the idea to run as fast as possible, you have to _rewrite_ in
C/Fortran/etc and then add bindings to to the language.

Julia lets you write inline C, which I believe means that you can still
express algorithms simply, but with some parts (trivially) optimized. (Not a
huge julia user, but thats my take on it)

~~~
edblarney
I understand that - but wouldn't a really smart compiler be able to do the
same thing?

And if you have to be able to write 'some parts' in C for speed ... then why
not just use C? And have nice libs for whatever you are doing?

Or intelligently distribute it across processes?

I find it hard to believe that the 'financial industry' has performance
requirements that have never been encountered in the entire rest of high tech
before ...

~~~
rifung
> wouldn't a really smart compiler be able to do the same thing?

Well compilers are tied to the language they need to compile so language
choice does still matters. As an extreme example, GHC can be much more
aggressive with inlining because it's pure, whereas most other languages don't
provide the same guarantees.

> And if you have to be able to write 'some parts' in C for speed ... then why
> not just use C? And have nice libs for whatever you are doing?

Yes I think they could do that, but then they'd need people to write and
maintain those libraries. My understanding is they want to find something
which strikes a better balance between productivity and speed. Having to write
your own libraries to do everything is likely considered a productivity loss.

------
rafinha
deja vu

~~~
jefferson666
just like pandas in python, I want to give a quick impl in go lang for my
backtest strategy system.

now it is open sourced in github:
[https://github.com/qingtiandalaoye/GoDataframe](https://github.com/qingtiandalaoye/GoDataframe)

