

My Experience at JuliaCon - ViralBShah
http://www.johnmyleswhite.com/notebook/2014/06/30/my-experience-at-juliacon/

======
srean
Would the video, slides (and corresponding source repos) be uploaded sometime
? Would love to see those.

Reading between the lines, devectorize.jl and the vectorization framework will
make for a heady mix. I am glad that Julia is challenging the traditional
mantra of "vectorize (as in the R/Numpy/Matlab sense) the loops". It is heart
warming to see that the language designers get it that this style has inherent
speed limitations (time is spent filling and copying temporary vectors).

If any Julia core developer is seeing this, just a shout out, (i) to say you
are doing awesome work, and (ii) @ macros available in the standard library
are not documented well. It would be good to have a page that describes what
is available now.

EDIT: Just clarifying my point. Macros as a feature are quite well described
in the docs. What I miss is an index of macros that come built in with the
standard library. They would help both with pedagogy and with putting them to
actual use, for example removing bounds checks. If they are a little more
discoverable than they are now that would be very helpful to a newcomer. For a
new'ish language the documentation is otherwise quite admirable. @ViralBShah I
filed an issue.

~~~
scott_s
I had to look up Devectorize.jl:
[https://github.com/lindahua/Devectorize.jl](https://github.com/lindahua/Devectorize.jl)

The point, I think, is not that vectorizing is slow. It's that you don't want
to be myopic when you do it. Your point about time spent filling and copying
vectors is independent of what Devectorize is trying to do.

Yes, when you vectorize code, you need to have a prologue and an epilogue
which gets the data in the correct format for the SIMD operations. This
prologue and epilogue has a cost, and if it's larger than the gain from the
SIMD operations, then it's not worth vectorizing. (This can happen if, say,
you only have 4 elements to vectorize.) However, we have to do this cost-
analysis _all the time_. Figuring out when it's beneficial to vectorize code
has been around as long as we've vectorized code.

Devectorize.jl is not about that problem. Rather, it's about when you have a
chain of expressions, and the operators in those expressions have implied
loops. The naive thing to do is to myopically execute each one of those loops,
creating and passing around temporary vectors. The Devectorize framework is
given an entire expression, and is able to analyze where the implied loops
are, and figures out how to express that computation in a single loop.

We can tell these two concepts are independent because the ideal situation is
to first use Devectorize on an expression, and then vectorize the result!

For the record, the approach taken by Devectorize.jl is similar to the problem
that expression templates
([http://en.wikipedia.org/wiki/Expression_templates](http://en.wikipedia.org/wiki/Expression_templates))
in C++ try to solve. Through template trickery and operator overloading, we
can put off evaluating an expression, avoiding the temporaries and unnecessary
memory traversal that a naive execution implies. The Boost library uBLAS
(micro Base Linear Algebra Subprograms,
[http://www.boost.org/doc/libs/1_55_0/libs/numeric/ublas/doc/...](http://www.boost.org/doc/libs/1_55_0/libs/numeric/ublas/doc/index.htm))
uses this technique.

~~~
simonster
I think you are getting at this above and what I've written below is somewhat
redundant with srean's followup, but to clarify, _writing_ vectorized code is
mostly orthogonal to using SIMD vector instructions to compute the result. A
sufficiently smart compiler can turn a loop into SIMD instructions, and these
days most compilers (including LLVM, which Julia uses) are pretty good at
that.

The problem that Julia solves here is that many scientific programming
environments _require_ you to write vectorized code, because they are
interpreted and so looping through the elements in a vector incurs massive
interpreter overhead. In some cases, this can make your code much more
difficult to understand than if you wrote an explicit loop, but even when it
does not, it often forces you to give up performance that could be obtained if
you could write the code in a devectorized manner.

For example, consider the problem of calculating the variance of a sample. To
calculate the variance in MATLAB, one might write:

    
    
      mu = mean(x)
      sum((x - mu).^2) / (length(x) - 1)
    

Although I don't know the details of how MATLAB optimizes this code, I do know
that it has roughly equivalent performance characteristics to Julia code that
first computes x - mu, then computes abs2(x - mu), then sums along the result.
With an interpreter, you can't do much better, and this is actually how the
MATLAB var function does it. In Julia, we compute the variance as:

    
    
      mu = mean(x)
      v = 0.
      for i = 1:length(x)
          v += abs2(x[i] - mu)
      end
      v / (length(x) - 1)
    

This is several times faster than MATLAB's included var function, because the
computations performed on each element are cheap relative to the cost of
memory access. If you wrap the loop in the @simd macro and add @inbounds to
eliminate the bounds check when accessing x, the LLVM loop vectorizer will
actually take that loop and translate it into SIMD instructions, making it
even faster. (The @simd macro is necessary to tell LLVM that it's okay to
vectorize, since accumulating into n variables and summing them at the end
gives different [but usually more accurate] results compared to accumulating
into a single variable due to floating point rounding.)

The promise of Devectorize.jl is that you will be able to write the first code
fragment and have it transformed to the second, which would be neat indeed.

~~~
srean
Thanks for providing more detail and context. I wonder how the clash in
terminology came about ! Who would have imagined naming things uniquely would
be so hard (pun intended).

I understand that your example is entirely pedagogic, but just a cautionary
note for the unwary (a) although it is tempting to fold the variance
calculation into a single loop (accumulate the totals of x and x^2), (b)
neither that obvious single loop version nor the code above are good ways to
compute variance if one cares about preserving precision, more so if x has a
wide dynamic range. In large scale problems it does raise its ugly head and
these bugs are difficult to catch because the code is mathematically correct.
Using double mitigates the problem to an extent (but then floats are faster
for SIMD vectorization).

Another side note, I have gradually come to realize and appreciate the unique
position that Fortran holds. It is not often that you have compiler writers
with a background in numerical analysis or vice versa. I BTW have background
in none and sorely miss that.

~~~
simonster
The way I compute the variance above is, to my knowledge, the standard
algorithm implemented by most software packages (apparently including MATLAB).
(The single pass computation you mention is subject to catastrophic
cancellation, and thus pretty terrible unless the mean of your data is very
small relative to the variance.) However, the "real" implementation in Julia
standard library is indeed a bit better: it performs pairwise summation, which
has O(log n) error growth instead of O(n) error growth at negligible
performance cost (see
[http://en.wikipedia.org/wiki/Pairwise_summation](http://en.wikipedia.org/wiki/Pairwise_summation)).

------
coldtea
It's unfortunate the Julia community doesn't have enough resources to maintain
their website.

The available version was 0.2 for ages (I see now they offer a pre-release
0.3), and the blog hasn't been updated for 9 months.

Some simple posts every month or so would signal that this is a live, worked-
on, language (which it is, and interesting stuff happens all the time).

It was difficult to even find out about JuliaCon -- no banner, no blog post,
nothing, just a small link in the main site navigation, that one can easily
not notice.

~~~
ihnorton
To get a full sense of the level of activity, see here:
[https://github.com/JuliaLang/julia/pulse](https://github.com/JuliaLang/julia/pulse)
(and that is only the core - there is also considerable activity in the
packages)

More frequent posts on the main blog would indeed be a good idea. There will
certainly be a few GSoC student posts as the summer progresses.

As of a few days ago there is a also Julia blog aggregator highlighting
community posts:

[http://www.juliabloggers.com/](http://www.juliabloggers.com/)

I believe we've had 0.3-pre nightlies posted since at least January.

As far as JuliaCon, it was announced on the mailing lists and did show up on
HN once I believe. It sold out very quickly, so wasn't really actively
promoted after that.

------
baldfat
Julia looks amazing BUT as an R user:

1) RStudio is just such an amazing tool with rmarkdown and pandoc. I run my
statistics in a document and can output to html, pdf or Word. I can't tell you
how important the ability to print out to Word is in my corporate setting.

2) The libraries and tools are vast (Seems like Linux tons of options but
really one or two are perfect for the individual use)

3) 99% of what I do takes less then 2 seconds with R

Personally if I have a need for large datasets I will look to Julia if I can't
get a solution with R and data.tables.

~~~
johnmyleswhite
Personally, I don't see Julia replacing R for a while. As you note, the R
community has much of what it needs already. When I initially switched from R
to Julia, it was because R wasn't really usable for the kind of work I do.
It's since become clear to me that I work on problems that don't come up for
most R users: problems involving large sparse matrices, including large-scale
optimization and MCMC. If you don't deal with those kinds of things, R is a
good choice.

~~~
peatmoss
It seems like MCMC is a potentially killer app for Julia. Any thoughts as to
how Julia's MCMC facilities compare to, say, something like Stan?

~~~
johnmyleswhite
I think MCMC.jl is pretty immature compared with Stan. But there is work on a
Stan wrapper. Personally, I mostly write MCMC code by hand these days.

~~~
peatmoss
Granted I don't have much experience with Stan, but from what little I've
poked around RStan, the workflow left something to be desired. Setting up a
C++ environment, and then embedding Stan code (which appears to be a C++ DSL
of sorts) as strings inside your R code seems... unpleasant.

Perhaps if MCMC.jl matures, and if it can offer performance competitive with
BUGS, JAGS, Stan, etc., I could see Julia's statistical fortunes rise along
with Bayesian methods generally. I'm working on a PhD in a discipline that is
just now beginning to dip its toes in Bayesian waters. I get the feeling that
the adoption surge is yet to come.

Interesting that you're implementing your own MCMC methods mainly. As part of
my coursework I did a little bit of that, but perpetually felt like I wasn't
smart enough to anything sophisticated--either in terms of exotic sampling
methods, or working with very complex posterior distributions. It may just be
my frequentist R toolbox experience messing with me, but I know I feel more
comfortable with the safety blanket of a framework.

------
ap22213
Anyone know the performance of julia for optimization in contrast to say, ilog
cplex, or gurobi?

~~~
cschmidt
It interfaces to a number of solvers, both commercial and open source.
According to this page [1] it currently supports:

COIN Cbc/Clp, GNU GLPK, Gurobi, Ipopt, Mosek, NLopt

So you can use it to generate models that are solved in Gurobi if you want. Or
you can use the available open source solvers.

[1] [http://www.juliaopt.org/](http://www.juliaopt.org/)

