
Julia parallel task runtime (WIP) - boromi
https://github.com/JuliaLang/julia/pull/22631
======
jrevels
This work has been a monumental collaborative effort and I think the eventual
outcome will be pretty spectacular.

I gave a talk at an MIT Julia meetup a few months ago that the author of this
PR also spoke at; I found his talk to be an insightful overview of this work:
[https://www.youtube.com/watch?v=YdiZa0Y3F3c](https://www.youtube.com/watch?v=YdiZa0Y3F3c)

------
eigenspace
Very excited to see this come to fruition. Julia is really nice to use on
parallel clusters and this will really improve that experience. Especially
since that exception stack PR just merged:
[https://github.com/JuliaLang/julia/pull/28878](https://github.com/JuliaLang/julia/pull/28878)

------
3JPLW
Is there a non-developer overview of what this patch does? Pull requests often
aren't the best thing for Hacker News submissions as they assume a very deep
level of understanding of a project… especially when they chronicle long
development tasks.

~~~
boromi
checkout
[https://www.youtube.com/watch?v=YdiZa0Y3F3c](https://www.youtube.com/watch?v=YdiZa0Y3F3c)
which explains the new features of the task scheduler in a simple fashion.

Basically Julia is employing a new algorithm to schedule tasks, where
traditional methods would get stuck/blocked. Essentially, if you run a job
with parallel tasks and one of the threads finishes, it's extremely difficult
to schedulate the free thread, since the data may still be used by the other
threads that are busy; thus you need to have clever way to determine how to
allocate computational resources. Julia is implementing results of a new paper
on how to optimally do this.

------
roknovosel
I've been researching parallel programming languages for my thesis and Julia
has been really fun to work with. Alongside Chapel, they really make parallel
programs fun to write. Although, in my experience, it's been really hard to
make them run as fast as C+OpenMP. Mostly because I got trapped in a lot of
performance pitfalls as a newcomer to the language. E.g.: parallel TSP branch-
and-bound algorithm where I can't seem to improve the performance -
[https://github.com/novoselrok/parallel-
algorithms/blob/maste...](https://github.com/novoselrok/parallel-
algorithms/blob/master/tsp/julia/main.jl)

~~~
ChrisRackauckas
Judging from the linked code, you're using multiprocessing. Using
multiprocessing is not similar to OpenMP. If you want something like C+OpenMP,
then you'd need to use threading. This parallel task runtime will use
multithreading like OpenMP, and should be similar in speed.

------
fundamental
I really look forward to using improvements in parallel execution within
Julia. I've been experimenting with it for years and in practice parallel
execution has been fairly hacky, though it has been improving. When it works,
it does seem to work quite well.

------
iopiop
It's just me or everytime Julia is in HN a veritable army of shills follows

~~~
boromi
Care to elaborate? I think a lot of HN posts around popular technologies will
clearly have a strong group of followers, not sure why it would be appropriate
to call them `shills`.

So maybe there is or maybe it's just genuine interest in the novel parallel
task runtime. I'd like to think it's the later.

