
Parallel – A command-line CPU load balancer written in Rust - 0xmohit
https://github.com/mmstick/parallel
======
RhysU
Further overloading this overloaded utility name screws up packagers and
hinders usage of any of the variants. Rename it please.

~~~
gcr
Agreed. GNU Parallel does something similar and this is confusing.

But this does claim to be a reimplementation of much of GNU Parallel's
functionality. If it's compatible, maybe there isn't too much of an issue.

In the interest of silly UNIXism, I hereby vote for the name "serial" instead.
It's just like "more" and "less", right?

~~~
kibwen
I believe it intends to support a subset of GNU parallel and be command-line
compatible with that subset. And "perpendicular" would be a far better joke
name, IMO. :P

~~~
ole_tange
Hitachi already made your theme song:
[https://www.youtube.com/watch?v=xb_PyKuI7II](https://www.youtube.com/watch?v=xb_PyKuI7II)
:)

------
bmalehorn
The code is more complicated than it needs to be. It spawns N threads, then
each thread forkexecs child processes. These threads communicate through an
atomic int.

There is no need for threads. Just spawn background processes:

    
    
        echo 1 &
        echo 2 &
        wait
        echo 3 &
        wait
        echo 4 &
        ...
    

The key is that the wait() system call will hang until _any_ child process
finishes.

~~~
the_mitsuhiko
I doubt that will make it measurable more efficient. Threads are cheap.

~~~
bmalehorn
My point wasn't optimization. Just that the author was making it more
complicated than was needed.

It's like discovering that "ls" is spawning 5 threads for internal
communication.

~~~
the_mitsuhiko
In a language with good threading support, internal threads are typically
making things easier rather than harder. I tend to spawn lots of threads in
Rust just because I can and it simplifies the code a lot over having to do
some async callback mess.

In particular there is no sane way to async waitpid() on POSIX.

~~~
geofft
pselect / self-pipe / etc. on SIGCHLD, and on receipt of SIGCHLD, loop through
all known child processes with wait(WNOHANG).

... I'm not sure I can disagree with "no sane way".

~~~
the_mitsuhiko
The only thing you can do from a signal handler is flipping a static and only
one signal handler can be set. Good luck making this reusable.

In fact, I challenge you to solve the problem "spawn process; waitpid for 15
seconds; otherwise kill hard" in Rust (or C++ if you feel like) on POSIX once
with threads and once without threads by sticking to what's permitted in the
standard and so that multiple processes can be waited for.

Then also measure CPU impact :)

~~~
gpderetta
> The only thing you can do from a signal handler is flipping a static and
> only one signal handler can be set.

This is false, you can call any async signal safe function. Incidentally write
is one of them.

Another trick is the close-on-exit pipe.

~~~
geofft
"Async-signal-safe" is a C concept (from the POSIX world where C is your
interface to the system, and library calls vs. system calls are behind the
abstraction layer), so it doesn't directly apply to Rust. But the underlying
semantics of signals are simple to describe: you get interrupted at some
instruction pointer and jump into a new function. You can do whatever you want
provided you uphold safety, correctness, liveness, etc.

If you change a variable, it has to be one that isn't prone to being cached in
a register or the stack by the main program. POSIX's sig_atomic_t does this;
in Rust you can use the normal atomic types. They are a tiny bit _too_ careful
if this is thread-local, but an ordinary thread-local variable is permitted to
be cached within the same thread, and signal handlers break that.

If you take a lock, you have to do something reasonable if the lock is already
held, _including by the code you interrupted_. So you probably shouldn't lock
at all. The biggest reason for a POSIX function not to be async-signal-safe is
because it wants to call malloc, which takes out a lock (at least a per-thread
or per-CPU lock) on the heap. If you get signaled during a malloc, and the
signal handler tries to malloc, you deadlock.

But anything that does not risk liveness or correctness problems is fair game.
In particular, basically all system calls are fair game, since they're just
sending a message to the kernel. C's fprintf() will want to buffer in
userspace, which involves an allocation, but write() will at most buffer in
the kernel, and the kernel-side code doesn't have the problem of having flow
control interrupted while you're in a signal handler. Even if you were
previously in a blocking write() when you received a signal, the kernel will
return from its implementation of write before delivering the signal back to
userspace, so there isn't a re-entrant call to the kernel-side write code.
libc's fprintf() doesn't have that luxury.

(And yes, the concept of Rust on POSIX is a bit ill-defined, because POSIX is
a set of C-language APIs, which can be implemented in any valid way in C,
including header macros. Rust threads use pthreads, yes, but inter-thread
communication doesn't involve whatever sig_atomic_t is typedef'd or #defined
to.)

~~~
gpderetta
Note that the lock is a red herring. A single threaded malloc wouldn't be
async signal safe either unless explicitly designed to be so which is hard. In
fact anything that touches mutable state, including thread local state is a
problem.

~~~
geofft
Yeah, good point. The weird part is that it _almost_ introduces a new thread,
in that it introduces a new flow of control (and, if you ask for it, a new
stack), but it doesn't count as a new thread in all the usual ways, like
thread-local variables or PIDs. So any code called from a signal handler must
be "thread-safe", but that thread-safety cannot rely on assigning an
identifier to the thread, because you're using the same identifier as the
parent thread.

~~~
the_mitsuhiko
> Yeah, good point. The weird part is that it almost introduces a new thread,
> in that it introduces a new flow of control (and, if you ask for it, a new
> stack), but it doesn't count as a new thread in all the usual ways, like
> thread-local variables or PIDs.

Worse: most signal handlers that do anything other than setting globals
destroy errno in one way or another and when you go back the code that was
interrupted has a good chance of malfunctioning.

------
mnkmnk
This looks like a perfect real world complete useful application that a
beginner in rust could look at.

~~~
0xmohit
A rust beginner could also look at Jonathan Turner's lessons from solving
first 12 Project Euler problems.

[0] [https://www.jonathanturner.org/2015/10/lessons-from-
first-12...](https://www.jonathanturner.org/2015/10/lessons-from-
first-12-euler-problems-in.html)

~~~
eriknstr
>The owner of www.jonathanturner.org has configured their website improperly.
To protect your information from being stolen, Firefox has not connected to
this website.

I added an exception as I pretty much always do but in case the owner of the
site is around here, they might want to investigate this.

~~~
jntrnr
website owner here: odd, what configuration of firefox gives you that warning.
When I connect with a recent OS X version, it seems to work okay.

~~~
sp332
Firefox on Windows here. Looks like it's getting the certificate for github,
but throwing an error because your domain isn't listed in that certificate.
[https://imgur.com/a/UeXtC](https://imgur.com/a/UeXtC)

~~~
jntrnr
Yeah, it's not set up for https. Do you see the same thing with
[http://www.jonathanturner.org/](http://www.jonathanturner.org/)?

~~~
sp332
Nope, that's fine. Not sure why 0xmohit posted the HTTPS link in the first
place.

------
thealistra
I always tended to autogenerate a makefile with fake goals like:

    
    
        g1:
            job1
    
        g2:
            job2
    
    

and then one goal to depend on them all and I used make -j 4

~~~
kstrauser
parallel is awesome for when you want to want to run a number of jobs unknown
in advance, such as using it as an xargs replacement:

    
    
      find . -type f -name | parallel --jobs 5 process_file
    

That will run "process_file foo" for ever file in a directory with 5 processes
running in parallel.

~~~
coredog64
Not being a jerk, but what does parallel bring to the table that xargs
doesn't?

    
    
      find . -type f -name | xargs -I {} -n 1 -P 5 process_file {}

~~~
perturbation
You can have remote workers with parallel (not sure if you can do that with
xargs or not).

    
    
        parallel --progress --wd '.' -S '4/fourworkers@blehworker.org,8/eightworkers@eightworkers.org,2/:'  "./complicated_command --param1={1} --param2={2} ::: $(some_shellcommand_to_generate_param1_values.sh) ::: $(some_shellcommand_to_generate_param2_values.sh)
    

It's very flexible, become my go-to for speeding up parallelizable work. Next
step up would be Celery or Spark or something.

------
Aissen
Not comparing it with the simpler, better moreutils parallel is really unfair.

~~~
ole_tange
I hope you wont take this the wrong way, but can you elaborate on why you feel
Tollef's parallel (from moreutils) is better than GNU Parallel?

~~~
Aissen
It was addressed elsewhere in the thread. My take is: 1/ it's written in C. 2/
it doesn't suffer from feature creep.

The code is 427 lines of C. GNU parallell is 10k+ lines of perl. Considering
mmstick compares the loading times, it's easy to see where the difference
comes from.

------
amattn
I wrote a go utility that is not designed to be direct replacement, but rather
nicer CLI tool that does something quite similar. My version is quite a bit
more limited, but totally meets my current needs.

[https://github.com/amattn/paral](https://github.com/amattn/paral)

------
zhanxw
Why this can be faster than GNU parallel?

~~~
mcpherrinm
One possible reason is that GNU parallel is a perl script.

~~~
ploxiln
a version in C that I think was first released before GNU parallel is in
"moreutils"
[https://joeyh.name/code/moreutils/](https://joeyh.name/code/moreutils/)

A few years ago, debian made GNU parallel provide the "/usr/bin/parallel"
executable, instead of moreutils. The maintainer of moreutils had some
interesting things to say about that: [https://bugs.debian.org/cgi-
bin/bugreport.cgi?bug=597050#75](https://bugs.debian.org/cgi-
bin/bugreport.cgi?bug=597050#75)

~~~
paulddraper
> It is 5143 lines of code, and the anthethesis of a simple, clean, well-
> designed unix command.

Only 5k? :D

But, yes, the criticisms are valid. I recommend moreutils.

~~~
dagw
He lost me at the point when he complained that GNU parallel "includes the
ability to transfer files between computers". For me at least that is _the_
feature of GNU parallel that actually makes it really useful. Which I guess is
the problem with all these discussions, one persons useless bloat is another
persons nr. 1 killer feature.

~~~
oblio
But in the spirit of Unix, shouldn't parallel exec some file transfer program?

~~~
dagw
The actual Perl code calls out to ssh and rsync (or can be configured to use
something else) when it's time to actually connect and transfer files. It just
does it in a way that is nice and reasonably transparent to the end user.

It felt like his complaint was that that was 'bloat' since Real Men can
achieve almost the same thing by just piping some output through some bash
scripts they just hacked together.

~~~
ole_tange
And it is exactly the hacking part that GNU Parallel tries to help with: A lot
of the helper functions in GNU Parallel could be done by expert users (--nice,
--tmux, --pipepart, env_parallel, --compress, --fifo, --cat, --transfer,
--return, --cleanup, --(n)onall).

But non-expert users will invariably make mistakes (e.g. get quoting wrong,
not getting remote jobs to die if the controlling process is killed, or re-
scheduling jobs that were killed by --timeout), and why not just have small
wrapper scripts built into GNU Parallel that are well-tested, so the non-
expert users can enjoy the same stability as the expert users?

~~~
dagw
Having written my fair share of those hacky wrapper scripts before I
discovered GUN parallel I certainly am very happy that they offer everything I
need in a single easy to use command.

------
kwoff
Like headlines that are questions (answer: "no"; next), I tend to skip over
headlines that say "written in $language". Usually means they've copied
something that already exists, or it doesn't have much else distinctive about
it than being written in that language.

------
sopium
I often use make to run jobs in parallel. It is nice to be able to continue
after an error or breakage, especially when working with large workloads.

~~~
ole_tange
Using Makefiles, however, depends on the results being files. If that is not
the case, make cannot tell how far you got.

GNU Parallel has --joblog and can continue from where it left off or retry all
failed jobs again.

