
Show HN: Pipelines – Language for scripting parrallel pipelines with Python - calebwin
https://github.com/calebwin/pipelines
======
tedmiston
This looks pretty interesting.

Just curious if you looked at Apache Airflow as the integration functionality
and pipeline / DAG configuration approaches are similar and it's also done in
Python.

[https://github.com/apache/airflow](https://github.com/apache/airflow)

The Nim language is new to me. What was the experience like implementing a
framework in it?

------
hultner
Looks interesting but I don’t understand if the “x as x” is mandatory? Seems
overly verbose if you don’t want to import it with another name?

~~~
StavrosK
That is unnecessary, I don't understand why the Readme includes it. It does
nothing.

------
nicois
I'm not sure I understand the wins of this over just using the standard
multiprocessing module.

You can trivially define a worker pool or arbitrary size and pipe a stream of
values through your function using multiple cores, avoiding GIL issues.

If you have something which scales to multiple machines just use celery or
redis streams.

~~~
mistrial9
DASK

------
natch
Nice readme! It's so common for projects to forget to include an introductory
blurb, it's great to see one that does not forget.

Small bug in the example code? I think this:

    
    
        return even % 2 == 0
    

Should be this:

    
    
        return number % 2 == 0

------
m45t3r
I didn't understand one thing: is this implemented in Nim and run Python code?
If yes, why don't run Pipelines in Nim too? Wouldn't this be faster?

~~~
iamwil
Most programmers don't write in Nim. More programmers write in Python. I
assume he's targeting the Python crowd.

------
i_phish_cats
wait... `/>` and `|>` are python operators?

~~~
otherme123
No. Those are Nim, or rather tokens that guides the Nim code to do the pipe
thing.

The code splits the line in function names, args and those operators, to then
orchestrate the pipeline flow.

------
Escapado
Should have named it Pypelines. But seriously though I am not sure what the
benefits over multiprocessing are. It looks a little cryptic at first sight.

~~~
PurpleRamen
[https://github.com/gpcimino/pypelines](https://github.com/gpcimino/pypelines)

------
ako
Looks very similar to R, dplyr and pipes.

~~~
rmbeard
More like magritter I thought. just wish everytime someone comes up with
something new you have to install yet another package manager.

