Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I agree that it is difficult to tackle bigger problems with this model, and everything that deviates from rss in -> manipulation -> rss out is maybe better served with textual programming. Though I did use pipes in a ruby program of mine basically as micro service for everything web related, shortly before it got turned off...

But Yahoo Pipes did have submodules. You could have one of your pipes as a block in another pipe (I'm planning on supporting that at a later stage). And it had userinput that could get assigned as additional input to blocks in a pipe (for that I don't have a mental model yet).



Checkout Rapidminer and Knime for great examples of box'n'arrows applied to machine learning pipelines. I've used Rapidminer a fair bit and they really have a great model including a dynamic number of inputs/outputs from a single box and ability to make boxes that encapsulate pipelines made from other boxes. I was able to do an order of magnitude more experimentation using this visual style than if I had tried to script it in code. I'm a software engineer who loves to program so the fact that I'd pick this tool for that job over programming I think says something.


For a more stable, and more business analyst friendly tool, check out Alteryx. It's not open source, but is powerful, stable, and has SDKs in multiple languages.


It only had submodules in the sense that you could embed a pipe, but IIRC, they were always sources -- you couldn't pipe data into them. It had a notion of the 'output' block but no input with which you could implement such decomposition.




Consider applying for YC's Winter 2026 batch! Applications are open till Nov 10

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: