Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I've been using Benthos for some one-off bulk ETL needs and it's been very easy to write a config to shovel data from here-to-there.

I'll keep an eye on Benthos Studio, though I confess I'd prefer a self-hostable version at some point.



Didn’t look too much into Benthos but what’s the difference to Nifi?


I don't know much about Nifi, but from re-reviewing the documentation, it looks like Nifi is a stateful application with orchestration and multi-user access, etc.

Benthos requires something else handle orchestration and source control and multi-user access and the conditions that trigger an ETL run.

On the plus side, the stateless nature of Benthos means there isn't any big setup process or cluster of containers to login to or anything. You just call benthos from the cli and pass in a config file.

Last week I needed to get 120k rows of data from an Oracle view that would crash on a few bad rows, and shove rows that worked into a SQL Server table.

Writing a Benthos config to tick through every id, fetch the relevant rows one at a time, annotate with batch information, and drop error rows and error messages in a log and healthy rows into SQL Server was, all told, about 120 lines of YAML. Roughly 3 hours to write while consulting the extensive documentation, and then trying performance tweaks until I could saturate my network connection (increase threads to 256) only took another 30 minutes.

Tweaking that YAML to cover a similar second scenario as a new config then was only another 15 minutes.

The week before that, I was trying Benthos at home to stress test MQTT on a Raspberry Pi with synthetic data to see which messages got dropped or mis-ordered.

(Depends on your QoS and in-flight settings, obviously)

I've never had "basic" performance testing be so simple.

If you are stuck on a Windows platform, it even works there, which is nice for one-off runs for dev work, troubleshooting, or break-fix work.


Nice, thanks for sharing this. I’ll have a deeper look into Benthos, sounds interesting and I certainly like the simple/easy part :-)


Really glad to hear the Oracle driver I added to it is proving useful! <3


Let me tell you Benthos is a real breath of fresh air after fighting a few different ETL tools that are more GUI, chrome and administrivia than function.

Thanks for contributing to Benthos!


Thanks for sharing, this is awesome to hear!


Benthos is much simpler, since it's stateless and it's a single static binary written in Go. I don't really know much about NiFi, but if you need to use a messages bus with Benthos, such as Kafka, you can. However, you don't have to.


Benthos looks like a cool project but thanks for turning me on to Nifi. Nifi has a lot more processors out of the box.

Edit: I’m speechless but see below


Which ones would you need? Happy to add more to Benthos. Feel free to open issues here: https://github.com/benthosdev/benthos/issues


Google Sheets, Drive and Slack. I'm not comfortable asking for it because I can't contribute.


OK, that's not a problem. There are a bunch of other channels that you can use to propose new features. Feel free to reach out via https://www.benthos.dev/community

LE: I took note of those 3 and they should be quite straightforward to add. Thanks!




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: