Hacker News new | comments | show | ask | jobs | submit login

Not all AWS services are created equal... Some are rock solid and others (cough Data Pipeline cough Kinesis Firehose cough) must have been written by interns.

I don't use data pipeline anymore, but 2 years ago when I was using it in my previous company there were a few moments that drove me nuts. I vaguely remember one day AWS upgraded data pipeline to a newer version and broke hundreds of our pipelines that write data to Redshift. We contacted AWS and they rolled that update back...

What issues did you have with kinesis firehose? I just deployed a couple of those and would like to know what to watch out for.

Lots of downtime. Probably 3-4 serious incidents within a 6 month period. We also had a high number of transient errors that AWS Support considered 'expected'.

One issue we've hit is that logrotate with copytruncate enabled breaks firehose, but afaik it's mostly been good.

Using copytruncate breaks a lot of software, not just Firehose. I generally discourage its use in favor of addressing whatever root cause is making you want to use it.

In my experience, any 'improve logging' ticket goes straight to the back of the dev's backlog. Followed the next week by complaints about the logging system not being all that good... :)

Why the hate for interns :'(

Regardless of how you look at it, writing software is a hard problem and takes experience to do reasonably well. There's nothing wrong with interns themselves, rather the practice that many companies seem to follow where unsupervised or poorly supervised intern code goes right into big products.

What issues did you have with AWS Data Pipelines?

Pipelines randomly failing with inscrutable error messages. High error rates.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact