
Ask HN: Why should one ever use AWS Kinesis when Kafka is out there? - amirathi
I mean apart from the fact that Kinesis is a hosted service and Kafka would most likely require you setting up a cluster. Are there any other technical advantages in terms of features, performance, handling volume at scale, stream ordering, authentication etc.?
======
imauld
>> I mean apart from the fact that Kinesis is a hosted service and Kafka would
most likely require you setting up a cluster.

AFAIK that's the only we use Kinesis over something like Kafka or Spark (much
to the chagrin of of our data team). We are planning on moving to Kafka I
believe but in the meantime while the Data folks sort all that out we are
using Kinesis.

------
barracoda
Another major upside to using Kinesis is how seamlessly it integrates with AWS
based consumers such as S3/DynamoDB etc. On the technical side of things, for
Kinesis "each shard can support up to 5 transactions per second for reads, up
to a maximum total data read rate of 2 MB per second and up to 1,000 records
per second for writes, up to a maximum total data write rate of 1 MB per
second (including partition keys)" (from AWS documentation), i'm not sure if a
hard limit exists for Kafka. W.r.t stream ordering kafka preserves ordering of
messages within the same shard i guess which Kinesis also preserves. Again
authentication is made easier with the rest of AWS provisioning (IAM) so
that's easier to build into Kinesis based application.

I think the bottom line is if you've other AWS components and are willing to
be less hands-on about tuning with ease of setup and scalability, Kinesis is
your best option. Else go with Kafka

------
anilgulecha
Hosted vs unhosted really.

I mean you can ask, why use EC2 instead of your own machine at the local data-
center?

Managed services typically already have taken design considerations for
failover, DR, and scaling, so you don't have to.

~~~
htn
Well, you can get hosted Kafka as a service as well. My company Aiven offers
one such service in AWS/Azure/Google Cloud Platform, DigitalOcean and UpCloud
([https://aiven.io/kafka](https://aiven.io/kafka)). CloudKarafka is an another
vendor with managed offering and Heroku too just launched Kafka service as
well.

I think Kinesis is an excellent service, but if you have an existing stack or
preference on OpenSource solutions, it's good to have choice.

~~~
anilgulecha
Hosted Kafka by a third party is not equivalent to (potential) Hosted Kafka by
AWS.

There's a lot of signalling/data at the back when AWS implements it, which you
will never have access to.

------
eb0la
Not a kinesis expert here, but think the need to reprocess stuff could be the
key.

With kafka you can reprocess data just creating a different consumer group.
You pay for instances.

I guess in kinesis you need to move data from kinsesis to s3 and back to
kinsesis to reprocess it, and it wil surely have a cost.

