

Amazon Kinesis Available For All Customers - seaghost
http://aws.amazon.com/kinesis/

======
6cxs2hd6
I wish Amazon hadn't given up even _trying_ to do RESTful APIs. Nowadays they
just shrug and bang it out RPC style (payload is JSON instead of XML, but the
song remains the same).

e.g. How do you delete a stream?

Is it a DELETE request for, say, /streams/<stream-id> ?

No, it's a POST to "/" with an entity:

    
    
      {
          "StreamName": "string"
      }
    			

Sigh.

~~~
mfenniak
Amazon success is a strong indicator: RESTful APIs don't really matter. You
can build a very successful business without focusing on the format of your
APIs, as long as you build something valuable.

~~~
atwebb
It helps if you're already one of the biggest players. A smaller company may
be able to make a more attractive offering with REST.

------
socialist_coder
I can't wait to rip out my "client => server => SQS => queue processor => S3
=> redshift" flow and turn it into a "client => kinesis => redshift" flow.

Web scale analytics capturing just got way easier.

~~~
itsmeduncan
We just wrote something very similar to this using Kafka which replaced a very
similar system you have. Do it. We would have used Kinesis if it was around.

------
conflagration
This looks interesting at first glance, but to be honest, I don't fully get it
from the description. Is it some kind of Storm/CEP/Graphite combination? Where
does the actual "processing" happen? Do you have rent additional EC2 resources
for your workers? It looks like a potentially powerful and interesting
service, but the documentation could be a little more clear and maybe include
an example of a DAG layout for a typical application.

------
metabrew
As I understand it, this is very similar to Linkedin's Kafka.

To understand how you would use it, read some of the kafa docs:
[https://kafka.apache.org/documentation.html#introduction](https://kafka.apache.org/documentation.html#introduction)

~~~
dkersten
From my reading of it (and my use of Kafka in the past, as an input to Storm),
Kafka is a message queuing system, where Kinesis is a message processing
system. That is, Kafka is more similar to SQS or RabbitMQ and Kinesis is more
similar to Storm or S4.

Of course, I haven't used kinesis and have only glanced at the docs, so I
could be wrong - it could very well be that Kinesis is the glue between data
input and data processor rather than the data processor itself, in which case,
yeah, it looks a lot like Kafka.

------
chollida1
Does anyone have any experience using this technology? I'd assume it works
best for post processing or pseudo real time data feeds( ie within a couple of
seconds as opposed to sub 20 millisecond reactions).

Where do people usually store the processed data from this system?

I can see something like this being useful for things like post trade
analytics or post day analyzing the efficiency of an given algo.

We use an inhouse CEP to do this sort of work currently but I'd rather rent
than buy servers for this offline processing.

By the way, if anyone is in the Toronto area and interested in working for a
hedge fund my contact info is in my profile

------
Kequc
When I think of streaming my first thought is that this is a cloud service
that processes audio and video in real-time. But none of the example use cases
mention that is a possibility. I'd love the ability to stream and convert
audio then forward converted files to S3 in a simple way.

Currently it involves running an EC2 instance which only converts after the
audio is received in full. Then when it's done, transferring it to S3 is...
slow.

~~~
n3thin
you might need to look into
[http://aws.amazon.com/elastictranscoder/](http://aws.amazon.com/elastictranscoder/)
if it can satisfy your use-case

~~~
Kequc
The audio file must exist somewhere, it offers no live audio support. So the
bottleneck would be at the beginning rather than the end.

~~~
theg2
Elastic Transcoder doesn't support this natively, we ran into this issue so
ended up rolling our own solution as the commercial versions were incredibly
expensive for a non-profit.

------
nnx
It sounds exciting but I don't fully get what this is good for. It sounds like
a stream (of "records") on this would be the equivalent of a queue and
messages in Amazon SQS? only Kinesis is lower latency and billing is both
capacity-based and quantity-based rather than only quantity-based in SQS?

------
dzhiurgis
How many visitors you'll need to have to make some sense out of their
behaviour?

------
lhaussknecht
Looks similar to my logstash -> elastic search -> kibana workflow.

