
Low Latency Architecture – AWS - minormind
I am looking to design a low latency (sub-second) real time system. Clients (~600k) will feed data to Kinesis to Lambda and then to DDB and Redis for fast retrieval. Issue is Kinesis is batched so there is default delay. Can I replace Kinesis with SQS and get better performance?
======
QuinnyPig
This is one of those "build a quick PoC and find out" moments.

Also consider what happens when latency spikes; performance isn't guaranteed
in this environment. At scale, edge cases become the common case.

On the DynamoDB side, I'm sure you've already seen the DynamoDB Accelerator
(10x read performance) already in preview, but just in case, see
[https://aws.amazon.com/dynamodb/dax/](https://aws.amazon.com/dynamodb/dax/)

------
kartiksura
I work in subsecond latency domain. Can you elaborate what do you want to
achieve in the sub-second: Does the ingestion have to be subsecond? or the
entire cycle of ingestion and data available into redis/DDB needs to be done
in sub-second?

~~~
minormind
Entire cycle in sub-second. I agree that it depends on many factors but as per
the system, we are looking subsecond from ingestion of data from client to
availability of data for query.

A -> stream -> datastore (subsecond)

~~~
kartiksura
I see. I would suggest you explore the following: 1\. CitusDB (an offshoot of
postgres with high ingestion rate, cloudflare used it) 2\. Druid.io (popular
in ad-tech) 3\. Linkedin Pinot (uber uses it for similar use-cases) 4\. memSQL
(used in ad-tech)

~~~
minormind
Thanks for the pointers but I am planning to revolve more around AWS ecosystem
so that I have less ops burden of managing servers + regionalization becomes
easy.

------
sathley
How about using Pubnub or Pusher for this use case?

------
sjeanpierre
If your batch settings are low enough Kinesis should be fine. SQS with long
polling would also be pretty fast if you have enough workers.

~~~
minormind
I am also okay to dump the whole architecture and re-architecture. Will do a
POC but looking to know, if some one has designed something of this sort.

