No I am not using Kafka. It is just basic python server. I want to calculate what is the nth percentile of size of my incoming request object over past one hour.
I don't want to store size of each request in memory. It will eat so much of my RAM.
Incoming traffic:
- 1st batch
--> 60 requests
---> size of 1st request is 10kb
---> size of 2nd request is 2kb
...
...
- 2nd batch
--> 10 requests
---> size of 1st request is 5kb
---> size of 2nd request is 8kb
...
- 100th batch
I am talking about percentile(10th, 50th, 95th) size of request.
It's very likely we're doing something wrong with this test, but after many hours of trying we couldn't get our simple test to pass for Logstash, even though it passed for others:
Yeah, sometimes I am really pissed off because many people just try to find issues in my PR. They don't care for the rest of the work. I would suggest you talk to your manager about this topic. You need a better environment and which is fine.
I don't want to store size of each request in memory. It will eat so much of my RAM.
Incoming traffic:
- 1st batch
--> 60 requests
---> size of 1st request is 10kb
---> size of 2nd request is 2kb
...
...
- 2nd batch
--> 10 requests
---> size of 1st request is 5kb
---> size of 2nd request is 8kb
...
- 100th batch
I am talking about percentile(10th, 50th, 95th) size of request.