Hacker News new | past | comments | ask | show | jobs | submit login
Ask HN: Best way to store integer array/list per key
1 point by sangupta on Feb 4, 2014 | hide | past | favorite | 7 comments
What is the best data-store or technique if we were to store millions of integer lists identified by a unique key. The following functions are intended on such lists:

1. Read for a given key 2. Add items to the end of the list against a key 3. Read portions of this list, from-to index 4. Rarely, remove items from the list

The data size is estimated to be roughly 100-million keys, with average of 2-3 thousand integer-list per key.

The key itself is an integer.




[deleted]


The lists are expected to contain around 2-3 thousand integers. The maximum may lie in the range of 10-11 thousand. As the lists are expected to mostly grow, the numbers would increase. But, the older elements can be purged to another storage mechanism as they might not be very heavily accessed.


Sorry I meant to say what size of integer you require, 8,16,32,64bit?


Sorry - my bad.

32-bit for now. I don't anticipate them going to 64-bit so soon.


Just working on something, will be right with you.


No problem - I will wait for your response. Thanks in advance.


https://gist.github.com/AndrewRose/8805629

Considering you are storing only millions of rows I figured a mysql implementation using blobs should do the trick .. but it all depends on the load you expect .. worth playing around with the idea I think.


Thanks for the code and helping with this. I have posted my queries to the gist itself.




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: