

Using C for a specialized data store - saulkw
http://pixenomics.tumblr.com/post/18892378997/using-c-for-a-specialized-data-store

======
JoachimSchipper
Umm, we're talking about just over 18MB here (1200 * 1000 pixels, 16
bytes/pixel, see [http://pixenomics.tumblr.com/post/16895861678/how-to-
send-1-...](http://pixenomics.tumblr.com/post/16895861678/how-to-
send-1-2-million-pixels)). That you can just dump over the wire as a binary
blob. Why are we talking about this again? Use your favourite language, just
keep it in a big blob in memory, and have fun.

~~~
saulkw
Memory isn't an issue. It's processing the data and turning the storage into a
format the client can read. A big blob isn't easy to send to the client unless
it's an image or something and then it becomes an issue when you want to
manipulate the data or process it.

~~~
blibble
it's a 1000x1000 image... this is a trivial problem

~~~
saulkw
Can you elaborate?

~~~
swah
What he means is: you wouldn't look for a "solution" for writing a 1 mb text
file to disk, because its quite trivial and fast in any language.

------
nknight
Seems to me they skipped right over the most obvious option: Redis.

It's quite fast, you can use a Redis string as a random-access array up to
512MB/each, and there are several good ways to handle persistence/backup. I
don't think there was a need for them to write any C themselves.

~~~
saulkw
We were looking at that (as well as riak) but processing the data would
require pulling all the data into PHP. I guess you could do the processing in
C but it's then just as easy to store it there as well.

~~~
Mikushi
Have you looked into the LUA scripting option for Redis? Allows for some
processing to happen on the server side, and it's quite powerful.

~~~
saulkw
That sounds like a good option. Thanks, will note it.

