
Antirez: You Need to Think in Terms of Organizing Your Data for Fetching - aespinoza
http://highscalability.com/blog/2012/10/10/antirez-you-need-to-think-in-terms-of-organizing-your-data-f.html
======
bialecki
Also keep in mind that learning to "organize your data for fetching" is not
necessarily something you can do before you start your project. Many (most?)
times you can't predict which data access patterns will be most common and
benefit from using Redis, etc.

Starting with a "slower, but flexible" datastore like a traditional relational
database, monitoring which access patterns need a boost, and then optimizing
or introducing a new datastore is almost always a solid plan of attack.

------
spenrose
I still feel the canonical answer to "what should my default data management
policy in write-some-read-a-ton situations" is to, at write time:

    
    
        1) store an appropriate write-whole/data-mining-friendly format
        2) ASAP, for each major view, write out a Redis-style O(1)-to-read data structure
        3) think carefully about backup and replay strategies
    

You trade a slightly stale read of the very hottest data for much improved
performance on everything else, and more importantly, much simplified view
code.

The best reference I have found for this pattern, and it isn't great (too big-
SQL-centric), is "command-query-responsibility segregation":

<http://www.udidahan.com/2009/12/09/clarified-cqrs/>

------
MatthewPhillips
if the CEO/owner/founder of your company is non-technical he/she _will_
request the data in ways you wouldn't have thought about in advance. That's
just reality. That makes Redis not appropriate for most companies. It's also
too expensive for side projects. So that leaves technically-led startups.
Which is a good chunk of companies (and probably the funnest to work for).

~~~
noelwelsh
Lots of people use Redis at a cache, not a primary data store. You can have
full querying in your SQL database and fast access to common requests using
Redis.

In what way is it too expensive for side projects? It's the easiest data store
to compile and run that I've used.

~~~
MatthewPhillips
Here are some Redis hosting options, you tell me if this is affordable for a
side project:

<https://openredis.com/>

<http://redistogo.com/>

~~~
res0nat0r
The smallest plans listed are $7 and $8 a month. Sounds affordable for a side
project.

~~~
benarent
FYI, <http://redistogo.com> offers a free plan. 5mb, 1 database. Perfect for a
side project.

------
mmariani

      So anyway if your data needs to be composed to be served, you are not in good waters. -- antirez
    

Perfect! This sentence puts an end to SQL vs NoSQL holy wars. There's no
silver bullet. But we 'all' knew that already :)

------
siganakis
Data should not be organised based around retrieval or insert / update
patterns but organised according to the model that best captures the essence
of what the data is. That may sound fluffy, but most of the time data captured
is captured following something real happening that caused data to be
generated. Your data model needs to make sense in the context of the thing
that happened in the real world, not in the context of what is inserted or how
it is read.

The issue being that youre methods of collection and retrieval will change
over time and your data model needs to support that and still make sense for
existing data.

~~~
shuzchen
While what you describe is certainly the ideal, and likely applicable in a
variety of situations, there are a lot of real world situations where it can't
cut it. Antirez says it best:

    
    
       remember all those stories about DB denormalisation, and hordes of memcached or Redis farms to cache stuff, and things like that? The reality is that fancy queries are an awesome SQL capability (so incredible that it was hard for all us to escape this warm and comfortable paradigm), but not at scale.
    

You'll be hard pressed to find any medium-sized project in the wild that
doesn't require a layer of denormalization or caching to be reasonably
responsive (I mean, some frameworks come with that built-in - their users
might not even be aware this is happening). You might have some beautifully
crafted model underlying that layer, but don't fool yourself into thinking
that's all there is.

------
nsp
QQqfzrqqztzs

