
Ask HN: Learning material about storage of gaming telemetries in column storage? - markus_zhang
Hi experts,<p>I&#x27;m looking for learning material regarding the following information:<p>- Event based social game telemetry ETL with high frequency (let&#x27;s say 50K JSON strings per second as a start) and high volume<p>(Think some events like game_load, level_beat, purchase_made)<p>- Must be loaded into a columnar database (think Vertica)<p>- Data modelling of the telemetries for business&#x2F;data analysis<p>I can skip the first two points as I don&#x27;t have control over that, but could be interesting to know the technology considerations. I think &quot;Data intensive Application&quot; would be a good read for those two.<p>I&#x27;m mostly interested in the last point -- how should I model the telemetry data, so that they can be easily used for business analysis? For games we typically look for answers for:<p>- How does an A&#x2F;B test fare (engagement&#x2F;revenue)<p>- Game Level difficulties<p>- Are we giving out too many coins? (Game economy)<p>My main concern is: with the introduce of columnar store, I see more and more often that we model the data into very wide tables. I understand it&#x27;s done so for speeding up query speeds (less joins), but data modelling is also about business side, so how do we approach the problem?<p>Sorry for throwing a vague problem but that&#x27;s the best way I can do...
======
markus_zhang
The reason for this question is that I see a lot of books talking about the
ETL part (e.g. Design Data Intensive App, and tons of others), and also a lot
of books talking about the business analysis part (data
science/analyis/mining), but except for Inno/Kimball I see very few books
talking about data modelling FOR COLUMNAR DB, FOR GAME TELEMETRY DATA, etc.
And I'm realy really not sure if star schema/dimensional modelling still makes
sense as everyone I talked replied that in the age of big data, columnar
database, we need very wide tables where we can find every fact we need and
every dimension we need.

