
Internals of Cubes – Lightweight Python OLAP Framework - stefanu
http://blog.databrewery.org/post/18013047462/cubes-python-olap-framework-architecture
======
Sukotto
On a related note. I was involved prototyping olap cubes (Microsoft stack) at
my dayjob. _HUGE_ success in terms of value added to the business. Sr
Management was used to getting stale data (at _least_ 24 hours old) of things
like trading positions, profit/loss, alpha/beta/gamma for portfilios, etc. in
_Excel pivot table_ canned reports. We turned that into same-day, build-your-
own reports in your web browser.

My group spun it off into a separate project group that took it all the way to
effectively-realtime views of the entire company's money. The project rapidly
became 100% mission critical and the related display framework absorbed just
about every other reporting tech in the entire company. (This was in a
9billion/yr trading company with a few hundred employees)

So, if you're a pythonista looking to make yourself more _visibly_ valuable to
the management of your company. Consider taking something like this technology
and running with it.

~~~
thibaut_barrere
Seconded - I work mostly with < 100 employees companies and with Ruby, but
I'll give the same advice.

There is plenty of very useful and fun projects to complete in this area (50%
of my yearly income as a freelance comes from data processing, etl,
datawarehouse etc).

------
stefanu
In addition to that, I've started to write a JavaScript library for the Slicer
server (not mentioned in the blog post). I am just starting with JS, so pardon
my style. The sources can be found here: <https://github.com/Stiivi/cubes.js>
No examples yet, however the goal is to be able to browse aggregated data
directly from JS and transform them into tables/charts.

Concerning backends: I have not much time to write other backends, however, if
anyone is interested in helping me with them, just drop me a line. I would
like to have at least nicer star-schema browser and perhaps the mongo DB
backend.

~~~
binarymax
Very nice. I did a day project a while ago to do something basic.
<http://www.binarymax.com/cube.html> (NOTE: doesnt work in firefox, but is
fine in all other browsers).

I'm pretty sure my algorithms would quickly run into stack and memory issues
quite quickly, but I havent looked at it in 6 months.

How long have you been studying OLAP structures and algorithms? Its the kind
of stuff that is very interesting, but I sometimes have trouble wrapping my
head around. I'm curious if you have any favorite reading materials on the
subject?

EDIT: also FYI, your link on the blog to the github js repo goes 404 since it
has an extra '.' on the end

~~~
aaronjg
For a good description of OLAP data structures and algorithms, I'm a fan of
_Data Mining_ by Jiawei Han and Micheline Kamber

------
asolove
Very interesting, thank you. There is a big opportunity for someone to re-
imagine the power of Lotus Improv with a more intuitive touch interface for
tablets. Perhaps it will be me, but I'd rather it be you than that it not
happen.

Data analysis as a service could be a big market. Why do Google Analytics,
Mint, my time-tracking service, my to-do list and every other online app I use
implement their own, custom, proprietary versions of the table/sort/pivot UI
for advanced use-cases? I'd rather they provided clean interfaces for easy
things and allowed me to join all their data together into one data cube for
serious reporting.

~~~
stefanu
Thank you.

There are couple of analytical services out there, as SaaS, that can connect
to various third-party sites, databases or other data sources. However, what
is missing is some nice, reusable, simple UI set of elements for pivoting and
drilling down. The idea is that apps can use Cubes as module for providing
aggregation and drill-down capability. Then you have a front-end module,
either within the app or any external, that can connect to the Cubes OLAP API
and you can do reporting as you like.

I'm not too much front-end guy, so any help would be appreciated. As mentioned
in one of my other comments here, I've started
<https://github.com/Stiivi/cubes.js> . Just proof-of-concept.

~~~
asolove
That's interesting because I happen to be something of a front-end guy :) I
will have a look at it when I have a chance.

