
How Heap's 5-line MVP evolved into a 30,000 TPS beast - mallyvai
http://codeinsider.us/i/2.html
======
shostack
It looks like Heap is making some progress with their UI which I'm glad to
see.

That said, I'd love to see Heap pull in the rest of the web analytics picture.
Event data is interesting, and they seem to have a decent solution for the
issue of setting up a massive amount of event tracking (and losing all the
data you haven't set events for yet). However event data in a silo without the
context of things like referral URLs, search query data, campaign metrics,
etc. are critical for gaining the full picture.

Perhaps I didn't tinker with the demo enough, but I didn't see those pieces of
the puzzle present. For any online marketer serious about analytics, you need
the complete picture under one roof.

That said, this is a great step forward in a space where this pain point
around setting up event tracking has been largely unsolved for some time. I
wouldn't be sad at all to see them acquired by someone like MixPanel,
KissMetrics, or Google.

~~~
matm
We automatically annotate each event with referrer, search keyword, and all
UTM properties. Full list of properties here:
[https://heapanalytics.com/docs#events](https://heapanalytics.com/docs#events)

Would love to know if there's other important context we're missing.

~~~
shostack
Awesome, clearly a miss on my part. Would love to see more interface elements
designed around exploring the web analytics piece of it though. Google
Analytics does a fantastic job of making this data intuitive (well, for
someone knowledgeable of the space) and explorable. Things like cross-channel
tracking and other items are also critical.

I'm playing around with the demo in our test environment right now and really
love your Event Visualizer tool. I need to play a bit more, but I'm hoping
that your ability to check/uncheck the components that define the event will
let us work around issues of dynamic selectors generated by YUI which was our
team's concern for why Heap might not work with our setup.

 _EDIT:_

Looks like YUI is still a major issue. If you're curious, you can look up my
HN username in your email list to find my account and see what I'm talking
about.

~~~
matm
Thanks for the heads-up. As mentioned over email: if the Event Visualizer
isn't providing quite the right precision, keep in mind that you can manually
specify hierarchical CSS selectors within the dashboard itself. It'll work as
expected.

------
pdog
_> Are you still on MongoDB?_

 _> Oh god no._

Pretty damning condemnation of MongoDB, no?

~~~
raverbashing
Humm... No

They also outgrew PostgreSQL, and gave up on Redshift, is that a condemnation
of them as well?

~~~
aaronblohowiak
Looks like they are still using PostgreSQL, just not a single instance. It
looks like they are using Citus Data to handling sharding and merging
responses.

------
integraton
So is the "massive custom data store" using Postgres or is it something
totally different? Dan's answer sounds like Postgres + Citus DB is the current
setup.

~~~
mallyvai
It's in the interview, like halfway down. They're using Citus over Postgres.

------
mstump
30k TPS is pretty damn low. That's less than the throughput of a single
Cassandra node...

------
nrbafna
I am wondering what the 5 lines for MVP were. Anyone mind taking a shot?

~~~
matm
This is referring to the integration code itself. It was something like:

    
    
      var DOMEvents = ['click', 'submit', 'change'];
      for (var i = 0; i < DOMEvents.length; i++) {
        window.addEventListener(DOMEvents[i], function(e){
          $.post('/collector', {type: e.type, target_id: e.target.id, target_class: e.target.className});
        });
      }

~~~
onion2k
That's interesting and immediately obvious how it could both be useful and
valuable in the right domain. Logging what people are doing on modern event-
driven websites definitely is a problem worth solving. My let's-made-a-
complicated-version-first mindset would have extended jQuery's .bind or .on
prototype in case the site is using event.stopPropagation(), and that's why I
never finish anything. Thanks for sharing.

