
The pain of a successful Hacker News launch - tipiirai
https://volument.com/blog/the-pain-of-a-successful-hacker-news-launch
======
jteppinette
This article comes across as being really fake. You can clearly tell that the
goal is to to drive additional traffic to their v1.0001 instead of actually
diving into what they did wrong. Were they really investigating an entire new
DB and user interface in the few days after launch? If your initial design is
so far off that it requires a complete rewrite on day 1 of launch, then you
have serious problems. That is not “avoiding pre-mature optimization”. That is
an architectural failure.

~~~
amirathi
> That is an architectural failure.

Let's not jump the guns too soon. Analytic products are notoriously hard from
scaling perspective. UI, APIs, Infra that works crisply for 1x data, starts to
become sluggish at 10x and really breaks down at 100x.

Startups solve the 100x scale problem when they get there (sensibly so). In
this case, they got there way too quickly thanks to HN traffic.

~~~
jteppinette
IMO, they probably failed way before 100x expected usage. It was probably
before 10x expected usage. This isn’t crazy. This is a problem that happens
for a lot of teams dealing with scale. You engineer for 1x (checking off
requirements) and anything over that tears everything to shreds. The
difference between developing a solution on your local and hitting your PERF
environment is massive. e.g. I can’t run a TB DB on my local, so... if I don’t
see queries running in less that 10ms-100ms, I basically know that the queries
are never going to complete in PROD. Scale is hard.

------
StavrosK
I wanted to try Volument but I noticed that their website wasn't redirecting
to https, they had set the HSTS header and forgot about http since they only
ever saw https on their browsers. This put me off because I thought, if they
got this wrong, who knows what else they got wrong?

I emailed them and they replied very soon and fixed the problem within a day,
though, so at least response time was excellent.

------
dsherry
I looked at your problem page and it's not true that Google Analytics bounce
rate is not useful for single-page applications.

You can trigger events with JavaScript and choose whether they affect bounce
rate, so you can manually trigger an event when the user scrolls down, clicks
something, spends a few minutes on the site, etc. So your problems with the
bounce rate metrics are just not true.

Also I saw you mention it doesn't track returning users, that's not true
either. I can actually see insights into how many users return month after
month, and it'll tell me what my activity is per channel and if one channel
outperformed others, so it's pretty powerful in this regard for a free
platform.

I skimmed it but from just reading those it sounds like you should try to get
better familiarity with Google Analytics and talk to users to see what their
main pain points with existing platforms are, as well as establish more
concrete value propositions. Google Analytics is free after all, I'm not
convinced yet to move to a paid platform that tries to address problems I'm
not experiencing.

~~~
volument
Google Analytics bounce rate isn't useful for single-page applications unless
you monitor user activity with JavaScript and GA event tracking. Not everyone
is aware of this and/or can do this. We should update the docs with regards to
events.

GA can do retention reports or "cohort analysis" and show how visitors return
in a monthly basis, but there are two critical limitations on it:

1) It doesn't show you which landing pages, target markets, or their
combinations are retaining the most.

2) You can only study one metric by the time and not the total retention +
conversions each segment generate, which should be the deciding factor

There is no initial interest, nor predictions either. So when doing a new
campaign you cannot take any insights from cohort analysis until you wait for
enough return visitor data.

We certainly agree that Google Analytics is an awesome tool and it has served
our sites really well in the past. It gives tons of general insights for a
free service, but isn't designed specifically for conversion optimization.
Hence Volument.

------
saagarjha
I was expecting this to be the usual "Hacker News commenters were awful and
made me feel horrible", but was pleasantly surprised that Volument found value
from launching here.

------
IAmEveryone
It's a strange type of depression where you come right back for more. "Learned
helplessness"? Masochism?

I'm slightly irked by this use of "depression" anyway. It somewhat trivialises
one of the top 10 deadliest diseases, whose sufferers are already facing the
constant pain of their disease being considered "fake" or a lack of character
or laziness etc.

Sorry for being the "language police". It's just a minor point, and I know the
word is commonly used in this way. But even ignoring the issue above, I think
there are simply better words for the specific situation: "depression" evokes
the spectre of inactivity, of helpless, and of despair. But you did tackle the
problems that came up, so words such as "challenging" (yeah, I know),
"frustrating", "stress", "maelstrom", or "adventure" may not only improve the
text's language, but also project a more positive image for potential
customer.

~~~
volument
Good point. Changed "most depressing" to "saddest" on the article ingress.
Thanks.

------
reilly3000
Cheers for shipping a product! I recall the post, and as a heavy Google
Analytics user the landing page made some very salient points. This post was
honest and challenges the fantasies of many hopeful hackers. "Catastrophic
Success" is no joke.

I can sympathize with the challenges of running a web scale analytics tool,
after helping build and run an analytics stack for an ad network. Even with
our relatively small group of customers and lots of planning, the data was
abundant, the queries became quite slow, and the pipeline was never "real-
time" enough for some.

The web is a noisy place! URLs parameters are tricky to canonicalize into
pages when dealing with many properties. There is a constant stream of
nonsensical referrers and even referrer spam. Traffic can be exceptionally
spiky, and cascading failure in the pipeline is inevitable without careful
planning. Data loss is irrecoverable, and any incident needs to be carefully
explained to users of the product so they don't deceive themselves into making
decisions based on missing data.

One specific point I'm concerned about from the post is that while comparing
HackerNews traffic to organic search, their analysis fails to account for
relative traffic volume. Its always easy for a small volume traffic source to
show outstanding metrics, but miss the broader point: quality traffic sources
usually don't scale. A useful analysis should assign weight to overall volume.
ORDER BY MAX(CTR) will tell me that some keyword/referrer was the top
performer at 80%, but without knowing that was for 3 visitors I could easily
get the wrong ideas. I think analytics tools aspire to make it difficult to
come to incorrect conclusions, acknowledging that most of their users will
have no meaningful background in statistics.

All of those challenges are surmountable, but it definitely gives some
credence to the talented engineers who have make analytics stacks look easy. I
wish Volument luck in their journey and will keep an eye on their progress,
because their pitch remains quite appealing.

~~~
volument
Volument shows the size of each segment and doesn't generate any report unless
the size is "sufficient". However, it doesn't reward bigger segments when
calculating the score for traction.

Thank you for the solid feedback!

------
jodrellblank
_Volument reveals us, for example, how the people that come from Google search
generate much more traction than the other market segments. A super valuable
piece of information that we will use on our marketing._

Does it show that people coming from HN want to glance at your startup and see
what it does, whereas people coming from Google have no idea what to expect
and take longer to work out who you are and what you do?

Does it show that neither Google users nor HN users know what you do, but HN
users are more experienced with websites and spend less time mentally
categorising them?

Does it show that HN users are less interested in marketing / conversion than
people who Googled for a related term enough to find your site?

What's the super-valuable insight and how will you use it?

~~~
volument
It indeed shows which market segments just glances the startup (3, 7, 15
second stay rates) and which segments find the content more interesting. It
doesn't tell you the exact emotional reason why someone leaves early.

The super-valuable insight is to know how the various segments proceed on the
AIDA funnel (attention, interest, desire action) so you understand how your
visitors convert.

It also reveals your best target markets and landing pages that generate the
most traction. This data is not available elsewhere.

------
sundvor
Nice write up. 3s, 7s etc stay times were great to see. Really liked this A-B
breakdown of user engagement.

------
amelius
Depression is a medical condition, not an emotion. Great to see the title
changed.

~~~
nmstoker
I agree about the title change but it's not exclusively medical (eg the Great
Depression)

------
JeanMarcS
I always ask my clients wanting to sell their manufactured products online :
are you ready if the website is successful fast ? Can you handle twice your
actual sales ? Ten times ?

You never know. Are you ready for success ?

This story looks like that.

