
How we validated our SaaS product without building it - bvanvugt
http://blog.sendwithus.com/how-we-validated/
======
JimmyL
I went though sendwithus zero as an unknowing participant in the experiment,
and it sucked.

At the time, the company I was working at was looking for a service exactly
like this - a service where our app could just fling events, and then we'd do
all the logic to control what emails got emitted on the basis of those events
on the external service (and A/B test them). One of our "growth hacking" guys
found SendWithUs somehow, and introduced us. We had a few calls with them -
the service seemed like it had potential, and we got beta access to it...which
was really sendwithus zero, but sending basic email.

We got most of the way though setting up an integration with them, but every
time we asked when the A/B testing framework - which was what sold us on the
product - was coming online, we were told "it's coming soon, it's what we're
working on right now." So we held on for a few more weeks, and gave them some
time...but nothing came of it. I was skeptical at the time that they were
actually building it, even moreso having read this.

The positive side is that while we were being strung along by sendwithus, we
found Vero ([https://www.getvero.com/](https://www.getvero.com/)). They're not
perfect either, but we've been on them for 6-8 months, and they've done well
for us. They're responsive to our issues, are willing to Skype or chat when we
try and develop some sophisticated campaigns on their platform, and have (or
so it seems) rejigged their roadmap a few times to add in small features that
we ask for. ExactTarget is still a more full-featured package, but I'm happy
with the price/value of Vero.

So well done, guys. You've got an interesting story about how you bootstrapped
your business, and you're evidently doing OK. I wish you the best, but I can
tell you that you won't be getting my business (or my recommendation) - at the
time I thought you were small and were actually on the cusp of delivering what
you had promised us worked; now I see that we were just one more data point
for a feature set that you never planned on having ready when you claimed.

~~~
mrmch
Hey Jimmy, I'm sorry to hear that was your experience.

I don't know when you found us, but it did take us time to roll out our A/B
testing; we never intended to string anyone along.

If you're willing to chat offline about your experience, please get in touch:
matt@sendwithus.com

~~~
thatthatis
Bullshit. Your entire "zero" approach is stringing customers along. Which is
fine. Just don't try to pretend like you weren't doing something you knew
would upset your test subjects.

------
KiwiCoder
> We spent the next 24 hours getting as many people as possible to view and
> interact with the experiment

OP - could you elaborate on how you got these 803 unique visitors within 24
hours?

~~~
bvanvugt
Sure - that's probably worth another blog post too. We used a combination of
twitter, adwords, marketing forums, and personal network.

~~~
lzecon
If you're relying heavily on your personal network I'd be worried that those
803 people were not true customers. Don't get me wrong, I think this is a
great idea, but I'm wondering how you know that a 803 person sample size is
big enough.

Also - could/would people click on more than one button once they found out
they were non-functional? It could be that some features would be best
developed in tandem with others.

~~~
bvanvugt
We intentionally diversified the traffic as much as possible, exactly for this
reason. Only a small portion of the traffic was from personal networks.

Repeat clicks are definitely interesting. It makes sense that once a button
was clicked and found to be non-functional, that may prevent future
interaction. But that first click still holds value. It'd be interesting to
pull stats on how many unique visitors generated those click events.

------
ForHackernews
Does anyone else find this kind of pre-development marketing kind of scummy
and deceptive?

It's basically lying to potential customers: You're pretending to offer to
sell a product that doesn't exist.

~~~
mrmch
Perhaps if we were accepting pre-orders or requiring a credit card. A soft
launch like this would be more comparable to a focus group for a traditional
product launch.

~~~
ForHackernews
But you're not being honest and up-front about the fact that it's a focus
group.

If it was like "Click here to take a survey about what features you most want
in an email service" that would be one thing, but that's not the messaging in
your case.

~~~
petervandijck
Effective focus groups aren't up-front or honest either. Effective experiments
don't tell the person what the experiment is.

~~~
ForHackernews
But they at least tell you that it _is_ an experiment. Maybe we need human-
subject ethics boards for marketers.

~~~
troymcginnis
Just knowing that it's an experiment defeats the purpose. By getting honest
responses and behaviors, it is easier to build a solution that people actually
need or would actually use.

~~~
eropple
And at the end of the entire process you've wasted their time with a
nonfunctioning product.

Focus groups et al. are consensual. This is deceptive.

------
sonink
\- Just because some people clicked on features they want, it doesnt mean they
will pay for it once you launch.

\- Just because some people are using your free features, it doesnt mean they
will pay for it once it is paid.

\- Just because you have 5 paying customers, doesnt mean it will be easy to
find out 500 more.

\- Just because you have 500 paying customers, it doesnt mean what you did to
acquire these 500 will scale to the next 5000.

\- Just because you have 5000 paying customers, it doesnt mean that there
won't be a new upstart which will dramatically undercut your offering and
acquire most of these.

There is a name for this validation process - its called a 'startup'.

~~~
troymcginnis
Well said. Everyone has to start somewhere :)

~~~
assasinine
exactly. startups are essentially always validating, beginning with what
sendwithus/others did to get a product and idea, and then going through the
steps sonink listed. like sonink is saying, there are no guarantees with
startups, so you might as well try to stack the deck in your favour right from
the get-go.

------
redguava
Are you sure the value you got from the metrics exceeds the bad will generated
by annoying your potential customers?

To me it's a lack of respect for your users and their time.

~~~
mrmch
Some people seem to be missing that first screenshot; the landing page clearly
started with "Start The Demo" ([http://sendwithus.wpengine.com/wp-
content/uploads/2014/03/Sc...](http://sendwithus.wpengine.com/wp-
content/uploads/2014/03/Screenshot-2014-03-09-14.27.00-e1394434326899.png))

~~~
redguava
Demo doesn't imply fake features. For me demo would be not my data, perhaps a
sample set in there.

I'd have no problem if it said "there's a bunch of features not yet
implemented, we have the buttons there to gauge interest and we'll work on the
most needed ones".

------
allanlundhansen
"We knew that transactional email was a problem worth solving" A story worth
telling in it's own right I believe, when clamoring on about lean
methodologies. Nice article would love the pre-phase lean story leading up to
this stage.

~~~
bvanvugt
Absolutely! Prior to this experiment we spent nearly two months talking to
customers/partners of all sizes.

The details of that period would make a great follow-up blog post.

------
DanielBMarkham
Relevant similar story: I had this site for years, [http://paycheck-
stub.com](http://paycheck-stub.com)

Developed it back when I was doing microsites, and I wasn't really sure what
to do with it. My main question: why do people _visit_ , anyway? I figured I'd
know what to do with it once I figured out why people were there.

So I did just what these guys did. I created a list of buttons down the page
with all the possible things people visiting a site called paycheck stub might
want. Hooked them all into Google Analytics. Paycheck templates? Fake
paychecks? Loans? Payroll services? A job? Sample paychecks for school or
presentations?

Very interesting results. First, about 90% didn't want any of that. God knows
why they were there. But out of the other 10%, I got a couple pieces of useful
information. They were looking for payroll services. So I put up some banner
ads. (I think I took down the Adsense. Can't remember)

That, in turn, provided more ad revenue than simply using Adsense on the site.
It also provided me with a list of topics to develop and some search terms to
try to hit.

So I developed those topics and tried to do better on those search terms. At
one point I was almost ready to jump into online payroll. Fun times!

Then Google killed my search ranking -- I wasn't paying for links or spamming.
Beats me why. I got so frustrated with them I gave up.

Still get quite a bit of traffic, though. But now, I'm sure, the traffic
pattern is completely different. I wonder if, for static sites with a lot of
content, this is something you should repeat every year or so?

Like these guys, I found it a rewarding experience, both for my understanding
of the site and for my readers. The reason I'm posting is that many times we
view this as a step on the way to creating SaaS, but it can be just a great
way to connect with your readers. I bet it's extremely applicable for a lot of
blogs.

Wish I could have figured out why those other 90% were there.

~~~
vertex-four
Perhaps the 90% were put off by the list of payroll-related things? It likely
looked like a slightly more elaborate version of one of those parking sites,
with the links to nowhere that are all vaguely related to the domain name (and
tons of ads).

Your current site again looks like a spam site. It contains heavily SEOed text
that looks like it came from a content farm, some ads, and many of the sort of
links that a parking site contains; lots of internal links to more heavily
SEOed text.

Basically, you tripped up against heuristics because you look almost exactly
like the sort of spam that Google is trying to clear out of its indexes,
whether you had good intentions or not.

~~~
DanielBMarkham
I don't know, vertex. They were organic traffic, and as I recall they were
coming for what seemed like relevant keywords. It's been a while, though.

My current site is a micro-site. I and my wife wrote the material. I hand-
designed the layout, hand-coded all the html. As a programmer and a writer, I
was pretty excited about the idea of microsites a few years ago. Here was a
chance to combine my love of writing and programming to create online
magazines around particular topics. I envisioned writing dedicated
applications, doing some live updating of relevant information, and so on. It
was (and is) a pretty cool idea: a spot between a blog and a dedicated topical
site. As far as the guts of how the site works, using internal links, scoring
highly on search terms, etc? Read that in an article somewhere. The goal is to
create a small spider site around a topic such that people land on one page
and see the top 3 or 4 related pages immediately, allowing them to browse
tangentially if they're in the wrong spot. There's probably a better way to
optimize around that than I did, but I never got around to it. And as far as
Adsense, I added on ad unit to pay hosting costs. As I said, once I figured
out why folks were visiting the idea was to get rid of it. But Google kept
emailing and suggesting I bump it up to three units. Go figure.

And yep, Google killed me because everybody else in the microsite business
used content farms, cranking out thousands of pages of crap, loading up with
Adsense, flash ads, grabbing emails, and so on.

I still think the microsite idea is a great one, although, you are correct,
anything that looks like one is going to get penalized. That's a shame.
There's still a lot of potential there. Check out my microsite for books that
hackers might like for an idea [http://hn-books.com](http://hn-books.com)
(This is also a good example of a totally-static site that still has a
database and lots of interactivity)

In either case, Google's algorithm decided my couple of hundred hours of
programming, design, and writing effort wasn't worth it. After horsing around
with the idea for a year, losing valuable time, I gave up. As somebody who was
not out to game the system, I was collateral damage in the bigger war against
crappy content.

------
bradhe
> We knew that transactional email was a problem worth solving and we were
> determined to build a great solution.

I have a problem with this phrase. Seems to me it needs justification maybe
just a verb? I don't immediately think of "transactional email" as a
problem...

~~~
bvanvugt
Hmm, how about "transactional email management"?

~~~
bradhe
just as vague. managing transactional email? What's wrong with doing that?
Maybe I don't have the problem...I don't get it.

------
nobodysfool
I did that with my father's computer store website back in 1998. We had
computer descriptions, a price, and a 'buy online' button. The button just had
a hit counter on the page, with directions to the store. Every hit meant
someone wanted to buy a computer through our website and couldn't. I showed
him the data that we could double our sales just by making an online store,
but he was not convinced. In any case, the cash flows of that business
wouldn't have worked out for him. Still, I thought this was a widely known
technique.

------
xg15
The apparent origin of the idea, the "button to nowhere" that they linked to
in the story is also a good read: [https://medium.com/design-
ux/77d911517318](https://medium.com/design-ux/77d911517318)

From a user perspective, I think there are both good and bad ways how this
technique can be used:

I actually like the way Sendwithus used it - as part of a very very early
"pre-alpha" version where you're not yet interested in providing a reliant
service and just want to get a feel for the demand. As a user you're informed
early-on that you're looking at a version in development - so you will not be
surprised by missing or incomplete features and will maybe even motivated to
provide additional feedback.

The downside of this approach is of course that the data you're gathering is
really only good for that - an early overview. Your test group will likely be
very different from the actual people who will later use the service. The
usage patterns will be as well.

The second way to use the technique is how Nick Kishfy describes it in the
"button to nowhere" article. In that approach, "buttons to nowhere" are not
just inserted into an early preview/development version, but into the actual
product as a continuous part of the development process.

This would solve all of the above problems and would give you a much richer
set of data over a much longer time span - but from a user perspective that
would be the worst thing possible.

~~~
mrmch
This is actually a good point -- the "button to nowhere" product is
particularly valuable at an early stage.

We're working on a followup story on how we do customer development/feedback
with a established product.

------
unfunco

        We knew that transactional email was a problem worth solving
    

Really? Hasn't this been solved already, by many businesses (SendGrid,
Mandrill, SES, MailJet) – I'm happy you've managed to build something and are
seeing some success, but I don't think you're solving a problem that hasn't
yet found a solution.

~~~
davidw
Altavista worked ok, but I'm glad Google came along.

I think that your comment isn't a _bad_ one, but should go more in depth. Some
markets really are unassailable - others aren't. The reasons are many and
sometimes complex, so analyzing one is often quite interesting.

~~~
chc
I don't think unfunco is suggesting that the market is unassailable. The point
seemed to be that this problem wasn't one that needed to be solved (because
there were already solutions), not that there isn't room for another solution-
provider.

------
patrickdavey
What's the story with using logos for integration (SendGrid, AWS etc.) without
actually supporting them? Is it not possible to get into legal issues there as
it's quite an effective boost to the site to make it _look_ like you're
associated with very reputable sites?

Or.. just something to worry about when people come knocking? :)

~~~
mrmch
if they come knocking... ;)

------
alexsilver
We are going through the same approach with our current beta customers (and
will be making our demos available to a wide audience in the future too) and I
can attest to the validity of this approach. We make it very clear to our
customers that this is a demo and their changes will go away if they refresh
the page. We don't have pop-ups integrated since we collect emails on another
page. I think event tracking should be sufficient but we'll have to see what
the data says in a little while.

We actually also use these semi-live demos instead of mock-ups (since we'll be
implementing the same pages afterward anyway) since only one of us is
proficient with Photoshop but everyone is comfortable with Bootstrap. It
serves as a testing ground for various approaches for JS code too.

------
BrechtVds
I'm wondering if the A/B testing score was skewed due to it being the first
non-working link in the menu. Someone going through the menu to see what's
possible will probably take a look at the top item first.

A random order for the menu items might have been better. Or maybe not.

------
alexeyza
Awesome way to design a product. I wonder though, how the users reacted when
they saw "dead end features". Were there any frustrated users when they saw it
was mainly a "mock-up" web site ?

------
ultimatedelman
this sounds like the most frustrating experience for a user ever

~~~
bvanvugt
In a way that's a good thing - having a user work through the frustrating UI
and still be interested in the product is amazing validation. Early adopters
can self select.

~~~
mkr-hn
It makes them resent you and long for a competitor.

~~~
mrmch
Our experience was to the contrary. It's not shown in the blog post, but users
were thanked for their feedback immediately.

The reaction was overwhelmingly positive; people were excited that we were
building something in this space.

------
edgecrafter
Obviously smart and handy way of getting metrics for a wannabe startup but as
some comments states I would feel cheated as no real product are offered, but
offers are pretended to be real. Thus wasting my valuable time. I bet this
tactic will backfire soon as more and more try to do the same. I would be more
positive if some incentive were offered, e.g. Thx for your interest get 3
month for free when we launch ...

------
joshdance
Can't decide how I feel about this. Does anyone feel that this type of testing
is kind of dishonest? On one hand, listing a product as available with a price
is a great way to gauge interest. On the other hand, telling someone it is
"sold out" when it was never available feels wrong somehow. What do you think?

------
rubiquity
I think this approach is very interesting and may give it a try on a product
in the future. How did you go about directing people to the demo though?

~~~
bvanvugt
Excellent question, and probably grounds for another blog post. We tried to
use a variety of channels, including twitter, marketer communities, and even
some google adwords.

------
pbreit
So did the API turn out to be a popular feature? I'm wondering if APIs prove
interesting on the surface but not always in practice?

~~~
bvanvugt
In our case it did! Our core product today is an API to send templated email.

