I went though sendwithus zero as an unknowing participant in the experiment, and it sucked.
At the time, the company I was working at was looking for a service exactly like this - a service where our app could just fling events, and then we'd do all the logic to control what emails got emitted on the basis of those events on the external service (and A/B test them). One of our "growth hacking" guys found SendWithUs somehow, and introduced us. We had a few calls with them - the service seemed like it had potential, and we got beta access to it...which was really sendwithus zero, but sending basic email.
We got most of the way though setting up an integration with them, but every time we asked when the A/B testing framework - which was what sold us on the product - was coming online, we were told "it's coming soon, it's what we're working on right now." So we held on for a few more weeks, and gave them some time...but nothing came of it. I was skeptical at the time that they were actually building it, even moreso having read this.
The positive side is that while we were being strung along by sendwithus, we found Vero (https://www.getvero.com/). They're not perfect either, but we've been on them for 6-8 months, and they've done well for us. They're responsive to our issues, are willing to Skype or chat when we try and develop some sophisticated campaigns on their platform, and have (or so it seems) rejigged their roadmap a few times to add in small features that we ask for. ExactTarget is still a more full-featured package, but I'm happy with the price/value of Vero.
So well done, guys. You've got an interesting story about how you bootstrapped your business, and you're evidently doing OK. I wish you the best, but I can tell you that you won't be getting my business (or my recommendation) - at the time I thought you were small and were actually on the cusp of delivering what you had promised us worked; now I see that we were just one more data point for a feature set that you never planned on having ready when you claimed.
Bullshit. Your entire "zero" approach is stringing customers along. Which is fine. Just don't try to pretend like you weren't doing something you knew would upset your test subjects.
I'm pretty sure I'm one of the "growth hacking" guys in this story.
Personally, I don't see this as an accurate description of what happened. We were seriously considering sendwithus and were looking at how to integrate their product. We talked a few times (maybe 4 times?) and definitely saw the value of what they were doing and where they were going.
We also were very transparent about the features we needed to move forward and they gave us a fair time range for when they could deliver them. Originally, we were comfortable with the timeline. Of course, you sometimes want things to be built yesterday and we happened to come across Vero which had all the requested functionality already built. That's what caused the switch.
Hope that clarifies things and honestly, I'm disappointed by the above comment. Anyone who's built something from nothing knows the hustle required and sometimes you have to risk annoying people to prove there's value in what you're doing. There are many examples of this. Here's the most recent one I've seen: HotelTonight (https://www.youtube.com/watch?v=EER3rdWXbNk).
If you're relying heavily on your personal network I'd be worried that those 803 people were not true customers. Don't get me wrong, I think this is a great idea, but I'm wondering how you know that a 803 person sample size is big enough.
Also - could/would people click on more than one button once they found out they were non-functional? It could be that some features would be best developed in tandem with others.
We intentionally diversified the traffic as much as possible, exactly for this reason. Only a small portion of the traffic was from personal networks.
Repeat clicks are definitely interesting. It makes sense that once a button was clicked and found to be non-functional, that may prevent future interaction. But that first click still holds value. It'd be interesting to pull stats on how many unique visitors generated those click events.
Perhaps if we were accepting pre-orders or requiring a credit card. A soft launch like this would be more comparable to a focus group for a traditional product launch.
But you're not being honest and up-front about the fact that it's a focus group.
If it was like "Click here to take a survey about what features you most want in an email service" that would be one thing, but that's not the messaging in your case.
Just knowing that it's an experiment defeats the purpose. By getting honest responses and behaviors, it is easier to build a solution that people actually need or would actually use.
Demo means "demonstrator". Usually you use it to demonstrate (show) what your product can do. If it doesn't, you show them something that doesn't exist, and it's more of an illusionist show than a real demonstration.
- Just because some people clicked on features they want, it doesnt mean they will pay for it once you launch.
- Just because some people are using your free features, it doesnt mean they will pay for it once it is paid.
- Just because you have 5 paying customers, doesnt mean it will be easy to find out 500 more.
- Just because you have 500 paying customers, it doesnt mean what you did to acquire these 500 will scale to the next 5000.
- Just because you have 5000 paying customers, it doesnt mean that there won't be a new upstart which will dramatically undercut your offering and acquire most of these.
There is a name for this validation process - its called a 'startup'.
exactly. startups are essentially always validating, beginning with what sendwithus/others did to get a product and idea, and then going through the steps sonink listed. like sonink is saying, there are no guarantees with startups, so you might as well try to stack the deck in your favour right from the get-go.
Demo doesn't imply fake features. For me demo would be not my data, perhaps a sample set in there.
I'd have no problem if it said "there's a bunch of features not yet implemented, we have the buttons there to gauge interest and we'll work on the most needed ones".
"We knew that transactional email was a problem worth solving" A story worth telling in it's own right I believe, when clamoring on about lean methodologies. Nice article would love the pre-phase lean story leading up to this stage.
Developed it back when I was doing microsites, and I wasn't really sure what to do with it. My main question: why do people visit, anyway? I figured I'd know what to do with it once I figured out why people were there.
So I did just what these guys did. I created a list of buttons down the page with all the possible things people visiting a site called paycheck stub might want. Hooked them all into Google Analytics. Paycheck templates? Fake paychecks? Loans? Payroll services? A job? Sample paychecks for school or presentations?
Very interesting results. First, about 90% didn't want any of that. God knows why they were there. But out of the other 10%, I got a couple pieces of useful information. They were looking for payroll services. So I put up some banner ads. (I think I took down the Adsense. Can't remember)
That, in turn, provided more ad revenue than simply using Adsense on the site. It also provided me with a list of topics to develop and some search terms to try to hit.
So I developed those topics and tried to do better on those search terms. At one point I was almost ready to jump into online payroll. Fun times!
Then Google killed my search ranking -- I wasn't paying for links or spamming. Beats me why. I got so frustrated with them I gave up.
Still get quite a bit of traffic, though. But now, I'm sure, the traffic pattern is completely different. I wonder if, for static sites with a lot of content, this is something you should repeat every year or so?
Like these guys, I found it a rewarding experience, both for my understanding of the site and for my readers. The reason I'm posting is that many times we view this as a step on the way to creating SaaS, but it can be just a great way to connect with your readers. I bet it's extremely applicable for a lot of blogs.
Wish I could have figured out why those other 90% were there.
Perhaps the 90% were put off by the list of payroll-related things? It likely looked like a slightly more elaborate version of one of those parking sites, with the links to nowhere that are all vaguely related to the domain name (and tons of ads).
Your current site again looks like a spam site. It contains heavily SEOed text that looks like it came from a content farm, some ads, and many of the sort of links that a parking site contains; lots of internal links to more heavily SEOed text.
Basically, you tripped up against heuristics because you look almost exactly like the sort of spam that Google is trying to clear out of its indexes, whether you had good intentions or not.
I don't know, vertex. They were organic traffic, and as I recall they were coming for what seemed like relevant keywords. It's been a while, though.
My current site is a micro-site. I and my wife wrote the material. I hand-designed the layout, hand-coded all the html. As a programmer and a writer, I was pretty excited about the idea of microsites a few years ago. Here was a chance to combine my love of writing and programming to create online magazines around particular topics. I envisioned writing dedicated applications, doing some live updating of relevant information, and so on. It was (and is) a pretty cool idea: a spot between a blog and a dedicated topical site. As far as the guts of how the site works, using internal links, scoring highly on search terms, etc? Read that in an article somewhere. The goal is to create a small spider site around a topic such that people land on one page and see the top 3 or 4 related pages immediately, allowing them to browse tangentially if they're in the wrong spot. There's probably a better way to optimize around that than I did, but I never got around to it. And as far as Adsense, I added on ad unit to pay hosting costs. As I said, once I figured out why folks were visiting the idea was to get rid of it. But Google kept emailing and suggesting I bump it up to three units. Go figure.
And yep, Google killed me because everybody else in the microsite business used content farms, cranking out thousands of pages of crap, loading up with Adsense, flash ads, grabbing emails, and so on.
I still think the microsite idea is a great one, although, you are correct, anything that looks like one is going to get penalized. That's a shame. There's still a lot of potential there. Check out my microsite for books that hackers might like for an idea http://hn-books.com (This is also a good example of a totally-static site that still has a database and lots of interactivity)
In either case, Google's algorithm decided my couple of hundred hours of programming, design, and writing effort wasn't worth it. After horsing around with the idea for a year, losing valuable time, I gave up. As somebody who was not out to game the system, I was collateral damage in the bigger war against crappy content.
Hardest part was coming up with the questions. I still feel like I was missing something. Some important reason they were visiting.
But when you have a site with traffic, you end up with tons of data and very little insight. The button thing moves the needle a little bit, and that's better than nothing.
ADD: One other thing: there's also a minimum amount of traffic you need to make this thing work. 100 visits isn't enough. I'd imagine between 1-5K? Otherwise there's too much noise. Was that your experience as well?
> We knew that transactional email was a problem worth solving and we were determined to build a great solution.
I have a problem with this phrase. Seems to me it needs justification maybe just a verb? I don't immediately think of "transactional email" as a problem...
I did that with my father's computer store website back in 1998. We had computer descriptions, a price, and a 'buy online' button. The button just had a hit counter on the page, with directions to the store. Every hit meant someone wanted to buy a computer through our website and couldn't. I showed him the data that we could double our sales just by making an online store, but he was not convinced. In any case, the cash flows of that business wouldn't have worked out for him. Still, I thought this was a widely known technique.
From a user perspective, I think there are both good and bad ways how this technique can be used:
I actually like the way Sendwithus used it - as part of a very very early "pre-alpha" version where you're not yet interested in providing a reliant service and just want to get a feel for the demand. As a user you're informed early-on that you're looking at a version in development - so you will not be surprised by missing or incomplete features and will maybe even motivated to provide additional feedback.
The downside of this approach is of course that the data you're gathering is really only good for that - an early overview. Your test group will likely be very different from the actual people who will later use the service. The usage patterns will be as well.
The second way to use the technique is how Nick Kishfy describes it in the "button to nowhere" article. In that approach, "buttons to nowhere" are not just inserted into an early preview/development version, but into the actual product as a continuous part of the development process.
This would solve all of the above problems and would give you a much richer set of data over a much longer time span - but from a user perspective that would be the worst thing possible.
We knew that transactional email was a problem worth solving
Really? Hasn't this been solved already, by many businesses (SendGrid, Mandrill, SES, MailJet) – I'm happy you've managed to build something and are seeing some success, but I don't think you're solving a problem that hasn't yet found a solution.
Altavista worked ok, but I'm glad Google came along.
I think that your comment isn't a bad one, but should go more in depth. Some markets really are unassailable - others aren't. The reasons are many and sometimes complex, so analyzing one is often quite interesting.
I don't think unfunco is suggesting that the market is unassailable. The point seemed to be that this problem wasn't one that needed to be solved (because there were already solutions), not that there isn't room for another solution-provider.
What's the story with using logos for integration (SendGrid, AWS etc.) without actually supporting them? Is it not possible to get into legal issues there as it's quite an effective boost to the site to make it _look_ like you're associated with very reputable sites?
Or.. just something to worry about when people come knocking? :)
We are going through the same approach with our current beta customers (and will be making our demos available to a wide audience in the future too) and I can attest to the validity of this approach. We make it very clear to our customers that this is a demo and their changes will go away if they refresh the page. We don't have pop-ups integrated since we collect emails on another page. I think event tracking should be sufficient but we'll have to see what the data says in a little while.
We actually also use these semi-live demos instead of mock-ups (since we'll be implementing the same pages afterward anyway) since only one of us is proficient with Photoshop but everyone is comfortable with Bootstrap. It serves as a testing ground for various approaches for JS code too.
I'm wondering if the A/B testing score was skewed due to it being the first non-working link in the menu. Someone going through the menu to see what's possible will probably take a look at the top item first.
A random order for the menu items might have been better. Or maybe not.
Awesome way to design a product.
I wonder though, how the users reacted when they saw "dead end features". Were there any frustrated users when they saw it was mainly a "mock-up" web site ?
In a way that's a good thing - having a user work through the frustrating UI and still be interested in the product is amazing validation. Early adopters can self select.
If someone is willing to use it even when it's absurdly cumbersome then they are going to turn into customers. Once you build it you can iterate to find the best patterns.
I think it's natural to think that people will react this way. It turns out that people react completely differently. In my experience employing this tactic across several different verticals (SaaS, ecommerce, games), I have found people to react positively (yes, please make this!!) or neutral/mildly disappointed (darn, I wanted to use that). Never resentful.
If a good one exists already that is. And if that is the case why are they checking broken demos? Why are they given their email addresses to broken demos? Either way you slice it, if something like this gains interest it certainly illustrates opportunities.
Obviously smart and handy way of getting metrics for a wannabe startup but as some comments states I would feel cheated as no real product are offered, but offers are pretended to be real. Thus wasting my valuable time.
I bet this tactic will backfire soon as more and more try to do the same. I would be more positive if some incentive were offered, e.g. Thx for your interest get 3 month for free when we launch ...
Can't decide how I feel about this. Does anyone feel that this type of testing is kind of dishonest? On one hand, listing a product as available with a price is a great way to gauge interest. On the other hand, telling someone it is "sold out" when it was never available feels wrong somehow. What do you think?
Excellent question, and probably grounds for another blog post. We tried to use a variety of channels, including twitter, marketer communities, and even some google adwords.
At the time, the company I was working at was looking for a service exactly like this - a service where our app could just fling events, and then we'd do all the logic to control what emails got emitted on the basis of those events on the external service (and A/B test them). One of our "growth hacking" guys found SendWithUs somehow, and introduced us. We had a few calls with them - the service seemed like it had potential, and we got beta access to it...which was really sendwithus zero, but sending basic email.
We got most of the way though setting up an integration with them, but every time we asked when the A/B testing framework - which was what sold us on the product - was coming online, we were told "it's coming soon, it's what we're working on right now." So we held on for a few more weeks, and gave them some time...but nothing came of it. I was skeptical at the time that they were actually building it, even moreso having read this.
The positive side is that while we were being strung along by sendwithus, we found Vero (https://www.getvero.com/). They're not perfect either, but we've been on them for 6-8 months, and they've done well for us. They're responsive to our issues, are willing to Skype or chat when we try and develop some sophisticated campaigns on their platform, and have (or so it seems) rejigged their roadmap a few times to add in small features that we ask for. ExactTarget is still a more full-featured package, but I'm happy with the price/value of Vero.
So well done, guys. You've got an interesting story about how you bootstrapped your business, and you're evidently doing OK. I wish you the best, but I can tell you that you won't be getting my business (or my recommendation) - at the time I thought you were small and were actually on the cusp of delivering what you had promised us worked; now I see that we were just one more data point for a feature set that you never planned on having ready when you claimed.