You caught us by surprise. As you can see we've built in the ability to gracefully handle a lot of traffic but it's not ideal because you need to wait in a queue until we have enough capacity. So I wanted to jump on here and give you an update of a release we're putting out next week.
FastOrSlow does actual browser simulations from around 12 locations around the world. When we first architected it, we wanted to provide as many locations as possible including places like South Africa. That meant in some cases we needed actual bare-metal servers. So we built it on bare metal in data centers around the world.
We experienced some moderate popularity about a month ago and realized we needed to do a better job of scaling on demand. So we re-evaluated moving on to AWS. Around the same time AWS launched in Cape Town and a few other locations, and so we were able to get the kind of coverage we wanted globally.
So we kicked off an AWS migration. We're days away from moving into AWS and using spot instances to ramp up and down fast based on demand. But as I said, you caught us by surprise.
So please give FastOrSlow a try today, but know that by around late next week we'll be doing a much better job of handling your requests FAST without having to wait in line.
Thanks for your interest and patience. Myself and the lead architect, Ryan, will be here to answer any questions and of course listen to your suggestions.
Second, though—this seems really expensive to run, based on what you're saying here. How are you monetizing or otherwise funding it?
Thanks!! I run Wordfence which does quite well. Profitable and the team is around 35 really great folks with a focus on solid engineering. So we fund it ourselves. For a going concern like us the costs are very modest and we can afford to run it indefinitely without charging.
In fact just the bit of brand awareness we get for our main product, Wordfence probably more than justifies the costs.
We actually created this as an internal tool because there wasn't anything that would give you a clear indication of site performance from DNS lookup to connect to TTFB to page render from around the world. Our own market is global and even a 1% drop in conversion really costs. After building it we realized that it's very little incremental effort to make it available for all.
As I mentioned we're about to migrate into AWS where we run much of our other infrastructure. The financial model and eng design we came up with keeps our costs surprisingly low. We can handle huge traffic spikes and it roughly doubled the cost, even with the spikes. So I'm confident we can run this for the long haul as a free product.
A few things I can commit to: Firstly, it's extremely unlikely that you'll see ads on FastOrSlow other than our own company name. That's never been our model and I can't stand when a great site or service gets smothered in ads.
In addition, our company has been around since 2006 and we've been made acquisition offers many many times by large companies that you know well. We've consistently turned them down and will continue to do so because we are profitable, love our independence and I think we all know what happens to many great products after acquisition. I'm telling you this because I want you to know that this won't be changing hands to a company that will try to monetize it in obnoxious ways.
No plans to offer a paid service right now. Our next release is for internal use and will allow us to do performance surveys across large numbers of sites and present statistically significant insights as research to the online community.
Happy to answer any other questions. Thanks for your interest!
In the world of Wordpress plugins it is a gem in the rough.
Out of curiosity, did you look into any paid services like SpeedCurve or Calibre? There are also OSS projects like SiteSpeed or WebPageTest.
Full disclosure: I work at SpeedCurve. We've had customers cancel their accounts in the past because they wanted to build their own performance monitoring tool. To the best of my knowledge, most of those customers have come back because building and maintaining a tool like that is no easy task.
...Which I guess is a strange way of saying congratulations on building a performance monitoring tool!
You mentioned that Wordfence has a team of 35 behind it. This seems like a large team, and I'm curious to know what the distribution of roles is like? Also interested to know how many users and paying customers Wordfence has?
I'm a dev (ceo these days) and I wrote the first version of Wordfence. Also worked at many big and small companies in South Africa, London and USA. We're US based now. And so we're both engineering focused, and our process is to only ship rock solid code via excellent QA.
Funny thing. Finding vulnerabilities is actually just QA. And a vuln is just a celebrity bug. So this culture really helps foster great security research too.
Another big area we invest in is customer service. Our CS team does a great job and we also focus substantial energy on community support in free forums, which really helps foster goodwill from the community and build a relationship with folks who use our free product, many of whom become paying customers.
I started my career in ops in the 90s before moving to dev and am passionate about high performance (as you can tell from fastorslow) - and so we spend a lot of energy and budget on solid operations to support our products.
Kerry is kind of a math guru and is our COO and CFO these days. So we run a tight ship in terms of financials, which is pretty much why we're still around. Cashflow plan updated daily. Awesome projections. Awesome granularity. Sorry if I'm over-sharing, but I love entrepreneurship and perhaps this will give other folks starting out some insight.
We have around 3 to 5 million free customers. Can't share how many paying, but it's a lot.
I haven't coded for some time now. In 2015 I completely handed off coding on Wordfence to a brilliant developer named Matt who is still with us. I've moved my focus 100% to leadership, strategy and somewhat of a CTO role, but I share that with the strong tech leads in our engineering team.
Hope that helps. We're a private company, so I couldn't just dump our internal data, but I hope that gives you some flavor of who we are, our focus, team distribution and culture.
The overview was insightful, to see what makes up a well-run operation: how the engineering, tight financial management, quality assurance, performance, security, customer support and fostering the user community - all tie together to make a solid, growing business.
It definitely earns trust and respect for the company's culture and decisions. As a dev learning the ropes in a CTO role, this kind of open communication about the internal workings of a successful company and product is great food for thought.
To decisively not seek acquisitions and venture capital, and instead value independence, profitability, good finances, engineering, QA.. This will stay in my mind as a role model.
If there's an opportunity where I work, I'll likely recommend WordFence or other services by the company. (In fact, performance profiling of large/heavy sites has come up a number of times this year.)
Elsewhere a possible blog post on migrating to AWS was mentioned. I love that style of articles, describing the thought process that goes into technical decisions, how it was planned and implemented. Hope I can catch it here if/when it's published.
Congrats on the success!
A slight tweak to your site allow you to save cash and handle surge traffic with your current server.
1. Returning cached results for sites that's have been recently checked.
2. Collect email addresses and mail the result when it's ready.
It's a much better experience than leaving the tab open, frantically refreshing the page to see if the queue is reducing.
Or maybe the page I want will already be there!
If you get a 502 or if it takes a few seconds to load, we're busy adding another DB reader, so just reload and it will appear. Should stabilize as the new DB reader comes online. HN is slamming us right now.
Ryan built this thing.
Is there any place we can sign up for updates on when the AWS migration is up and running?
FastOrSlow is built using Lighthouse. :) We do actual browser simulations using headless Chrome and lighthouse to get the performance data. We do it programmatically from all the locations, bring the data back and build the report.
Lighthouse returned error: ERRORED_DOCUMENT_REQUEST. Lighthouse was unable to reliably load the page you requested. Make sure you are testing the correct URL and that the server is properly responding to all requests. (Status code: 429)
Also: If you type "Google" into Google - you can break the internet. Nobody has ever tried, not even for a joke.
I have to imagine your bandwidth costs from AWS aren't gonna be cheap.
It'll be based on sustained queue length averages rather than any kind of load number. We can also bump it up manually.
Our estimates for some fairly severe traffic based on historic data are quite reasonable for bandwidth costs. But we're a medium sized business, so we have revenue to support that. I think individuals and very small teams can get really surprised by this.
If you're just starting out and don't need dynamic scaling and the tooling that AWS provides, I'd seriously look into colo so that you can buy bandwidth using 95th percentile billing. We use a combination of servers that we own in data centers using 95th billing and AWS. Depends on use case and we have gotten very good at financially modeling future costs for a given config.
Remindme! in a week ;)
Mozilla has a nice one as well: https://observatory.mozilla.org/
Edit: From our eng team:
"We're moving about 5000 events per second now. Definitely shrinking now, and memory usage is dropping too."
Should be good to go now. Sorry, but massive traffic right now.
Wondering if you've had any pain points integrating Lighthouse that you wouldn't mind sharing. Anything more difficult than it should be, or were the docs lacking in some way, etc.
Should be good to go now.
A checkbox to receive marketing emails as well solves the issue (and yes, I do sometimes click I want to be mailed news and marketing).
The reason I ask is that I did a report last night, and checked it on my phone in the early morning. I put my site's address in again from my desktop now, and I see that I am 1,000+ in line. Even though a recent report was made for this site already, I can't seem to access it.
In my case it actually was sort of interesting - it definitely suggested that the redirects don't have the correct caching headers, because they clearly weren't being served off the CDN. However, this redirect is a pretty unusual case for us, because most of our traffic is either from SEO or links from notifications we send users.
I agree that you should be looking at the experience most users are going to have, but if the first experience for most users is a redirect you should probably do something about that.
I am going to look into getting those redirects cached though, there's no reason we shouldn't be able to serve them off the edge.
It was AHA moment for me when i looked at "Geography" and "Location" section. Pretty cool stuff.
Should be fixed.
WPT is mainly a supertool—here's a ton of data, go dig into the waterfall and network requests with some visualizations. I think there's a page for recommendations but to be honest I've never used it nor seen anyone talk about it.
With Lighthouse's performance report, we aim to provide specific actionable advice with estimates for how impactful it will be.