
Update on InfluxDB Clustering, High Availability and Monetization - KyleBrandt
https://influxdata.com/blog/update-on-influxdb-clustering-high-availability-and-monetization/
======
23david
I've advocated for and implemented several InfluxDB installations in
production over the last year+, and one of the considerations was always that
non-alpha (prod-ready) clustering was always promised in the 'next version'
that was just around the corner.

Several months ago it seemed clear that the team was overly optimistic, and
it's just disappointing to see that now the clustering will be available only
in a paid (minimum $400!) option or on their hosted service.

I understand the business considerations here, but it feels like a bait n'
switch for all the people who evaluated/used InfluxDB in single-node operation
as a temporary measure while giving the team ample time to work out the
clustering kinks.

Lesson learned I guess... but dang what an expensive lesson.

~~~
infinotize
I agree, I've invested a bit of time building a monitoring stack with influx,
and one of the main reasons I chose it was development velocity vs some of the
competition. And from a while back, clustering was supposed to be a top-level
issue, coming just around the corner. At this point I'm going to have to bite
the bullet and go with a more cumbersome but mature solution, like opentsdb.

------
jrv
This makes me think: an open-source project can be better off if it's _not_
controlled by any one company. While in Prometheus
([http://prometheus.io](http://prometheus.io)), we might still take a while
before we have a clustered remote long-term storage, we'd never _prevent_ it
because the project is independent of any company and we'd want the open-
source project to be as good as it can be.

Also, there were some tentative thoughts about using InfluxDB as the main
long-term storage backend for Prometheus, but that has become pretty much
uninteresting now that clustering support (needed for LTS and durability) is
basically cancelled for the open-source version.

Still, I guess I can understand that when you're a company, you need to focus
on making money.

~~~
mindcrime
This is concerning to me, not because I use InfluxDB, but because my company
is also a "pure play" OSS vendor. So it's always concerning to see a company
fail to achieve success with the pure-play model, and wind up having to go
"open core".

 _Also, there were some tentative thoughts about using InfluxDB as the main
long-term storage backend for Prometheus, but that has become pretty much
uninteresting now that clustering support (needed for LTS and durability) is
basically cancelled for the open-source version._

True, but consider this: Given that InfluxDB (core) is Open Source, there's
nothing to stop somebody else from coming along and building a version with
clustering support. Whether that will happen or not is, of course, an open
question.

~~~
bbrazil
_There 's nothing to stop somebody else from coming along and building a
version with clustering support. Whether that will happen or not is, of
course, an open question._

Clustering is really really hard. My personal experience with that class of
problem is that it took 3 years to get it production ready, and that was for a
simpler problem than what needs to be solved here.

The chances of a 3rd party implementing this given that InfluxData plan to
have a commercial version seem very low. I'd expect that most companies who
want to stay open source will treat this as a standard database that needs
client-side sharding and HA (which is the recommend approach - see
[https://influxdata.com/high-availability/](https://influxdata.com/high-
availability/)).

------
KyleBrandt
I'm interested in what the pricing will be like at various scales, on Twitter
the CEO (Paul Dix) indicated $400 month will be the basic offering for limited
cores.

I guess time will tell, but a bigger deal than the money to me is will the
clustering actually work well? Clustered databases is an extremely difficult
problem. Seen it get better in the years in things like MsSQL, but even with
that sort of resources it took a long time (years) for the newer availability
model to become stable.

At Stack Overflow we are still using OpenTSBD behind Bosun. But HBase sucks to
manage if you don't have any other reason to be using the technology. So I see
a lot of users interested in using InfluxDB as the backend (and some do, Bosun
can talk to it) because it is easier to get started. But if you know you will
have to scale up eventually, all the options right now are not appealing in
TSDB land :-/ So if some $$ really does get a good TSDB that scales and
reasonable to manage then great, but I'm skeptical.

~~~
pauldix
We're committed to making the clustering a world class product. You're right,
it'll take time. However, we'll get there. We'll put the same focused effort
and testing into it over the coming months and years that we put into the TSM
storage engine, which is working remarkably well at significant loads.

In the meantime, the stuff we have going into the open source is going to make
the standalone server (and thus anyone using open source options for HA or
clustering) even more performant, stable, and scalable.

~~~
jrv
_...anyone using open source options for HA or clustering..._

What open-source option for clustering would this be if the main InfluxDB
open-source version won't have clustering anymore? A fork or something built
on top of multiple standalone servers somehow?

~~~
KyleBrandt
I think maybe the other HA options, such as using a relay or sharding as
stated in [https://influxdata.com/high-
availability/](https://influxdata.com/high-availability/)

~~~
jrv
That sounds plausible, thanks!

------
noir-york
This basically kills Influxdb for me. We're evaluating influxdb in stand-alone
mode to collect limited metrics (grafana) with a view to eventually moving
more and more data to Influxdb once clustering became available. $399 per
month is ridiculous. Clustering is table stakes.

Lesson learnt: _before_ deploying a new OSS, check if they have a _credible_
plan to support the project. Otherwise skip.

~~~
akanet
I mean, honestly, $400/mo is table stakes too, if you're actually using a
clustered database in production for anything serious.

I think HN in general is being a bit too cynical here. This is an OSS company
trying to find a sustainable way to survive. There are going to be some tricky
tradeoffs. "They should just charge $20/mo for us small guys!" is not a
reasonable course of action.

Regardless of whether this particular pricing plan works out or not, I think
it's a good idea for programmers to support other programmers trying to make
money on OSS. We are all not served very well by the general attitude of "I
heard about this technology last year, it should be $0 now".

~~~
noir-york
How much do you pay per month for: Apache, Nginx, Varnish, Redis, memcached,
compilers, interpreters, Nodejs, Cassandra, Linux, Postgresql?

We used to pay for this stuff - remember Sun? Things change.

I suspect what happened with InfluxData is that being venture funded, they see
that the funding environment has changed and are racing towards revenue/cash-
flow positive. And clustering as a premium feature is, apparently, the answer.

If clustering had been pegged as a premium feature _up front_ then neither I,
nor I suspect would anyone else, have a problem. But it wasn't, hence the sour
taste.

~~~
anarazel
Yea, but even free software maintainers need money from somewhere. Especially
if you want $software to continue improving.

It's fantastic that free software has increased so much in market share. I
just wish that more companies would recognize that that only works if they
actually pay for development of some projects in some way. That can be
directly employing maintainers, contributing patches, sponsoring feature
development.

Since that often doesn't suffice you sometimes end up with companies trying to
make ends mean with "open-core" type models; even if they'd otherwise prefer
not to go down that road.

------
ericb
Ugh, we are a startup, we invested in influxdb on faith that it would "get
there", developed around it, and now we can't afford what they want to charge
for the scalable version.

Maybe we can avoid clustering with some workaround, (sharding) but I feel
tricked.

~~~
pauldix
I'm sorry that you feel tricked. I hope that one of the open source options
handles your use case if commercial software doesn't work for you.

However, we think that the community will be better off if we can continue to
contribute to the open source InfluxDB. We want to invest heavily in open and
closed source software. Ultimately the open source community will benefit from
an open InfluxDB

~~~
ericb
A pricing model that scaled with usage and was more inexpensive at the low-end
would, in my opinion, better serve your company and customers. That way you
pull more people into the product and increase the overall size of the pie.

------
gtirloni
People will find increasingly clever ways to work around the lack of
clustering, as they always did. This could mean only the top-tier users will
be paying, which must be a very small portion of the pie.

Even though this announcement is sold as something that will empower
InfluxData to add even more cool features to the open source version, I'm
actually worried that it won't be around as an open source product for very
long. Let me explain.

InfluxDB is a very fine product but I don't think it has all the momentum it
needs to keep going in the long term yet. That means it's still building that
critical mass of supporters in an open source ecosystem. With this
announcement, InfluxData is eroding the trust that small to medium-sized users
had in it so the momentum slows down. If InfluxDB was a no-brainer for anyone
starting a time-series project, it is not anymore. All the other alternatives
need to be carefully analyzed and, even if InfluxDB is chosen, there is that
voice in the back of your head saying you'll be in trouble if you exceed a
single node's capacity. Eventually people will jump to the next TSDB solution
that offers clustering as soon as it becomes avaiable. Then where are all
those paying customers going? It'll then be easier and cheaper for InfluxDB to
become a proprietary software company.

While there are some comparisons being made to what Nginx does with its Nginx
Plus offering, I think it's the opposite situation. IIRC, since the early
days, Nginx has offered a paid product with more features. And recently they
have started to add those features to the open source version, so people got
happier. InfluxDB has always promised clustering (it was a major selling
point, go watch any presentation about it from 2 years ago), shipped the code
(even if it's half-working as of now) and now announced it's removing the
functionality. Suddenly any InfluxDB node you have (even if it's running just
fine without clustering) looks like a lemon.

It's all such bad PR. And it's 2016, haven't hundreds of OSS companies been
through this already? Oh well.

------
damm
I really don't see a point to worry about. So as of 0.12 the OSS product will
change; however a new product will come out to bring in replication. Somehow
they will produce a private binary that will do the same thing

Single node usage is great for most people; sure it can be useful to cluster
and you have that option.

If there's limitations it's open source; those can be removed. Then people
would likely use the fork that doesn't have that limitation.

I don't see evil here. Just trying to work on their product

~~~
jrv
I don't think anyone is saying anything about it being evil, and it may even
make sense for InfluxData the company. But for users, not having clustering
(for scalability, long-term storage, and durability) will limit the value of
the open-source version a lot in the long-term. As for the viability of a
fork, see the other comments around that.

------
kev009
"less than ½ of a percent are running active clusters." because it doesn't
work. They have been TheatricDB to me for a long time, this is just another
nail in the coffin.

------
sp1982
We use blueflood developed by Rackspace at Square which uses cassandra ,
coupled with a query layer called MQE
([https://github.com/square/metrics](https://github.com/square/metrics)).
While it's actively developed, I definitely recommend taking a look if you are
interested in highly scalable metrics system with decent strategy for rollups.
(100k+ metrics/sec).

~~~
pauldix
The reason we think InfluxDB is compelling as a standalone server is that you
can get 300k metrics/sec on a single server. The scale is there for many use
cases in the current release

------
hoov
I'm pretty damn angry right now, but I'm sure that in a few days I'll get over
it.

I run the tech side of a decent sized startup right now. Having joined ~4
years into the venture, one of my immediate concerns was the lack of
visibility into how our system works. Keeping tabs on the performance of
several hundred ETL pipelines is not super easy. I decided to double down on
InfluxDB, changing the road-map of my infrastructure team.

Then, InfluxData introduced the TICK stack. I can get monitoring and alerting
as well? Let's double down again!

I bought hardware, set up a cluster (painful), we filed bug reports, learned
what sorts of queries not to run, and we were in good shape. I purchased
tickets for a training session and a flight (out of pocket; we don't have
travel budget). And then I saw the blog post last night.

All along, I knew that the promise of all of this great technology for free
was too good to be true. There was always a little voice in my head asking me
about how they actually made money -- I knew that a paid offering +
professional services was not sustainable.

I'm angry, but we'll be fine. We're going to break apart our cluster and
shard. We've already got code written to do a backup/restore that actually
works (slowly) from one cluster to another. If we hit the point where sharding
doesn't work, then it'll also probably be financially viable to pay the
$399/month (on top of server costs). I'll still go to the training, but I'm
not sure what I'll get out of it. The reason why I was going to the training
was the section on "Cluster Administration".

The only thing that I'm raw about is the lack of apology. You have to make
money, and you had to do a bait-and-switch as a result. I totally understand,
but the tone of that blog post was too defensive and unapologetic.

------
tlipcon
Hopefully this isn't too "pitch"-y, but: if you're looking for a database
that's good at time series, will always be open source, and does support
scale-out and HA, you might be interested in Apache Kudu (incubating).

Feel free to drop by our Slack ([http://getkudu-
slack.herokuapp.com](http://getkudu-slack.herokuapp.com) ) if you have any
questions.

------
terom
Congrats on the decision. Standalone InfluxDB can be scaled up just fine to
meet most usecases, and it's better to have the long-term project
sustainability that a hosted/enterprise offering can bring to the standalone
offering.

InfluxDB 0.9 still had plenty of bugs, and I'd rather see a high-quality
standalone server than any not-quite-there-yet clustered version.

~~~
bogomipz
Agreed I would have like to have seen these bugs ironed out before the whole
splashy rebranding to Influx Data or whatever they are calling themselves now
and their business pivot to a "data platform." I think the timing is
unfortunate.

------
spotman
> ... "customers eventually drop support as their infrastructures mature and
> they look to reduce operating costs"

This is part of the game called software. In some worlds this is actually the
goal; that the software works so well and is so reliable that your customers
eventually don't all need to keep paying you.

So you find new customers and add new, never advertised before features and
enterprise clients with SLA support contracts, etc.

It's very understandable that it's hard to monetize, but giving people the
impression clustering was going to be included long term and taking it away is
not going to score you points.

Furthermore you want to get all the folks doing a startup on a budget hearing
a story that works for them. Most ( especially new or young ) cofounder
programmer types rarely plan to not need clustering, even if the reality is
most won't need or use it. Saying this is for the big kids only is a turn off
at this point in the influx story.

Wish you the best of luck

------
troyk
I understand the need for businesses to make money, but I do not understand
how a business can understand OSS and implement the OSS core-only model. If
your product has demand, at some point, an OSS project will arise to displace
you.

Are there many winners with this model to name. Maybe NGINX, MongoDB?

~~~
seiji
_I do not understand how a business can understand OSS and implement the OSS
core-only model_

Whenever you see "open core," you should read it as "VCs said we need to do
this to keep them happy."

VCs love the _idea_ of how you can give _part_ of something away for free then
lock people in to auto-renewing subscription contracts for the whole banana
(with fees increasing 7% to 17% every year for ongoing "maintenance"). Many
VCs still actively recoil at the thought of giving away _everything_ for free
then just hoping the best will happen.

Nginx is a weird example because I'm not sure how well they are doing or who
would buy "nginx pro" instead of haproxy+nginx or haproxy+varnish+nginx some
other combination of already existing stuff.

MongoDB is an interesting example because they went with the "provide open
software, but with really serious, game-stopping flaws, then force people to
pay for support because nothing works" route. That's almost literally bait-
and-switch (bait-and-support?). But, that hasn't ended up so well for them.
From a recent WSJ article: _Fidelity has cut its valuation of MongoDB in eight
of the nine quarters since Fidelity made its investment in December 2013,
valuing the shares 58% below what it paid._

~~~
bogomipz
Isn't this what Elasticsearch is doing now? Security is deliberately left out
of the offering unless you purchase a very expensive support contract that
give you access to Shield? I'm not sure I would call them a winner, its
unclear how well they are doing. I think they are mostly running on VC money
at the moment. I agree "Open Core" should be regarded with some suspicion.

Do you have a link to the MongoDB WSJ article? They are pretty terrible, even
with a support contract bugs have a way of not being resolved or addressed.

------
rodionos
Axibase Time Series Database is free for pseudo-cluster installations and
ships with SQL and visualization included [0].

We also have a fully functional Grafana driver in case the customer prefers it
over programmable visualization that we ship.

By programmable visualization I mean a way of building dashboards using toml-
flavored configuration [1] and templating language, as opposed to manual
design.

[0] [https://axibase.com/products/axibase-time-series-
database/](https://axibase.com/products/axibase-time-series-database/) [1]
[https://apps.axibase.com/chartlab/2ef08f32](https://apps.axibase.com/chartlab/2ef08f32)

------
fangjin
Druid ([http://druid.io/druid-powered.html](http://druid.io/druid-
powered.html)) is another option for similar workloads. Druid is a community-
led open source data store used by many technology companies at very large
scale. Comes with multiple visualization/open source applications, SQL
interfaces, Grafana extensions, and a community to help with issues.

------
bussetta
it appears that paul dix deleted his answer in this stackoverflow question
[http://web.archive.org/web/20150416175827/http://stackoverfl...](http://web.archive.org/web/20150416175827/http://stackoverflow.com/questions/25540722/how-
to-make-a-choice-between-opentsdb-and-influxdb-or-other-tsds)

------
icbm504
It seems like there are 2 OSS models other there: 1) community supported; 2)
company led.

Most of the "community supported" projects that I have used are libraries.
Since shifting from developer to the devops team, nearly everything that I
rely on is "company led": saltstack, docker, grafana, influxdb, ELK, ngnix,
sensu, rabbitmq, etc.

That being said, we are now re-evaluating the use of influxdb.

------
siscia
An interesting way to monetize software could be to have people pay to access
official docker images.

You want to pull the latest images an unlimited number of times ? Sure, it is
just XXX$/month

Nothing will stop people to build open source containers of the same product,
but those won't be "official" and there won't be any update guarantees...

------
reubensutton
I wonder if clustering will be available in the cloud service?

We use a single server on Influx's cloud service for some internal metrics and
it has been running without any interruptions for >6mo.

~~~
pauldix
It will be available in the cloud service in the next few months. But if you
want to continue running single server that's ok too

------
shaneduan
Hello, I was directed here through the mentioning of blueflood. I hope it is
ok to share some product information. Please let me know if there is a more
appropriate way to do it. My only goal is to offer solution to solve the pain
we run into frequently ourselves.

Rackspace Metrics (the product team behind blueflood) is currently doing many
times more than 300 metrics per second. Our goal of this year is to reach to a
new scale from where we are at, driven from the needs of big internal adoption
this year. So blueflood project would be live and kicking.

Here is the most recent update on Rackspace Metrics product. All the
functional work are through blueflood: [http://bit.ly/rax-metrics-
mvp](http://bit.ly/rax-metrics-mvp)

Also, there is a hidden gem in monitoring unknown for most of the startups. My
other product, Rackspace Monitoring, is a SaaS product designed to assist
Rackspace business instead of making big bucks. If you have a Rackspace cloud
account, which requires minimum $50 service charge, you will be able to use
Rackspace Monitoring for free. You can use it to monitor any other servers you
host anywhere. This means that if your monitoring budget is already above $50
(all cost considered), you really should look into Rackspace Monitoring.

To learn about Rackspace Monitoring, please go here:
[https://www.rackspace.com/cloud/monitoring/](https://www.rackspace.com/cloud/monitoring/)

Please also feel free to reach out to me. We are located at the same location
of Rackspace Startup Program in San Francisco. We talk to startups all the
time and always enjoy the learning.

