If you're interested in learning more about how Veneur works and why we built it, I gave a talk at Monitorama last year that explains the philosophy behind Veneur. In short, a massive company like Google is able to build their on integrated observability stacks in-house, but almost any other smaller company is going to be relying on an array of open-source tools or third-party vendors for different parts of their observability tooling. When using different tools, there are always going to be gaps between them, which leads to incomplete instrumentation and awkward (inter-)operability. By taking control of the pipeline that processes the data, we're able to provide fully integrated views into different aspects of our observability data.
The Monitorama talk is a year old at this point, so it doesn't cover some of the newer things Veneur has helped us to accomplish, but the core philosophy hasn't changed. I've given updated versions of the talk more recently at CraftConf (in May) and DevOpsDaysMSP (last week), but neither of those videos are online yet.
 e.g. ELK/Papertrail/Splunk for logs, Graphite/Datadog/SignalFx for metrics, and maybe a third tool for tracing if you're lucky.
I'm not picking on this package. I see it all the time.
Can someone here explain to me what the use case is for this software?
It also has a separate, related purpose as a statsd protocol transport. You run it on the statsd port and it receives standard (or DataDog-extended) UDP traffic, then it forwards metrics via TCP to another backend. This has reliability benefits when operating over a network that might drop UDP packets, such as the public internet.
"If you monitor applications, Veneur can save you money and increase reliability"
Take a look at the software this "pipeline" is supposed to integrate with. It's a commercial product. Their landing page could not be clearer:
"Modern monitoring & analytics. See inside any stack, any app, at any scale, anywhere"
Perfect. Now I know what they do in plain English, how to mentally categorize their product, and even how to bookmark the page for the future when I might need their product.
Tells me an awful lot with only one jargon word: declarative. But I can still categorize it mentally and bookmark it even if I don't know what that words means.
Make it easy for people to understand what you do, and you'll get more interest. People will bother to bookmark it. And maybe come back.
When I run into one, I optimistically assume I'd know if I needed it (but, unfortunately, it means I'm very unlikely to remember a specific package if I do eventually run into the problem.
If they don't already exist, maybe there's room in the universe for some README best-practices (like those floating around for writing commit messages, user stories, issue reports, etc.) that might nudge more maintainers to include at least one lucid example of a problem it could solve.
In short, it's reasonable to expect that people who see a project already understand the problem space and write for the ones who can say "yep I need this".
This particular project is probably only useful to people who know what observability data means.
Here's the opportunity: I've been programming for decades. I'm always on the lookout for new and better solutions. I might not have a use for the software today, but if I can understand what it does and bookmark it, there is a good chance I will come back when I do need to solve that problem.
This means a lot to me and allows me to bookmark it both mentally and in my browser:
"This package helps reduce DataDog (<-- link) costs by aggregating data before sending and increases monitoring reliability by being distributed and fault tolerant."
Pretty much plain English, and now I can bookmark it because I know I will be evaluating my monitoring needs in a few months.
I read the entire README and had no idea what the software did. At first I thought it might have something to do with RxJS and observables...
> This particular project is probably only useful to people who know what observability data means.
Sorry if I'm repeating myself, but no, it's useful to other people. Like me. The software might be useful to me because I am interested in monitoring an app pretty soon. I've never heard someone call it "observability data".The Wikipedia article on observability left me even more confused.
This happens a lot, and it's a mistake that can usually be fixed, which is why I wanted to point it out.
If it can't be fixed and the software truly is extremely niche, then zero problem.
1. What problem does it solve
2. How it solves it
3. How is it better than the most common alternatives to solve the same problem
Not bothering with doing that is just OSS malpractice.
If you can't explain it simply, you don't understand it well enough.
- Distributed - Got this one, but it's an adjective and doesn't mean much on its own.
- Fault-tolerant - Same.
- Pipeline - OK, great, totally meaningless noun. I guess stuff goes in it?
- Observability data - What the heck is observability data?
Nginx is a web server, plus a heck of a lot more. Their website does a terrific job of explaining it. https://www.nginx.com/resources/glossary/nginx/
For example, _I_ know what observability data is, but I'd have a difficult time explaining the problem Redux tackles. If you've spent most of your time building user facing apps + web apps, how would you immediately understand a problem that someone working on large scale payment infrastructure has to solve?
Do you think you couldn't understand a short description of the problem it tackles? Because if you could, then a short description in a readme would be valuable to you, and presumably the developers of the thing are able to explain the problem.
> Can you explain what nginx does to someone who doesn't
> have a smart phone in any kind of meaningful way?
M3 aggregator, Uber’s metrics aggregation tier is similar, except it has inbuilt replication and leader election on top of etcd to avoid any SPOF during deployments, failed instances, etc. Also it uses Cormode-Muthukrishnan for estimating percentiles by default, it has support for T-Digest too. Although these days submitting histogram bucket aggregates all the way from the client to aggregator then to storage is more popular as you can estimate percentiles across more dimensions and time windows at query time quite cheaply. You need to choose your buckets carefully though.
It too is open source, but needs some help to make it plug into other stacks more easily:
I can't speak for the industry, but here's why I chose it for this project:
* I hadn't yet written anything in Go and wanted to try it on a side project / experiment
* I knew that my eventual deploy target (if the project turned out useful) would be lots of machines and I wanted to minimize the deployment requirements. Static binaries are good for that.
* I wanted to distribute the work across many cores and felt that Go's channels would make be a useful mechanism.
* I benchmarked my initial PoC against some other implementations (Python and JVM/Scala) and found no major reason to not use Go
The contributions of Stephen Jung (https://github.com/tummychow) and Aditya Mukerjee (https://github.com/chimeracoder) elevated it from a glimmer in my eye to system you can trust across your infrastructure.
So, in summary it was a confluence of interest and convenience with a strong hint of "if I use, this, it needs to be easy to deploy" and here we are 2 years later. :)
Its build system is poor if you want to rebuild the binary from the same
sources (you can't precompile the libraries used), and the statically linked
binary may be nice for a one-off deployment ("fire and forget" mode), but for
repeated deployments and multiple versions running at the same time it reminds
me the Chinese torture "death by a thousand cuts" in what you can't do with
such a binary (dozens of small things that are hard to remember, each on its
own not being enough to go away from static linking, but boy, they do add up).
And then there's also sharing memory between different processes that use the
same library. You don't have that for a statically compiled binary.
It's not something to regularly rely on, but something that helps in debugging
and troubleshooting. Not for a programmer, but for a sysadmin.
> Having shared objects on the machine actually makes this impossible because your tests are running with different libraries than in production.
In such a case you have your deployment process broken. And if your testing
and production environments differ in this matter, they differ enough bite
your ass even with your statically linked binary.
> Shared libraries on a machine are a non-hermetic input to your build and are to be avoided.
This is merely stating a generic opinion. I want to see a concrete, coherent,
technical argument supporting this.
> In addition, runtime shared objects (especially of something performance-critical like crypto) inhibit all of the most important compiler optimizations like inlining.
Especially crypto should not be called in a tight loop, but passed a large
chunk of data. Otherwise you inhibit all of the most important defence
against side channel attacks, and I guarantee that you are not competent
enough to defent against that on your own.
> As for ltrace, there's a million ways to trace function calls these days, like uprobe or perf.
So let's break one of them for no good reason?
And still, lack of any of the mentioned things is merely an annoyance once you
hit it, but as I said, they are numerous and add up, while the other option,
static linking, provides little benefit apart from supporting broken workflows
(like different environment in testing and production).
I didn't imply that it shouldn't be.
I am genuinely interested to know the thought process and why it seems to be the defacto language these days.
Your arguments are strong, thanks.
In addition to Intercom (mentioned in a sibling comment), some other companies off the top of my head who are also using Veneur: Sentry, Bluecore, Axiom, Quantcast, and at least two more that I'm not sure if I have permission to name publicly.
There are more too - since Veneur is an OSS project, we only find out when people submit submit PRs to the project or happen to contact us for another reason. It makes for a pleasant surprise when we find out!
I consider it absolutely essential for collecting metrics on large instances that run hundreds of busy tasks emitting hundreds or thousands of metrics per second.
Source: am engineer at Intercom.
This is awesome to hear! Are you referring to this Intercom? https://www.intercom.com/
2016 was the first public release, but the project has grown a lot in that time. You can take a look at the changelog to see what's new, ever since we switched to a six-week release cycle last year: https://github.com/stripe/veneur/blob/master/CHANGELOG.md
Source: I work on the Observability team at Stripe and I am the PM for Veneur.
Is this a fancy way of saying "privacy-sensitive user data"?