So it's not so much "a free splunk alternative" as it is "a system with slightly different free tier than splunk"?
> Splunk® Free (...) Scale up to 500 MB data per day
You can grab a 10GB/day Dev license for Splunk for free. IANAL, but should be suitable for home use. Lasts a few months, and renewals are free.
Unfortunately Splunk's installation process is much more streamlined and junior-sysadmin-friendly than ELK, at least in our limited view.
I need to look again at ELK to see if someone has created or improved the log input UI and indexing UI. Just something that lets you select log locations and types on a client in a web UI "somewhere", and something similar on the server to perform various operations for indexes and other such stuff.
Is that what this is? Where’s the source? How do you limit the ingestion capacity of an open source project?
If the core isn’t open source, that’s fine! But that makes “Community Edition” a misleading name.
So for example, if someone states C# cannot be used for writing an OS, then I can point to Midori.
And even then, they might still dismiss them even if proven wrong.
How embarrassing, it's me! :)
Splunk on the other hand has dashboard examples on almost every page (on their homepage, a carousel with five different examples of the things they purport to solve).
(another blog post does a great job with showing off capabilities: https://www.gravwell.io/blog/gravwell-and-collectd )
The last time I put up with this kind of thing I was on an hour-long call with a HashiCorp sales guy who didn't have the answers to any questions but did want to tell me that support would be $25,000/year for 9x5 access (lol).
Some people use it more like a black box and don't issue many searches so a single node with a shitload of storage is just fine. Others rely on active searching to monitor security incidents and KPIs so they want responsiveness and immediate insights.
I typically use AWS's built-in tools, but I've been looking for something for home, so I'll be checking this out. Thanks.
I wanted to give this a chance but I see we're up from Splunk's 500MB to a mere 2GB, and I have to learn and deploy a completely new product and train everybody else on how to use it as well.
I've got a team of users who are familiar with Splunk, have written code and parsers for Splunk and simply want to use Splunk. I can't get the funding for it though. If Splunk raises their cap to match yours, what is Gravwell's advantage (besides presumably licensing costs)?
We intend Gravwell Community Edition as a way for home users to experiment on their own network, or to try things out at work to decide if they want a full license. We of course also do unrestricted evaluation licenses for interested parties.
I'm particularly fond of this article because it showcases the Turing-complete scripting interface which I built. The blog post shows how to set up a script which runs on a schedule and emails you if your disks get too full.
We'll be posting more articles over the coming days.
1. Your docs site stays as a blank white screen for me.
Some chrome extension is interfering, but uBlock Origin isn't flagging it as blocking anything and the site stays white even with it disabled. It displays in Chrome Incognito mode (Windows 10)
2. Is there a query language, or is this not designed to ingest textual/json logs? A let-down with a number of services is how opaque querying is (Scalyr for example). Examples would be good.
Is Gravwell as extensible, and, if so, is it easier to work with?
The core of Gravwell is the search. You can run searches in the web GUI, or in the CLI client. You can schedule searches to run at certain times (specified with a cron spec, currently) so you could have search results ready first-thing every morning.
More powerfully, you can write scripts to run searches. These scripts can be run on a schedule, or you can run them by hand using the CLI client. Check out our other blog post https://www.gravwell.io/blog/gravwell-and-collectd for an example script that runs a search on disk stats entries from collectd, checks if any machine is running out of disk space, and emails someone if there's a problem. Of course, scripts can be complex to write, so we're exploring options for simpler flowchart-like scripting within the GUI too.
You can also run scripts within the pipeline, which we frequently do when existing search modules don't quite meet our needs.
We've open-sourced the library that lets you ingest data, so you could pretty quickly ingest anything you want. Unlike Splunk, you don't have to massage every data source into a key-value sort of text format; we'll just take binary if you like, we don't care.
Gravwell has a REST API, so you could also interface with it that way. We'll probably open-source our Go client library at some point, but we want to clean it up a little first and make things more idiomatic in places.
I hope that answers your question a little? All the functionality I mentioned here is included in the Community Edition, of course!
To your second question, our entire system is built around processing local, copying is a HUGE no no in the platform until you absolutely have to. We're VERY happy with the performance we're getting out of this sucker. That's one of the primary reasons we were dumb enough to start from scratch on the storage and search architecture. We're just glad it paid off.
Edit: we're putting screenshots into the blog post now
2. Our open-source components (github.com/gravwell) are written in Go
I would absolutely love to see a competitor emerge that addressed the migration problem through a compatible search api. Handling other timeseries data like metrics would just be icing on the cake.
Interestingly, you don't need a special language. The relational calculus is isomorphic to the regularity calculus, which, in practical terms, means that SQL is a perfectly good language for Splunk's use case.
Splunk is far from the only way to do what it does.
I'm fortunate to work for a company that invests 8 digits per year in their Splunk infrastructure, it's a travesty that I don't leverage more of its capability.
I tried installing Gravwell from the Debian repo. This unfortunately seems broken.
W: Failed to fetch http://update.gravwell.io/debian/dists/community/InRelease Unable to find expected entry 'main/binary-i386/Packages' in Release file (Wrong sources.list entry or malformed file)
Once installed from the tarball, it failed to start.
The gravwell_webserver process is not running!
If you kept an old configuration file the configuration parameters may have changed. Try manually starting the services and looking for errors.
$ vim /opt/gravwell/etc/gravwell.conf
$ systemd start gravwell_webserver
$ systemd status gravwell_webserver
gravwell_webserver.service - Gravwell Webserver Service
Loaded: loaded (/etc/systemd/system/gravwell_webserver.service; enabled)
Active: failed (Result: start-limit) since Thu 2018-07-12 02:15:07 UTC; 23min ago
Process: 3411 ExecStart=/opt/gravwell/bin/gravwell_webserver -stderr %n (code=exited, status=255)
$ journalctl -u gravwell_webserver
-- Logs begin at Fri 2018-07-06 20:35:20 UTC, end at Thu 2018-07-12 02:41:26 UTC. --
$ ls -la /opt/gravwell/logs/web
drwxr-x--- 2 gravwell gravwell 4096 Jul 12 02:07 .
drwxr-x--- 4 gravwell gravwell 4096 Jul 12 02:17 ..
$ ls -la crash
drwxr-x--- 2 gravwell gravwell 4096 Jul 12 02:17 .
drwxr-x--- 4 gravwell gravwell 4096 Jul 12 02:17 ..
-rw-r--r-- 1 root root 322 Jul 12 02:07 gravwell_webserver.service_2018-07-12T02:07:36Z.log
-rw-r--r-- 1 root root 354 Jul 12 02:15 gravwell_webserver.service_2018-07-12T02:15:07Z.log
$ less gravwell_webserver.service_2018-07-12T02\:15\:07Z.log
API Version 0.1
Build Date 2018-Jul-06
Build ID bcd7739a
Cmdline /opt/gravwell/bin/gravwell_webserver -stderr gravwell_webserver.service
Executing user gravwell
Parent PID 3411
Parent cmdline /lib/systemd/systemd --system --deserialize 14
Parent user root
Failed to wait for new license: listen tcp 0.0.0.0:80: bind: address already in use
Nope. Not there. Nothing defaults to port 80 or looks like it would change it, except for Web-Port.
So, I really tried to give your product a shake. I'm very interested in having some centralized logging for my hobby projects, but not at the cost of nginx on port 80. So...\o/
The port 80 listener is just a redirect to https. Add `Disable-HTTP-Redirector=true` to gravwell.conf to disable that redirector. The option is documented in the document you linked, but I can see how you'd miss it: we talk about "HTTP" and "HTTP" rather than "port 80" which would make it more of a pain to search for.
Changing `Web-Port` will change the HTTPS listener port, as you figured out.
We're working on populating our new knowledge base now with the answers to this and other questions which came up during our community edition rollout: http://help.gravwell.io/knowledge/