
Gravwell Community Edition: A Splunk Alternative Built with Go - floren
https://www.gravwell.io/blog/gravwell-community-edition
======
e12e
> The Community Edition is limited to 2GB/day of ingest (real ingest, not
> indexed data) and should handily cover any home use and most smaller I.T.
> and Security shops.

So it's not so much "a free splunk alternative" as it is "a system with
slightly different free tier than splunk"?

From
[https://www.splunk.com/en_us/software/pricing.html](https://www.splunk.com/en_us/software/pricing.html)

> Splunk® Free (...) Scale up to 500 MB data per day

~~~
big-endian
[https://dev.splunk.com/page/developer_license_sign_up/](https://dev.splunk.com/page/developer_license_sign_up/)

You can grab a 10GB/day Dev license for Splunk for free. IANAL, but should be
suitable for home use. Lasts a few months, and renewals are free.

~~~
krageon
You are then chained to a renewal process and the whim of the agency giving
out those licenses. That isn't freedom, that is servitude.

~~~
bloopernova
Every single time we have to renew our Splunk license, it's a hassle.

Unfortunately Splunk's installation process is much more streamlined and
junior-sysadmin-friendly than ELK, at least in our limited view.

I need to look again at ELK to see if someone has created or improved the log
input UI and indexing UI. Just something that lets you select log locations
and types on a client in a web UI "somewhere", and something similar on the
server to perform various operations for indexes and other such stuff.

~~~
thaeli
Graylog2 (the open source version) is batteries-included and a very easy
install. Definitely a lot easier than ELK stack.

------
tptacek
The convention in software marketing is for a “Community Edition” to be the
batteries-not-included DIY-install version of an open source project. That’s
what the “community” in the name refers to.

Is that what this is? Where’s the source? How do you limit the ingestion
capacity of an open source project?

If the core isn’t open source, that’s fine! But that makes “Community Edition”
a misleading name.

~~~
mandeepj
Microsoft visual studio also has a community edition but they also have not
released its source code. So, I don’t think it is expected or mandatory to
share product’s code with public

~~~
tptacek
That's a good point.

------
packetized
Why does it matter what it’s built with, if it’s closed-source?

~~~
omeid2
Never underestimate the power of hype.

~~~
packetized
Generally, I agree with you - but it’s 2018, and I can’t see how hyping a SaaS
by name dropping Golang is actually hyping anything.

~~~
omeid2
The simple answer is that it helps. The longer answer involves cost analysis
and whatever it creates a positive connotation or not, and what is the risk of
dropping this name, basically, marketing.

------
foobarbazetc
Isn’t this just an ad since this isn’t open source and the free version is a
trial?

------
withinrafael
Some feedback: I clicked on this community edition blog post to learn more
about the product, only to be immediately smacked in the face with a product
demonstration of a user installing a license file. Yikes. That turned me off
immediately. Will re-review later, still sounds cool.

~~~
floren
You're right, it was kind of a weird and glaring thing to put there. We've
removed it and replaced it with a screenshot of Gravwell in action.

~~~
withinrafael
Great, thanks!

------
orand
“Built with Go”. Seems like there are a lot of “built with X” articles. When I
see these I always think who really cares that much which language something
is built with? Isn’t there anything more headline-worthy about it than the
programming language?

~~~
pjmlp
It gives leverage when someone states that X cannot be done in Y.

So for example, if someone states C# cannot be used for writing an OS, then I
can point to Midori.

~~~
krageon
I haven't yet met anyone that wasn't just trolling when they claimed something
like that. It _should_ be fairly obvious you can do practically everything at
any level of abstraction. It will just get more or less efficient (and even
that is not a given).

~~~
pjmlp
You had luck, some people only accept certain truths specially in IT, if it is
shown running before them.

And even then, they might still dismiss them even if proven wrong.

------
floren
Just for fun, we ingest comments from Hacker News, so I made a quick search to
find out who's been posting about Gravwell the most:

[https://i.imgur.com/1il6PuJ.png](https://i.imgur.com/1il6PuJ.png)

How embarrassing, it's me! :)

~~~
NoNotTheDuo
Please, please, please stop using pie charts to try to visualize ratios! See
here[0] from 2007 for better alternatives. I'm sure there are newer
visualizations as well.

[0]:
[https://www.perceptualedge.com/articles/visual_business_inte...](https://www.perceptualedge.com/articles/visual_business_intelligence/save_the_pies_for_dessert.pdf)

~~~
robertely
100% Agree it's been beaten to death but people still do it:

[http://www.storytellingwithdata.com/blog/2011/07/death-to-
pi...](http://www.storytellingwithdata.com/blog/2011/07/death-to-pie-charts)

[http://www.businessinsider.com/pie-charts-are-the-
worst-2013...](http://www.businessinsider.com/pie-charts-are-the-worst-2013-6)

[https://qz.com/1259746/when-should-you-use-a-pie-chart-
accor...](https://qz.com/1259746/when-should-you-use-a-pie-chart-according-to-
experts-almost-never/)

------
floren
This is John from Gravwell; I'd be happy to answer any questions about
Gravwell or Community Edition. We're rolling it out to the public today, and I
thought our network security focus might interest the HN crowd!

~~~
colonelxc
I think this post (and the home page) would benefit greatly with some example
dashboards and queries. You really have to dig deep into the docs to see what
it is capable of.

Splunk on the other hand has dashboard examples on almost every page (on their
homepage, a carousel with five different examples of the things they purport
to solve).

(another blog post does a great job with showing off capabilities:
[https://www.gravwell.io/blog/gravwell-and-
collectd](https://www.gravwell.io/blog/gravwell-and-collectd) )

~~~
floren
It's a good point. What would you find particularly compelling as an example
dashboard? We've built dashboards around hardware stats (cpu temp etc.), port
scanning & brute-forcing attempts, even Reddit comments. All of the above? :)

~~~
colonelxc
I like the blog post I mentioned above, which covers cpu/disk/ram monitoring.
I'd also like to see a fleshed out network analysis example.

------
anothergoogler
This looks great, is it free as in speech or free as in beer? I didn't see it
on [https://github.com/gravwell](https://github.com/gravwell)

~~~
floren
Gravwell Community Edition is a free-as-in-beer license for our core product.
Our github contains associated tools that we've made free as in speech: the
ingesters, which gather data and store it in Gravwell; the ingest library,
which you can use to write your own ingesters; and some additional libraries
of more niche interest.

~~~
anothergoogler
Why do we care that it's written in Go then?

~~~
floren
1\. HN users often seem to like knowing how something was made

2\. Our open-source components (github.com/gravwell) are written in Go

~~~
anothergoogler
Got it, well thanks for open sourcing some of it!

------
condiment
Speaking frankly, unless a Splunk alternative implements API compatibility for
search, it’s a nonstarter in the marketplace. At the enterprise I work at,
developers and operations teams both use splunk to observe and analyze
application behavior, and have thousands of dashboards and alerts set up and
integrated into mature operational processes. Migrating this over to another
application is nontrivial. And the biggest problem isn’t even a technical one,
it’s a social one - how do you train three hundred engineers to use a
different log search tool.

I would absolutely love to see a competitor emerge that addressed the
migration problem through a compatible search api. Handling other timeseries
data like metrics would just be icing on the cake.

~~~
bovermyer
Just because it's not API-compatible with Splunk doesn't mean it's a
"nonstarter."

Splunk is far from the only way to do what it does.

~~~
condiment
I don’t disagree. What I meant was that it’s a nonstarter for replacing any
existing installation of splunk, which is desirable for me as an enterprise
customer who spends a nontrivial amount of money every year on this sort of
tool. There are a lot of obstacles to replacing an existing, effective
implementation of a tool, and I listed what they are in the hopes that
somebody pays attention.

~~~
cthuen
I get what you're saying. I don't think you're wrong. This isn't currently a
priority for us but I hope that someday someone builds something like that.
That's one of the aspirations of releasing a Community Edition. Our API docs
are open:
[https://dev.gravwell.io/docs/#!api/api.md](https://dev.gravwell.io/docs/#!api/api.md)

------
jcims
Has anyone tried using jupyter + plotly + pyspark or similar as a poor-man's
splunk?

~~~
MasterScrat
The usual "poor-man's Splunk" is the ELK stack.

~~~
jcims
For search yes but I haven’t figured out how to do any real analytics with it.

~~~
meowface
Even for regular search, Splunk's search language absolutely blows
Elasticsearch's/Lucene's out of the water. Splunk is one of the best software
products I've ever used and developed extensions for. A shame it's so
obscenely expensive.

~~~
jcims
Yeah it's going to sound ridiculous but there are occasions where I really
feel like I'm sculpting with information when I'm using Splunk...and that's
with me using maybe five different commands from this list -
[http://docs.splunk.com/Documentation/Splunk/7.1.1/SearchRefe...](http://docs.splunk.com/Documentation/Splunk/7.1.1/SearchReference/ListOfSearchCommands)

I'm fortunate to work for a company that invests 8 digits per year in their
Splunk infrastructure, it's a travesty that I don't leverage more of its
capability.

------
ultimoo
Any plans of having a SaaS offering?

~~~
cthuen
It's easy to deploy to the cloud and right now we're doing that on a customer-
by-customer basis. As we grow I could see turning it into a formal SaaS
offering or finding a partner to do so.

------
Xorlev
I'll re-post my experience here:

I tried installing Gravwell from the Debian repo. This unfortunately seems
broken.

    
    
      W: Failed to fetch http://update.gravwell.io/debian/dists/community/InRelease  Unable to find expected entry 'main/binary-i386/Packages' in Release file (Wrong sources.list entry or malformed file)
    

(also, that should probably be
[https://update.gravwell.io...](https://update.gravwell.io...))

Once installed from the tarball, it failed to start.

    
    
      The gravwell_webserver process is not running!
    
      If you kept an old configuration file the configuration parameters may have changed.  Try manually starting the services and looking for errors.
    

Sure, I need to go update the web ports to not interfere with nginx. Makes
sense.

    
    
      $ vim /opt/gravwell/etc/gravwell.conf
      s/Web-Port=443/Web-Port=1443/
      $ systemd start gravwell_webserver
      $ systemd status gravwell_webserver
      gravwell_webserver.service - Gravwell Webserver Service
         Loaded: loaded (/etc/systemd/system/gravwell_webserver.service; enabled)
         Active: failed (Result: start-limit) since Thu 2018-07-12 02:15:07 UTC; 23min ago
        Process: 3411 ExecStart=/opt/gravwell/bin/gravwell_webserver -stderr %n (code=exited, status=255)
    

Huh, maybe I missed something else.

    
    
      $ journalctl -u gravwell_webserver
      -- Logs begin at Fri 2018-07-06 20:35:20 UTC, end at Thu 2018-07-12 02:41:26 UTC. --
      <EOF>
    

That's odd, where's the logs?

    
    
      $ ls -la /opt/gravwell/logs/web
      drwxr-x--- 2 gravwell gravwell 4096 Jul 12 02:07 .
      drwxr-x--- 4 gravwell gravwell 4096 Jul 12 02:17 ..
    

No logs. Lets check the crash folder?

    
    
      $ ls -la crash
      drwxr-x--- 2 gravwell gravwell 4096 Jul 12 02:17 .
      drwxr-x--- 4 gravwell gravwell 4096 Jul 12 02:17 ..
      -rw-r--r-- 1 root     root      322 Jul 12 02:07 gravwell_webserver.service_2018-07-12T02:07:36Z.log
      -rw-r--r-- 1 root     root      354 Jul 12 02:15 gravwell_webserver.service_2018-07-12T02:15:07Z.log
    

There it is. Double click the filename to copy to...oh, it has colons.

    
    
      $ less gravwell_webserver.service_2018-07-12T02\:15\:07Z.log
    
      Version         2.0
      API Version     0.1
      Build Date      2018-Jul-06
      Build ID        bcd7739a
      Cmdline         /opt/gravwell/bin/gravwell_webserver -stderr gravwell_webserver.service
      Executing user  gravwell
      Parent PID      3411
      Parent cmdline  /lib/systemd/systemd --system --deserialize 14
      Parent user     root
      Failed to wait for new license: listen tcp 0.0.0.0:80: bind: address already in use
    

Why's it listening on port 80? I must be missing a config option. Lets check
the docs...
[https://dev.gravwell.io/docs/#!configuration/parameters.md](https://dev.gravwell.io/docs/#!configuration/parameters.md)

Nope. Not there. Nothing defaults to port 80 or looks like it would change it,
except for Web-Port.

So, I really tried to give your product a shake. I'm very interested in having
some centralized logging for my hobby projects, but not at the cost of nginx
on port 80. So...\o/

~~~
floren
We got a support email about this, if it was you then you probably already
know the solution, but otherwise:

The port 80 listener is just a redirect to https. Add `Disable-HTTP-
Redirector=true` to gravwell.conf to disable that redirector. The option is
documented in the document you linked, but I can see how you'd miss it: we
talk about "HTTP" and "HTTP" rather than "port 80" which would make it more of
a pain to search for.

Changing `Web-Port` will change the HTTPS listener port, as you figured out.

We're working on populating our new knowledge base now with the answers to
this and other questions which came up during our community edition rollout:
[http://help.gravwell.io/knowledge/](http://help.gravwell.io/knowledge/)

------
hidiegomariani
click-bait

