
Show HN: LogDNA – Easy logging in the cloud - leeab
https://logdna.com/?ref=showhn
======
leeab
Hey everyone. I'm a co-founder / CTO of LogDNA. We were in Y Combinator's W15
batch working on a eCommerce marketing platform (PushMarket), only to realize
we built a powerful logging system many of our friends wanted.

We had our “slack moment” and decided to pivot. We were frustrated with the
current logging landscape and built LogDNA around 3 philosophies:

1\. More Storage - give away ample storage so you can log everything

2\. Longer Search - faster and longer search retention

3\. UI/UX - a much cleaner experience to interact with your log tails

I’m happy to answer any questions you may have!

~~~
tyre
Your marketing seems to emphasize storage heavily. Is that the biggest pain
point in logging today? If so, my guess would be that retention period could
be a bigger differentiator than sheer gigabytes.

What's the killer use case or differentiator (vs. loggly+kibana+elasticsearch
for example)? With Slack, it is integrations. If you could do something
similar, like a bugsnag + new relic that aggregates all of our logs and
notifies us when things happen (either individual incidents or aggregates like
"nginx error log above 1 event/10s), you can have all my money.

Another idea would be to standardize transactions across the stack, so I could
trace a request from nginx through rails/phoenix, to postgresql, etc. Again,
take my money.

A better UI only helps me if I know when to look at it or what I'm looking
for.

~~~
chrissnell
Ding ding ding ding.

It's not the log storage that we care about, it's what we can do with those
logs.

I've been through three logging vendors in the last two years (Loggly,
LogEntries, now Sumo Logic) and I'm looking to move once again. The secret
sauce is not the storage volume; our volume is relatively small: 4.3 GB/day.
The secret sauce is the intelligent structuring of unstructured log data,
analytics+alerting on the now-structured data, and live tail'ing of logs.

Sumo Logic does an okay job at the parsing/querying/alerting/analytics piece
but live tailing is a brand-new feature for them and it has limitations. Sumo
Logic's pricing is also frustrating--they don't apply volume discounting until
you're pushing something like 500 GB of logs each month. Again, we need
affordable analytics; storage is a small piece of the puzzle.

Honestly, our next move may be to an internal "HEK" stack: Heka,
Elasticsearch, and Kibana. I've looked at the cost of doing this and it's
actually more expensive when you consider what we'd spend to host a big ES
cluster internally, but cost is not our #1 consideration, usability is.

~~~
leeab
The main reason we emphasized storage is because we built it with scaling that
in mind, that's all. We've also built it with live tail in mind too. It's an
okay differentiator since we believe we can handle the volume with our
infrastructure. We also followed up with UI/UX which at the end of the day
should make us a platform you can goto, get what you need and go back to work.

Give us a try, even if you don't think you'll use the GBs. We have alerting,
field parsing and live tail. Analytics is the only piece we don't have today,
but we're working towards it.

~~~
chrissnell
I understand. For me, though, scaling is just a small piece of the puzzle and
not really a differentiator. To give you an idea of our size, we run about
~500 instances in a private cloud hosted on about 45 dedicated servers at
Rackspace. We use OpenStack for stateful services and data storage and we use
Kubernetes/Docker/CoreOS for 12FA/microservices. We have many dozens of apps
and even with all of that, we're still only pushing about 4-5 GB of logs each
day.

The number one most important problem we're trying to solve is the actual
parsing of logs during ingestion. We need logs from commonly-used apps &
frameworks to be automatically identified and parsed and we need an easy way
to set up a custom parser for internal logs that doesn't require crazy hand-
written regex. How does LogDNA handle parsing? What apps/frameworks are
supported?

~~~
leeab
We automatically detect common formats using regex during ingestion. Many are
based on logstash patterns and we have several of our own (including one that
just tries to strip out timestamp + log level). If you use a common format, we
can usually add it. If you have a custom format that your app defines, we
don't have a way for you to add it yet but it'll likely be a regex as you
said. (We can always help you create it if you don't want to).

------
cporios
Scrolling down your website is very slow, weird and annoying. Also the scroll
left to go back gesture isn't working.

~~~
mbrock
It's just _astounding_ how web sites still insist on replacing the normal,
perfectly fine scroll behavior with weird, buggy, slow, crazy, half-assed,
dizzying, and _totally unnecessary_ custom scroll implementations.

------
agentgt
I have few questions and comments.

* Love the price

* Can or will you add name value pairs? Logging for us is no longer just a single line of text. We have attached metadata. Kibana handles this nicely.

* How good is your security? This is actually our biggest reason why would not use a service like this. Not that we put passwords or user sensitive data in logs we still have to keep them extremely secure due to compliance.

I'm sure you have decent security but I would recommend emphasizing that
and/or proving it.

~~~
leeab
Thanks for the feedback.

Yes, just send us JSON in a line and we'll parse it. You can actually search
on that today, we just haven't made a way for you to see the parsed fields yet
(coming soon!)

We're fully encrypted end-to-end in transit (but not at rest, currently).

Yeah we need documentation mainly and couldn't get that in time for launch.

~~~
zobzu
encryption at rest would be quite amazing.

------
doxcf434
I'd recommend making your pricing model simply per GB and then on a rate tier,
don't play games with your customers. The quota system ends being a game, and
simply ends up wasting management's time trying to ensure the company isn't
either getting charged for overages or not using up the quote they have. I'd
quite frankly not recommend companies like Sumologic solely based on the
massive amounts of time they waste with these things.

------
compumike
Our team at Pantelligent has been happily using the LogDNA beta for the last
few months. The product is well designed and easy to use for many members
across engineering, devops, and marketing roles. Very impressed so far, and
this is only the beginning of where this product can go! Setup takes about 2
minutes -- give it a try today.

~~~
tyre
I downvoted you because this sounds too much like an infomercial. Be honest,
share your story, support your YC brethren, but we're users, not VCs.

------
Swizec
Question: Why does a new startup/solution for this pop up every few months?
Why is this space so difficult?

I've definitely thought about building this myself, but haven't gotten around
to it. It's just sending some text to a server, isn't it? What's so hard about
it?

Obviously I'm missing something. I'd love to know what :)

~~~
leeab
Hi there,

I wasn't aware there were several startups every few months in logging. For
us, it was an interesting space. Lots of changes happening.

Yes you are somewhat right. It is sending text to a server. But then you send
more text and more text and you need to search for all this text in a split
second. If you've ever run your own ELK stack for example, you'll realize
quickly that you're spending most of your time scaling elastic search to
handle the data volume. There is a challenge to doing that.

~~~
matzipan
It's idiotic. You're not bringing in any major differentiation. Your idea will
be implemented by your competitors in less than a week and you will have no
competitive advantage and way less clients.

You've been through YC? How the hell did you graduate? Because you just did
mistake number 4:
[http://paulgraham.com/startupmistakes.html](http://paulgraham.com/startupmistakes.html)

~~~
halayli
PG's writings aren't the bible. Situations and verticals vary. You still need
to use your reasoning after all and judge yourself. Case in point, PG himself
agreed to this business idea which in your opinion contradict himself.

~~~
matzipan
True, PG's writings are not the bible just as his agreement with this business
idea doesn't mean it's a good idea.

I used my own judgement and concluded that this is a worthless idea, as
competition is high and differentiation low. I referenced that article to give
credit to the original author.

~~~
halayli
But that same author approved the idea. You see the contradiction?

~~~
matzipan
On his side? Yes...

------
jorgecurio
What AWS stack will let you build something like this? I don't think I would
trust a 3rd party with all of my log data for...I can't really see what the
benefit of moving to cloud would be, a log file sits on a local disk or if you
had to move it to the cloud you could use S3 or SQS even no?

~~~
tommoor
Another benefit is if you have many services then viewing the logs in
timestamped order can make debugging across services far easier. This is
really quite difficult without a single service aggregating and indexing all
your logs

~~~
maerF0x0
It gets especially useful when you start to get unique ids across "jobs".
Imagine a single flow of work hits 10 services, but you can search for
job_id=123 and get all the logs for the job . Now its easier to see what went
wrong and where in the flow.

------
phoenix24
I've been using scalyr.com for logging extensively, for last few months. How
do you differentiate against them?

~~~
leeab
Other than the post by their founder above, I'm not really familiar with them.
But overall, the 3 points I mentioned in my opening comment is generally
similar. But to really find out if you'll like us, you should just try
installing us. We have a working live demo on
[https://logdna.com](https://logdna.com)

------
chrissnell
I think your "Features" page could use some work. I don't consider syntax
highlighting themes to be much of a feature. What I want to hear about is how
you can take logs from all of these <dozens> of applications and intelligently
parse them and deliver analytics and alerting on them.

------
jmtulloss
My favorite part of logentries is the fact that you can use structured logs to
build dashboards showing service health. My least favorite part of logentries
is the UX around that whole thing. Any chance more features are coming in the
structured logging area?

~~~
midnightjasmine
Agreed on the LogEntries UX. I find it really frustrating.

------
koolba

        echo "deb http://repo.logdna.com stable main" | sudo tee /etc/apt/sources.list.d/logdna.list
        sudo apt-get update
        sudo apt-get install -y --force-yes
        logdna-agent sudo logdna-agent -k 6a7b7c622290d1a49bbe5a94dc6828d
    

Haven't tried this yet but the install instructions don't include adding a GPG
key. Instead it has the option "\--force-yes" which I believe skips that
check.

That would work for the initial install but wouldn't it complain down the road
when you try to update the package?

~~~
leeab
No, sorry, I just didn't have time to set this up. We'll get this before the
updates are made.

------
nodesocket
Looks nice, but I am solid lover/supporter of Papertrail. I really love their
simple/minimal approach.

I think the missing piece of Papertrail is the ability to send JSON logs, and
filter inside of a JSON documents.

~~~
leeab
We'll parse your JSON if you send it in one line. And you can search on it
like this: fieldname:keyword

I think we're pretty good on our UI/UX. Give us a try!

~~~
nodesocket
So we currently ship around 10GB a month, and it costs us $75 a month from
Papertrail. Cost is really a non factor, wouldn't matter if is cost us $200.
The ability to search and grep older documents is a bit slow on Papertrail
though. Additionally, sending deeply nested JSON, and being able to search on
props would be huge.

Does LogDNA have a shipper from syslog? Do you have a node.js client lib?

Finally can you talk about security? Is traffic encrypted over the wire? Do
you encrypted disks at rest?

~~~
leeab
We parse JSON and you can search your own fields.

We do not have syslog yet or code libraries. We have an agent that ships logs
today via secure web socket.

Everything is encrypted in transit, end-to-end. We haven't not researched into
encryption at rest on EC2 yet.

~~~
lobster_johnson
I'd like to use LogDNA, but we'd require syslog support, I think. We use
Rsyslog to spool logs off to a central server, and we're not about to trust a
third-party vendor to keep our data (ie., we want our own copy). Rsyslog can
forward to any other syslog server and supports TLS.

Edit: Rsyslog can pipe data to a command-line program via the "omprog" plugin,
which might be an easier option for you.

------
askedrelic
Looks promising! I was able to get it installed and delivering logs in 5 mins.

I really like the high volumes you offer; I really agree that volume should be
cheaper. What Papertrail charges per GB seems really high IMO.

That said, I'm just a hobbyist doing under 1GB month of traffic. I will try to
stick with the free plan for as long as possible. If you could offer some nice
features for $5-10/month, I would upgrade for that.

------
ohblahitsme
Maybe I'm missing something, but it looks like this has nothing to do with
DNA? Why is it called LogDNA? Seems a little misleading.

That aside, it looks very interesting!

~~~
leeab
Lol sorry! It took us a bit to find a domain that was available. We do like
the DNA part though...very logo-able and short!

~~~
matzipan
Then use your logo and slogan to indicate something else other than DNA.

------
rgacote
Can you tell me how wide I have to open our Firewalls to get to your servers?
(i.e., 1 or 2 IPs or an entire AWS range). We're in a locked-down compliance
environment and opening all our servers to a wide range of external IP
addresses is a problem.

Would love to hear you have an aggregator/forwarding tool that I could run all
my logs through to forward to your environment.

------
tomkazarian
Set a canonical to [https://logdna.com](https://logdna.com) to help with
indexing/attribution issues of
[https://logdna.com/?ref=showhnshowhn](https://logdna.com/?ref=showhnshowhn)

------
encoderer
Congrats on shipping! At Cronitor, we have a lot of users pushing events from
PaperTrail for monitoring and alerts. I'd love to work with you on an easy
integration at some point, though I do have an appreciation for how busy you
are. We didn't add our first real integration -- PagerDuty in our case -- for
a few months after launch.

~~~
leeab
Thanks! Send me an email at lee[]logdna.com

------
circuitpeople
AWS SNS -> Lambda -> Firehose -> S3 | Redshift

Done.

~~~
rendambathu
Anything other than SNS? Within AWS or outside of it?

------
melted
This only solves a part of the problem. This is basically to collect and grep
your stderr, useful for when you need to debug something, but that's about it.
A far more interesting problem is to collect, ingest, and query structured
(Proto, Thrift, etc) _application_ logs that people should also be writing.

~~~
leeab
We do automatically parse most common log formats and you can query on those.
You can also send us JSON and query on that too.

~~~
melted
How do you deal with hierarchical data and repeated fields? Do you support
those in queries (a-la Google BigQuery)? Do you support aggregations?

------
leef
Why the severe restriction on users? $300/mo to go beyond five users
eliminates any advantage on log volume.

~~~
leeab
To be honest, we weren't really sure what to pick. We just wanted to have
something in place. It's not set in stone and we'll likely change things as we
see different use cases.

------
packetized
How are you not hemorrhaging money by running this on top of AWS/RS/et al?

~~~
packetized
For what it's worth: I've recently built an internal logging platform that
handles ~300GB/day on spare hardware with Heka/RMQ/ES/OTSDB/Kibana/Grafana.
Running something similar from a SaaS logging provider would be (at bare
minimum) a five digit monthly bill.

------
guillegette
ITT: when your "Show HN link" turns into your competitors list

------
paulspringett
Nice! Have you got plans to support JSON / logstash formatted logs?

~~~
leeab
We actually already do this, but we don't show the results in the log viewer,
but we do parse JSON and a whole bunch of various logstash patterns + some of
our custom ones. You can even search on them today by doing: fieldname:keyword
but it's just hard since you don't know what the fieldnames are...you can
guess but we're exposing those soon!

------
kevincox
First of all, your website is painful to use. It requires a ton of scrolling
to see anything and as previously mentioned scrolling is painful.

Secondly there is almost no detail so while your product sounds cool I can't
see what it is like to use your product. Give me examples of creating queries
or dashboards. There should definitely be a documentation section on your
site!

Also, a really cool feature I can't see anywhere else is reliable journald log
uploading. You can dump as JSON which does most of the work but it is a pain
implementing a reliable uploader with a HTTP API.

~~~
leeab
Hi there,

We just built our website this week since we were heads down focused on
product. Our documentation is coming soon. We just wanted to get a product
into the hands of users and had to make tough decisions on what we needed.

We could be perfect and launch 2 months from now or just launch now with what
we have. But you're right, we will be plugging the holes quickly.

------
jaequery
Anyone know where this fits into servers requiring PCI-compliances?

~~~
leeab
Having gone through PCI compliance before, technically you cannot send us data
that includes card numbers since we're not PCI compliant (yet). But if you
don't log that type of data, there is no issue.

------
matzipan
Yet another startup solving yet another already solved problem with little to
no differentiation.

That's why we need to encourage more kids to do STEM and tackle hard problems,
rather than encouraging "makers" and getting them all to be app builders.

I have yet to live the day I will regain my faith in startups.

~~~
leeab
I'd love to see what you have worked on. :)

~~~
jessaustin
Haha, mischaracterizing valid arguments as " _ad hominem_ " seems more common
than logging startups!

~~~
matzipan
How is that?

~~~
jessaustin
[https://hn.algolia.com/?query=ad%20hominem&sort=byDate&prefi...](https://hn.algolia.com/?query=ad%20hominem&sort=byDate&prefix&page=0&dateRange=all&type=comment)

The current first comment:

 _It 's a straight up ad-hominem attack against Justin Keller, instead of
criticising_ [sic] _his writing it attacks his character._

That's just as fallacious an invocation of _ad hominem_ as your link was.
Description of questionable behavior is not fallacious, in an article the
purpose of which is to complain about questionable behavior. Likewise, it is
entirely reasonable to expect withering unsupported criticism like yours to
spring from some a personal position of some relevant experience.

