
Dejavu – Web UI for Elasticsearch - reactivesearch
https://github.com/appbaseio/dejavu
======
giancarlostoro
Ok so maybe someone can tell me what I (we) did wrong at my job we tried using
the ELK stack, and it's probably still running but it is such a resource hog.
I do not understand why they built Elasticsearch. I've read in a couple places
you need like 32GB of RAM[0] just to run this thing to do queries, and having
crashed Kibana / Elasticsearch a dozen times I believe it's designed poorly. I
had hoped I could drop in MongoDB instead, but saw no indication of this being
a fluid change. How many resources are any of you allocating towards your
'ELK' stack (I say 'ELK' cause now they have other software in the mix)?

Needless to say, I rather build my own solution for logging instead using a
database that's not in-house having experienced all of this.

[0]:
[https://www.elastic.co/guide/en/elasticsearch/guide/current/...](https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html)

Oh wow now it says 64GB of ram is the sweet spot.... What the heck is this
thing doing that couldn't of been accomplished with MongoDB or PostgreSQL?
I've got busier data sets that don't need 16GB of RAM, and yes we pound the
database with logs of sorts and query in all sorts of ways and still I don't
get it... I wouldn't recommend this stack to a friend unless they've got
plenty of hardware to spare.

~~~
jstarfish
> tell me what I (we) did wrong at my job we tried using the ELK stack, and
> it's probably still running but it is such a resource hog.

Part of the problem is what it's promoted for. It's a great drop-in,
horizontally-scalable, full-text search engine, that's inexplicably become
popular for log ingestion and analysis.

To those ends, I hate it, I hate every bit of it, from the atrocious JSON-
based query DSL (seriously thought it was a joke at first) to its
unpredictable timeouts, shard storms, mapping conflicts and other problems at
scale. Elementary SQL concepts aren't possible ('select
someone_elses_poorly_named_key as first_name', nope, you gotta reindex). High-
cardinality aggregations fail in spectacular ways. High-anything aggregations
fail in spectacular ways. The scroll API returns results unordered. There's no
way to properly spec your cluster; the docs explicitly take a trial-and-error
approach to design.

It's not just you. Elasticsearch does me no favors with the task of log
analysis. I'd sooner normalize and grep a pile of gzipped log files than keep
dealing with this mess, but this is the second job I've been at that's built
their logging infrastructure on top of it.

> I do not understand why they built Elasticsearch.

"You know, for search."

It's great for searching proper text. Documents, comments, blog posts, etc.

> I've read in a couple places you need like 32GB of RAM[0] just to run this
> thing to do queries, and having crashed Kibana / Elasticsearch a dozen times
> I believe it's designed poorly.

You don't necessarily need 32GB to do anything. The required heap size scales
with your intended workload, but it's not like you can do a back-of-napkin
calculation to figure out the relation. I run a 1GB instance for development.

It uses the memory to do a lot of caching so the queries you throw it are
lightning fast. Mongo does something similar (but crappier IMO) with its
concept of working sets.

~~~
fareesh
I've deployed it a few times. In some cases it randomly spikes to 100% cpu
usage until you restart it.

Is there a way to know how much RAM you are going to need for your dataset? I
think I was using it for looking up restaurant names from a database of
100,000 and I wanted to factor in misspellings and partial matches.

~~~
catmanjan
Not really, depends on the size of the dataset and the complexity of the
operations - when did you use it? Sounds like you were on a dodgy build or set
up

I think Elastisearches policy on sizing is pay us or a partner to have a look
and give you a guesstimate, which is pretty standard

------
burner8309
> dejavu is the missing web UI for Elasticsearch. Existing web UIs leave much
> to be desired or are built with server side page rendering techniques (I am
> looking at you, Kibana).

When did server-side rendering become a bad thing? It tends to be much faster.
Kibana feels slow to me, and that's because of the JavaScript it does have.

~~~
rpedela
I don't agree that server-side rendering tends to be faster. I think it
largely depends on the use case. I am going to use the word "responsive"
rather than "faster" since it is more descriptive of how a UI feels. If you
don't need to go to the server, performing an action will be more responsive
if done client-side in general. If you do have to go to the server, then you
are at the mercy of the network. However the less data transferred, the better
in terms of responsiveness. In many cases, asking the server for JSON rather
than HTML/JS/CSS is more responsive. Render time is also a factor. If client
render time is slower than network/server time, then rendering on the server
is more responsive in general.

~~~
purerandomness
I agree that client-side UI modifications tend to be faster if no server
round-trip is necessary.

Rendering HTML will, however, tend to be faster than having a client parse and
interpret JSON, and then manipulate the (shadow) DOM accordingly.

And then it always felt wrong to me that I have to force countless clients to
do the exact same view transformations, which could have been done once, on
the server, and cached.

For a new project, I'm prerendering HTML fragments via AJAX with IntercoolerJS
- it's a pleasure to work with, and the developer and user get both the best
experience of those two worlds: A dynamic, blazingly fast UI that feels like
an SPA, but degrades gracefully even for the Emacs-w3m user and Googlebot.

I also found it's really worth it to move servers or reverse proxies closer to
the client - independent of client/server rendering. Users shouldn't wait more
than 200ms for an Ajax request to finish.

------
dabeledo
I've been using elasticsearch-head [1] for four years now and it seems it has
more features than dejavu? Specially the node/shard visualisation.

[1] [https://github.com/mobz/elasticsearch-
head](https://github.com/mobz/elasticsearch-head)

~~~
reactivesearch
Yes, we don't focus on admin related features atm. Elasticsearch head or
Elasticsearch HQ [1] are good options for doing this.

Fun fact: We actually started out with elasticsearch-head, but realized it was
really hard to hack on (this was ~2y ago) and decided to create Dejavu. Imo,
the key difference is we have focused a lot on how to get the data indexed
(mappings, bring your CSV/JSON files) and have the raw data visualized (via UI
based filters or DSL queries).

[1] [https://github.com/ElasticHQ/elasticsearch-
HQ](https://github.com/ElasticHQ/elasticsearch-HQ)

------
throwaway2016a
> dejavu is the missing web UI for Elasticsearch. Existing web UIs leave much
> to be desired or are built with server side page rendering techniques (I am
> looking at you, Kibana).

So less so "missing" UI and more "outdated" UI.

I personally like Kibana but I am happy to see some innovation in the space.
Competition can only help.

I love that this can be run as a Docker container or as a Chrome extension.
With the quick setup I'll probably give it a try. It looks nice.

~~~
weego
Jesus, since when is server side rendering of server side data supposedly an
anti-pattern?

------
artellectual
One thing to note for anyone using elasticsearch in production, if you setup
your own cluster. It might be wise to put nginx as a reverse proxy. By just
doing that we reduced the amount of crashes when the server got hammered.
Since nginx will help with request queueing

------
forgotcreds
>built with server side page rendering techniques (I am looking at you,
Kibana).

I'm not sure if this is actually accurate:

1\.
[https://github.com/elastic/kibana/search?utf8=&q=React](https://github.com/elastic/kibana/search?utf8=&q=React)

2\. [https://elastic.github.io/eui](https://elastic.github.io/eui)

~~~
reactivesearch
I will have to relook at this - the comment is from a year ago when they
weren't using React.

On a cursory look, it still seems to use a Node.JS based server [1]. One of
the key side effects of needing a server is that there is a more involved
distribution + installation process. You just can't run it on a static server
like github pages or as a browser extension.

[1]
[https://github.com/elastic/kibana/search?utf8=%E2%9C%93&q=ha...](https://github.com/elastic/kibana/search?utf8=%E2%9C%93&q=hapi&type=)

------
bestest
I've stumbled on a couple of bugs with Dejavu, and wanted to see if I can
submit a bug fix. And, well, submit and fix them I can, but boy oh boy is the
code terrible. Only use this on your own risk and on a throwaway ES instance,
as the code quality in this case kind of implies there's major risk of
screwing things up.

------
ninjaranter
> dejavu uses a websockets based API and subscribes for data changes for the
> current filtered view. For this to work, the Elasticsearch server needs to
> support a websockets based publish API

Is websockets a requirement just for the real-time updates, or for the entire
UI?

~~~
m1sta_
Why do you ask? Are you planning on accessing an elasticSearch admin tool from
a browser that doesn’t have web socket support?

~~~
milesokeefe
There’s no open source websockets-based ES publish api, you have to use the
closed source private service made by the company which created this project.

------
tenken
Is there anything similar for Solr??

~~~
reactivesearch
Not that I know of, but given the similarities in the underlying APIs - we
would be open to supporting this and help guide the implementation if someone
is interested in sending a PR. Feel free to file an issue -
[https://github.com/appbaseio/dejavu/issues/](https://github.com/appbaseio/dejavu/issues/).

------
braderhart
I thought for sure that you were going to be a PDF alternative, but then I
remembered the spelling is different.

~~~
romwell
Yeah, when I looked at the title, I was confident I've seen that before, but I
hadn't.

You might say I had a case of, ah, never mind, bad pun.

On a serious note, I understand there's only a finite number of words, but a
limitless number of software project - and yet surely there are still enough
names that aren't used for something well-known and widely used. Yes, the DJVU
and Dejavu are spelled differently, but it is the same word.

