
Show HN: AvionDB: A Decentralised Database with MongoDB-like Developer Interface - Alex-Potsides
https://github.com/dappkit/aviondb
======
davidajackson
One thing I don't like about the MongoDB dashboard is that if I'm already
logged in and go to Tools, I still have to use commands to export data. Why
can't I just press a button in their UI and export the data however I want?
I'm already logged in. This might seem trivial but it would be faster.
Especially if you're working with someone semi-technical and who has only a
basic understand of looking through documents, you don't want to have to walk
them through different commands and using the CLI.

~~~
Alex-Potsides
I never tried exporting data using the Dashboard (always used mongo dump).

> One thing I don't like about the MongoDB dashboard is that if I'm already
> logged in and go to Tools, I still have to use commands to export data.

Man, that sucks. Why would they do that? Anyways, we will definitely have an
export button for AvionDB.

~~~
davidajackson
Yeah, I would say UI + analytics/filtering for documents built into the
dashboard are things that I wish existed. I know there's are lots of third
party tools but a lot of times in the early stages of projects I feel those
aren't yet a priority.

~~~
Alex-Potsides
We would love to build a Dashboard soon. It's actually in our Roadmap.
[https://github.com/dappkit/aviondb/issues/8](https://github.com/dappkit/aviondb/issues/8)

It would be great if you could join our Discord channel:
[https://discord.gg/88YpNuQ](https://discord.gg/88YpNuQ)

I would love to have your feedback on our Dashboard+what features that you may
want. I can get that up in a few days.

------
Alex-Potsides
Some useful links:

How to Install:
[https://github.com/dappkit/aviondb#install](https://github.com/dappkit/aviondb#install)

How to Use:
[https://github.com/dappkit/aviondb#usage](https://github.com/dappkit/aviondb#usage)

API Documentation:
[https://github.com/dappkit/aviondb/blob/master/API.md](https://github.com/dappkit/aviondb/blob/master/API.md)

Roadmap:
[https://github.com/dappkit/aviondb/issues/10](https://github.com/dappkit/aviondb/issues/10)

FAQs: [https://github.com/dappkit/aviondb#frequently-asked-
question...](https://github.com/dappkit/aviondb#frequently-asked-questions)

Contact Us on Discord:
[https://discord.gg/88YpNuQ](https://discord.gg/88YpNuQ)

------
vaultec81
Hey hello everyone. One of the Aviondb developers here. Feel free to ask me
any questions.

------
cryptoquick
Neat to see this made it to HN! Congrats.

So, my major concern with OrbitDB is the use of in-memory indexes to hashes.
It's a very naive approach for multiple reasons. It just doesn't scale well.
Node.js handles these particularly poorly, and develops memory leaks in my own
experiments with AOILs (append-only immutable logs) inspired by Secure
Scuttlebutt.

What I mean is, take a look at this code: [https://github.com/orbitdb/orbit-
db-kvstore/blob/master/src/...](https://github.com/orbitdb/orbit-db-
kvstore/blob/master/src/KeyValueIndex.js#L5)

And this: [https://github.com/orbitdb/orbit-db-
docstore/blob/master/src...](https://github.com/orbitdb/orbit-db-
docstore/blob/master/src/DocumentIndex.js#L5)

I'd suggest perhaps building that in SnapDB:

[https://github.com/ClickSimply/snap-db](https://github.com/ClickSimply/snap-
db)

Though, there are some really hard limits to file locks / mutexes that are
needed for a decent persistent K/V store. Since, pretty much any database is
going to need an index, and an index is just a K/V store. So, in essence, if
your platform can't be relied on to implement a good embedded K/V store, it
can't be relied on to create anything other than toy databases.

I considered adding SnapDB to OrbitDB, but it would've required a lot of
refactoring to support asynchronous lookup. I eventually realized I could do a
lot of what I wanted to accomplish without the elaborate CRDTs like OrbitDB
employs to keep an ordered log of information.

I would also add that, although it should be obvious, just in case it isn't,
AvionDB now builds on OrbitDB which itself builds on IPFS. So, that's a big
consideration for those looking to work with this.

(The below is mostly from personal experience and not terribly relevant to
AvionDB and OrbitDB)

I've been meaning to do a more formal post-mortem on the project, but haven't
gotten anything fully organized yet. I discovered all this after creating a
project a while back for what was essentially a P2P image gallery built with
IPFS and Electron. I wrote it in TypeScript, React (with Hooks), Redux Toolkit
(which makes Redux a lot less insufferable than it used to be, my god), and
SnapDB and OrbitDB (but not together). I also had go-ipfs running in a process
managed by the Electron client, and included it with the build.

As a result, the app image binary would generally be in excess of 500MB. And
this was just our 0.1.0, we hadn't gotten to the really good features yet. The
performance was decent in the UI, but the app itself couldn't scale very well
past 1000 images. go-ipfs was the real limiting factor at the time; it would
just fail in really bizarre ways, and I'd have to not only write code for
restarting the go-ipfs process in case it died, but also retry requests that
didn't go through. I suspected it might've been due to me having pubsub
enabled, but my ability to debug that was hamstrung by the fact that OrbitDB
hard fails without pubsub enabled. I haven't tried it with go-ipfs 0.5.0, I
think pubsub is enabled by default with that now, and I've already moved on
from that implementation.

My biggest concern in terms of performance aside from go-ipfs (which was
disappointing and unexpected), was the fact that due to the design, there
would be multiple copies of the same data in-memory everywhere: the in-memory
indexes used by OrbitDB, the data in Redux, the data in the DOM, etc. I'm able
to effectively integrate all of those using a Rust desktop binary. I'm using
Iced for the UI, and I've moved on from bundling IPFS entirely. It allows me
to build desktop and WASM binaries from the same codebase.

It's possible I could probably make the Electron app work better at this
point, but there are still the fundamental limitations I pointed out earlier.

For me, content-addressable hashes and IPNS signatures are perfectly fine. I
value performance (CPU, RAM, and disk) over anything else, primarily because
I'm developing P2P apps, and without a server, the nodes have to do all the
work itself, including the UI. That's why I moved off of TypeScript and into
Rust. PubSub would be nice, and OrbitDB has a decent implementation of Lamport
Clocks and CRDTs for decentralized ordering of events, but it's just often
overkill for most things that could just be content-addressable.

\---

Just to be clear, I've made a lot of claims here, and I'm hardly an expert on
any one of these things, so if I've gotten any of this wrong, I'd really like
to know more. Also, the code I wrote earlier isn't available to the public. If
I have any say in the matter, it will be locked away within an enormous
warehouse, Indiana Jones style, hopefully never to see the light of day again.

~~~
markhenderson
Hey cryptoquick, I remember you from a while back. I'm
[https://twitter.com/aphelionz](https://twitter.com/aphelionz), one of the
maintainers. Nice to see you :)

You're not wrong about the memory usage. We decided early on that the tradeoff
would be more RAM than to take the performance hit with the I/O (either to and
from IPFS or to and from the filesystem in the case of SnapDB).

That being said, I have been experimenting with removing indexes like the ones
you pointed out in favor of generators that emit the values as they are
traversed. There are two main efforts here, one is feasible and the other is
more difficult.

1\. The `entryIndex` inside of ipfs-log can probably go, and the ipfs-
log#traverse function can be made into an async generator function that passes
the oplog values up to the store 2\. The indexes you linked to that hold the
calculated STATE are harder to get rid of - they could be persisted to IPFS or
the file system as well

Open to ideas on #2. SnapDB might work for all I know, as I haven't attempted
it yet.

I can't speak to Electron, React, Redux, Iced, etc, but my guess is there are
optimizations one can do there as well.

~~~
markhenderson
Also, speaking of Rust - we'd love more contributors over at
[https://github.com/ipfs-rust/rust-ipfs/](https://github.com/ipfs-rust/rust-
ipfs/).

------
uberdru
I don't get it. How is this decentralized?

~~~
Alex-Potsides
Hi @uberfru

AvionDB is built on top of OrbitDB[1] which implements the core decentralized
database logic/protocol, including CRDTs, Lamport clocks, immutable logs, etc.

If you want to see for your self and dive a bit into how it works under the
hood, you can check OrbitDB's welcome page[2] or field manual[3]

1: [https://github.com/orbitdb/orbit-db](https://github.com/orbitdb/orbit-db)

2: [https://github.com/orbitdb/welcome](https://github.com/orbitdb/welcome)

3: [https://github.com/orbitdb/field-manual](https://github.com/orbitdb/field-
manual)

------
jason0597
Another database? I feel like there are too many database solutions nowadays.

~~~
eV6ahne6bei
The problem is with quality not quantity.

And the quality is severely lacking in most databases.

~~~
Alex-Potsides
Just out of curiosity, what are the parameters that decide the quality of a
Distributed Database Project?

