
Show HN: Minimalist Object Storage Project - y4m4b4
https://github.com/minio
======
acdha
The first questions which I look for on objects stores are how it approaches
data integrity – things I'd like to see on the homepage:

1\. Does it have the S3/Swift/etc. semantics that you will always get the bits
first stored or an error but never a corrupt or truncated file?

2\. Does it have the S3 and (IIRC) Swift feature that allows you to provide an
expected hash so data corrupted in transit will be detected and an error will
be returned instead of storing the object?

3\. How does it verify data integrity? Only on access, background scrubs, via
API request, etc.

4\. Which strong hashes are used for all of the above? Are they configurable /
selectable so e.g. if I wanted something strong than S3's Content-MD5 header
for uploads can I use SHA-256 instead?

5\. The classic easy-to-ask, hard-to-answer one: what level of testing backs
up all of the previous answers?

~~~
y4m4b4
> 1\. Does it have the S3/Swift/etc. semantics that you will always get the
> bits first stored or an error but never a corrupt or truncated file?

Yes you are right the behavior is exactly similar to how an S3 API would
behave for Get/Put().

> Does it have the S3 and (IIRC) Swift feature that allows you to provide an
> expected hash so data corrupted in transit will be detected and an error
> will be returned instead of storing the object?

Yes, just like how Content-MD5 is set through a client request for per uploads
in S3, Minio honors that as well.

Minio provides three modes of operation

\-- memory (in memory object storage)

\-- filesystem (point to any directory or disk, all files are contigous and
not erasure coded)

\-- donut (experimental, distributed erasure coded on disk format)

In Donut individual parts or chunks are separately sha256 summed along with
erasure coding on top of these individual parts.

[https://github.com/minio/minio/tree/master/pkg/storage/donut](https://github.com/minio/minio/tree/master/pkg/storage/donut)

Donut also implements rubberband erasure which is a dynamic erasure coding
style where in cumulative parities are created for all objects depending on
the available disks, total usage etc.

> How does it verify data integrity? Only on access, background scrubs, via
> API request, etc.

Donut also does data integrity checks on a regular basis as an operational
process, where even silent data corruption is fixed automatically without the
need for any user intervention.

We are in early stage of our project, as of today Donut is experimental -
currently runs on a single node, does sha256 sums for erasured chunks and
erasure codes your data on disk.

Much of the operational process and distributed donut is a work in progress,
we are planning to get them ready by end of this year.

> Which strong hashes are used for all of the above? Are they configurable /
> selectable so e.g. if I wanted something strong than S3's Content-MD5 header
> for uploads can I use SHA-256 instead?

Yes while Content-MD5 is what we do now, along with erasure coding and sha256
summing individual chunks. We can provide the similar semantics of ``x-amz-
content-sha256`` from client side. We would be happy to implement it feel free
to open a bug -
[https://github.com/minio/minio/issues](https://github.com/minio/minio/issues)

> The classic easy-to-ask, hard-to-answer one: what level of testing backs up
> all of the previous answers?

I agree with hard-to-answer :-). Minio is written in Golang, which largely
drives us to "write tests first" style and we have rightfully followed it on a
day to day basis. Not being limited to that we are also working towards
building a continous integration testing framework which would be baked into
the product, which would help any user perform self diagnostics of his/her
setup. In-turn generating reports for quality of their setup, any unknown
parameters which might show up etc.

Also testing is an ongoing process, so we are striving to be better everyday
:-)

Thanks for the questions, feel free to hop on to - gitter.im/minio/minio and
find us there.

~~~
acdha
Thanks for the reply – it sounds like you're definitely hitting the areas I
care about.

One quick thought: I was at USENIX the year that ZFS was announced and one
thing which was _very_ popular was when the Sun team talked about how much
random testing they'd done with billions of runs exercising the major features
with simulated failures, memory corruption, etc. so they could stress-test the
detection and recovery mechanisms:

[https://blogs.oracle.com/bill/entry/zfs_and_the_all_singing](https://blogs.oracle.com/bill/entry/zfs_and_the_all_singing)

You could see a lot of sysadmins nodding their heads as they described each
class of failure mode and how they were trying to simulate it.

These days I'd probably toss in something like running afl-fuzz against your
storage code, too.

~~~
y4m4b4
> Thanks for the reply – it sounds like you're definitely hitting the areas I
> care about.

Thank you, appreciate the inputs :-)

------
fake44637
I'm new to the concept of object storage. Is the thing which distinguishes
object storage from key-value storage the ability to also store metadata?

I.e., Instead of providing (key: value), it provides (key: (metadata, value))?

~~~
acdha
I've seen this an an outgrowth of the file storage world:

1\. Flat data model – i.e. no directories, just a single globally-unique
identifier 2\. Much larger sizes than many key:value stores are designed to
handle, typically bounded only by physical storage 3\. Updates are complete
and atomic: you don't have to support all of the crazy permutations like
streaming writes from a file handle which could be kept open for years,
dealing with simultaneous writes, etc. which make life interesting for
filesystem implementers 4\. None of the more complicated filesystem behaviour
like locking, alternate data streams, etc. which add so many edge cases

------
fiatjaf
I don't understand what is this about.

~~~
kelvin0
Why the downvotes? I am genuinely curious also, but there is next to nothing
in terms of details (uses cases and such) on their site!

~~~
y4m4b4
We don't have any yet, we are just few months old project. We will will have a
distributed server by end of this year.

We have stand alone server now which includes currently three modes of
operation

\-- memory

\-- filesystem

\-- donut(experimental)

A small video to help you download and start a memory mode -
[https://asciinema.org/a/21575](https://asciinema.org/a/21575)

Client tools and libraries are ready to be used by community to give us
feedback on.

\- [https://github.com/minio/mc](https://github.com/minio/mc) \- Minio client
collection of tools like `ls`, `cp`, `sync`, `cat`

\- [https://github.com/minio/minio-go](https://github.com/minio/minio-go) \-
Golang

\- [https://github.com/minio/minio-js](https://github.com/minio/minio-js) \-
Nodejs

\- [https://github.com/minio/minio-java](https://github.com/minio/minio-java)
\- Java

~~~
jahewson
What's "donut" storage?

~~~
y4m4b4
Donut - is a distributed erasure coded backend format, designed specifically
for immutable data.

We call it "do not delete" in short donut.

[https://github.com/minio/minio/tree/master/pkg/storage/donut](https://github.com/minio/minio/tree/master/pkg/storage/donut)

It is currently experimental, but a distributed version is a work in progress.

------
fweespeech
While the concept is interesting, it hasn't built the only real detail that
matters from an Object Store perspective:

Cheap, Multi-Datacenter, Performant, Highly Available storage.

"YAS3I" / Yet-Another-S3-Implementation isn't really compelling even if it is
open source.

I can pull that out of
[https://wiki.openstack.org/wiki/Swift/APIFeatureComparison#A...](https://wiki.openstack.org/wiki/Swift/APIFeatureComparison#Amazon_S3_REST_API_Compatability)

if I really need that.

~~~
y4m4b4
We are in an early stage, we will have a distributed version by end of this
year.

It isn't about "YAS3I". It isn't compatible just for the compatibility sake.

Minio implementation is opinionated: "There is only one way to do it". Given a
choice between Amazon S3 and OpenStack SwiftAPI compatibility, we picked
Amazon S3. As SwiftAPI gains adoption, we may switch over to SwiftAPI, but
will have to drop S3 then. Let the community decide.

~~~
vruiz
Tiny piece of advice, declaring a possible total revamp of the API as soon as
you launch might not be the best way to attract early users.

~~~
y4m4b4
Advice accepted. The choice is for the community to make, we will not make
that choice unless community strongly wants us to do so.

------
jamescun
What specifically separates this from Swift or Ceph, both of which have S3
compatible APIs and are considerably better documented/hardened/maintained +
distributed?

~~~
y4m4b4
Minio is about minimalist design, take a look at the blog post to get an idea
[https://medium.com/@minio/minio-no-knobs-to-turn-no-
button-t...](https://medium.com/@minio/minio-no-knobs-to-turn-no-button-to-
push-196f62754403)

At this early stage of the project it is hard to explain why another approach
is necessary.

------
jarito
Why not just use Swift?

[http://docs.openstack.org/developer/swift/](http://docs.openstack.org/developer/swift/)

It can be run as a stand alone app so it's generally pretty simple and can
include support from [https://swiftstack.com/](https://swiftstack.com/).

~~~
jimbokun
First time I'm seeing Swift.

Are there use cases where Swift is a better fit than Cassandra?

~~~
neverartful
Swift is part of the larger family of OpenStack. Swift should provide (haven't
verified this myself) better integration with all of the other OpenStack
components (Keystone, Cinder, Neutron, etc.).

~~~
j05h
Swift can integrate with OpenStack, but you can very easily run it stand
alone, and many do. It has erasure coding, geo replication, and its a fairly
well baked product.

Here are good overviews:

* [http://docs.openstack.org/developer/swift/](http://docs.openstack.org/developer/swift/)

* [https://swiftstack.com/blog/2012/09/16/globally-distributed-...](https://swiftstack.com/blog/2012/09/16/globally-distributed-openstack-swift-cluster/)

And how Rackspace runs it at scale;

* [https://www.openstack.org/summit/openstack-summit-hong-kong-...](https://www.openstack.org/summit/openstack-summit-hong-kong-2013/session-videos/presentation/an-intimate-look-at-running-openstack-swift-at-scale)

