

Are we consistent yet? Eventual consistency of cloud object stores - SwellJoe
https://github.com/andrewgaul/are-we-consistent-yet

======
jayp
What does it mean by "observed instances of eventual consistency"? Shouldn't
it be close to 100% given enough time? The claim of zero for 100k operations
is jaw dropping (and requires more evidence, or simply, a better explanation
of the experiment).

~~~
jcrites
The author ran a test in which he wrote an object and then attempted to read
the object. Across many trials, he counted the number of times the object was
not immediately available.

When the object is not immediately found, it's an occurrence of observed
eventual consistency. In a strongly consistent model, that will never happen
(by contract). The author generalized the experiment across creating,
updating, and deleting objects, followed by either reading or listing objects.

Operations on eventually consistent systems can behave in an outwardly similar
way to strongly consistent systems in the average or happy case: when they
manage to reach consistency faster than your application can follow up with
its next request. The consistency model tells you whether this is
_guaranteed_.

A number of eventually consistent systems do a pretty good job of providing
apparent strong consistency in the typical case, such that eventual
consistency only occurs when recovering from a hardware failure or network
partition.

~~~
jayp
Thanks Justin.

Ahh -- "observed instances of eventual consistency" as in "able to observe
poorer than strong consistency". I was imagining it as able to eventually
confirm __consistency__. However, was dumbfounded by the lack of information
on delay before check.

I wonder if the experiment normalizes for the client origin. The farther the
client is from the actual server, the more time it takes to do a round trip,
and less chance for an observed instance of eventual consistency.

~~~
khc
I helped Andrew ran the S3 tests. For us-standard bucket, we used an m3.medium
instance in the us-east region, and for us-west bucket, we used a t2.micro
instance in the us-west-2 (Oregon) region.

It is also possible that S3 somehow tries to pin requests from the same client
to the same storage nodes. A better test setup would coordinate writing and
reading across multiple clients, but is beyond the scope of this experiment.
[http://www.researchgate.net/publication/259541556_Eventual_C...](http://www.researchgate.net/publication/259541556_Eventual_Consistency_How_soon_is_eventual_An_Evaluation_of_Amazon_S3%27s_Consistency_Behavior/links/0deec52c6e04b49921000000.pdf)
describes such a system and their results.

~~~
jcrites
Cool! Thanks for the research and experiment, and for following up on HN. I
appreciated the attention to detail that was paid to documenting and
explaining the consistency models.

Be careful about running experiments from "t"-class instance types. Due to
their ability to "burst" and consume extra CPU, the amount of system resources
available to them is not consistent. This poses an experiment design challenge
of ensuring that available system resources do not vary. It may be easier to
run reliable experiments from another type.

> Traditional Amazon EC2 instance types provide fixed performance, while T2
> instances provide a baseline level of CPU performance with the ability to
> burst above that baseline level. They are intended for workloads that don't
> use the full CPU often or consistently, but occasionally need to burst.

[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/t2-instan...](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/t2-instances.html)

I am not sure if this will have meaningfully affected the accuracy of the
results (probably not), but I thought I'd point it out for future reference.

~~~
khc
Yup I am aware. For reference, the test took about 1000 minutes on the
t1.micro (I misspoke earlier) and about 642 minutes on m3. I didn't test to
see if the difference was due to machine class or region. Note that each run
has 6 tests, each test is repeated 1000000 times and involves multiple
requests.

