
Scaling Services with Shard Manager - vquemener
https://engineering.fb.com/production-engineering/scaling-services-with-shard-manager/
======
rubiquity
This seems very similar to Google's Slicer system[0][1] which was published in
2016.

> The concept of using sharding to scale services is not new. However, to the
> best of our knowledge, we are the only generic sharding platform in the
> industry that achieves wide adoption at our scale.

In typical Facebook engineering parlance they have to claim to be the first
ones ever to do it. Fragile egos over there.

Putting hubris aside, I think it's great that a decade or so into large scale
computing we're starting to see patterns emerge for scaling stateful systems
and be able to build good generic solutions to them. This is sorely needed
especially on the control plane side which historically hasn't gotten the
attention that data planes have.

0 -
[https://www.usenix.org/system/files/conference/osdi16/osdi16...](https://www.usenix.org/system/files/conference/osdi16/osdi16-adya.pdf)

1 - [https://blog.acolyer.org/2016/12/02/slicer-auto-sharding-
for...](https://blog.acolyer.org/2016/12/02/slicer-auto-sharding-for-
datacenter-applications/)

~~~
fizx
Introducing Gizzard, a framework for creating distributed datastores

[https://blog.twitter.com/engineering/en_us/a/2010/introducin...](https://blog.twitter.com/engineering/en_us/a/2010/introducing-
gizzard-a-framework-for-creating-distributed-datastores.html)

~~~
thawan
Thank you for pointing us to Gizzard. Based on our understanding of Gizzard’s
blog, we would like to highlight the key differences as follows.

The key advantage of Gizzard is that it divides the data into key ranges and
treats it as a shard. If a shard is too hot, it can dynamically adjust the
size of the range. Gizzard also acts as a traffic routing layer that sends
requests to the right server and coordinates data replication.

Shard Manager (SM) uses fixed-size shards and uses shard migration to reduce
load on hot servers. Unlike Gizzard, SM focuses on being a control plane
service and client traffic does not flow through SM. That functionality is
delegated to Facebook’s separate traffic routing framework. SM supports more
generic sharding applications and uses add/drop shard protocol to coordinate
state migration. Gizzard focuses on data store applications and requires write
operations to be idempotent and commutative in order to support state
migration.

------
jvikstrom
This seems very similar to slicer [0], no? At least the idea behind the system
(to have a general purpose sharding "thing"). Or have I misunderstood what
slicer's used for?

[0] -
[https://research.google/pubs/pub46921/](https://research.google/pubs/pub46921/)

