
Ethereum Sharding FAQ - dmmalam
https://github.com/ethereum/wiki/wiki/Sharding-FAQ
======
pavlov
Something about Ethereum feels like a 19th century perpetual motion machine.
"Ok, so the overbalanced wheels didn't quite work, but we'll add ball bearings
and flowing water and gears and..."

In 1872, a man named John Keely demonstrated a machine that operated on what
he called the "vaporic" or "etheric" force:

[https://en.wikipedia.org/wiki/John_Ernst_Worrell_Keely](https://en.wikipedia.org/wiki/John_Ernst_Worrell_Keely)

The same year he raised 5 million dollars, equivalent to about $100M in
today's money. Two years later he demonstrated a model that was described as a
"hydro-pneumatic-pulsating-vacu-engine" with "quadruple negative harmonics".
No doubt he'd be a blockchain entrepreneur today.

(Edit: Someone actually maintains a big wiki dedicated to Keely's "Sympathetic
Vibratory Physics" and "Etheric Elements": [http://www.svpwiki.com/tiki-
index.php?page=Sympathetic+Vibra...](http://www.svpwiki.com/tiki-
index.php?page=Sympathetic+Vibratory+Physics) Amazing that an investment fraud
from 150 years ago is still producing "research"! A sobering reminder of the
power of lies.)

~~~
jessriedel
Perpetual motion machines are disfavored because they break a law of physics,
not merely because they involve a stack of complicated mechanisms, each of
which fixes a problem introduced by the mechanism below. If you've ever had a
complicated medical problem, you know that very often you get drug A to
counteract side effects of drug B, which was used treat an infection caused by
procedure C which was a necessary precaution before performing treatment D to
treat the underlying illness. Yes, sometimes such complicated mechanisms are
obfuscating a doomed process, but other times engineering is just complicated.
The Saturn V had a lot of interdependent parts too...

~~~
cocktailpeanuts
I would say the "Proof of Stake" they're planning to move towards also defies
the law of physics.

~~~
Hermel
Intuitively, I think "proof of stake" could violate Gödel's incompleteness
theorems. These basically state that no complete system can proof its own
correctness. And that's exactly what "proof of stake" tries to do: proof its
correctness without external anchor of truth.

Of course, Vitalik Buterin knows that so he resorted to "weak subjectivity",
basically saying that if you are not online often enough, you need to trust
someone with providing you check points to decide which fork is the "true"
one.

~~~
tuesdayrain
To me the logic behind Proof of Stake is more like "so much money is bet on
the answer being correct, that we'll assume it is. If the majority is wrong
then feel free to permanently leave this chain and start your own". I consider
it to be a superior system to Proof of Work in nearly every way.

~~~
tfha
The problem with that is that you only know that 'so much money' is being bet
because you know who owns the money. But you only know who owns the money
because that same money was bet earlier by people that you know owned the
money because the same money was bet earlier by...

It's completely circular. It's like the Bible trying to proof itself. With no
anchor to the physical world, you can't actually ever know

~~~
zodiac
It's not completely circular. For example in casper-ffg if you as a full node
log in every 4 months starting from ethereum's genesis block (about 3 years
ago), the chain of logic you outlined terminates at the genesis allocation and
PoW rewards.

I'm not saying that requiring clients to log in every 4 months is a reasonable
security assumption, just showing that hand-waving it as "completely circular"
is incorrect. It's more nuanced than that.

~~~
tfha
How did the node get started in the first place though? Unless you were part
of the genesis block yourself, you only believe the history at all because
someone told you it was the right place to start.

~~~
zodiac
In full PoS in the model I described you'd need social consensus to determine
the allocations in the genesis block. In PoW you need social consensus on the
genesis block as well - the PoW algorithm, the initial difficulty and block
reward, the difficulty adjustment algorithm, the block reward schedule

------
seibelj
One theory I have on the hate towards Ethereum (and cryptocurrencies /
blockchain in general) on HN is that it's a radically different way of solving
problems, something that is alien to software engineers who have put decades
of time becoming experts at solving problems a certain way. So rather than
invest more than 15 minutes learning about it, they say "Blockchain is just a
distributed database and it's useless!" or attack it from a variety of
minimum-thought / minimum-effort angles.

There is genuine innovation in this space and it is going to impact a lot of
areas. Decentralization, immutability, protocols monetized with market forces,
fee-less payments that cannot be prevented, to name but a few interesting
areas.

And of course there are numerous downsides. Difficulty of keeping keys safe,
bugs that cost millions of dollars, issues with scaling.

But these are problems that people are solving right now. As a technologist,
you should be looking at the possibilities! Now is the time to start a new
company or project, not 5 or 10 years from now when the tech has matured.

~~~
dmitriid
As a technologist I look at vaporware and call it vaporware.

I look at get-rich-quick schemes and call them get-rich-quick schemes.

I look at how blockchain don't solve a single problem that hasn't already been
solved more efficiently on much grander scales, and I call them out as such.

Your words, and images of grandeur, and promises of great things to come are
just that: words, and images, and promises, and I call them as such.

Blockchain is not "the next Internet that haters cannot see", as the overused
saying goes. So far it's nothing but the next Tomagotchi or Pets.com, and
_nothing_ that's happening in blockchain can prove it otherwise. And so, we
call it as such.

~~~
jMyles
> I look at how blockchain don't solve a single problem that hasn't already
> been solved more efficiently on much grander scales, and I call them out as
> such.

This position is the most confusing of all the blockchain FUD out there. It's
easy to show that it's false: today, people are able to purchase proscribed
plants and medicines in a way that completely subverts the efforts of state to
prevent that.

Now, I'm not saying that's sufficient to prove the kind of meaning you're
looking for, but doesn't it give you the slightest bit of pause? Doesn't it
make you say, "Wow - that's something people have been trying to do since the
garden of eden and nothing has ever done it so successfully. I wonder what
else this thing can do?"

I think that for most open-minded observers, something like this thought
process is what keeps our attention.

I think that undermining drug prohibition is a wonderful thing and an obvious
net good for the world. But I also think that it's a sign that there are more
mundane / lawful-good elements that will arise in time.

Porn has largely driven the internet forward at various critical junctures.
But if we had stopped and said, "oh, is this thing just about porn? How
useless," then we'd be in a much worse place.

Give it some time and an open mind.

~~~
Sangermaine
>people are able to purchase proscribed plants and medicines in a way that
completely subverts the efforts of state to prevent that.

No it doesn't. This is the kind of absurd nonsense that earns blockchain
proponents this level of scorn.

People could buy drugs before. Sure they can use cryptocurrency instead of
cash, but the old problems remain: getting the product from the buyer to the
seller. The state can, and sometimes does, prevent that because until
teleportation is invented getting the drugs you bought online into your hands
opens you to risk. Buying drugs is certainly easier, but it doesn't help to
talk in these ridiculous terms. I bought pot in college just fine. If I buy
drugs and have it shipped to me I'm at the same risk whether I used cash or
Bitcoin.

>"Wow - that's something people have been trying to do since the garden of
eden and nothing has ever done it so successfully. I wonder what else this
thing can do?"

You've got this backwards. It's been 10 years. People have been asking all
that time "so what else can you do", and the answer so far is...nothing,
really. This despite the relentless, utopian hype all during that time.

If proponents want people to keep an open mind and not be so cynical, maybe
they shouldn't have spent years promising that blockchain technology is the
digital coming of Jesus that will completely change everything in every way
and usher in an endless golden age of peace and prosperity.

~~~
stale2002
How do you send cash to your drug dealer on the internet, instantly?

I agree with you that you still have to get the product to the person, and
blockchain doesn't solve this. But it doesn't claim to.

What cryptocurrencies do is solve the payment part of it, so that you can send
money to someone without your bank finding out and freezing your account.

If I send money to coinbase, buy Bitcoin, and then convert that to Monero, the
only thing my bank knows is that I sent money to coinbase. And coinbase only
knows that I sent money to a different exchange.

~~~
dmitriid
> How do you send cash to your drug dealer on the internet, instantly?

By instantly you mean "up to 200 hours"? [https://blockchain.info/charts/avg-
confirmation-time](https://blockchain.info/charts/avg-confirmation-time) (at
the time of writing this comment the spoke on the chat was at 12000 minutes).

> the only thing my bank knows is that I sent money to coinbase. And coinbase
> only knows that I sent money to a different exchange.

Money laundering is as old an art as the banking services themselves. I guess
blockchain made it easier then? True innovation.

~~~
stale2002
By instantly I mean, on average around 10 minutes, if you pay an appropriate
fee. This is miles ahead of credit cards, for example, as that takes days to
clear. Yes they do, as the transaction isn't confirmed until days later.

Obviously, if you pay too low of a fee, it could take a long time.

> Money laundering is as old an art as the banking services themselves. I
> guess blockchain made it easier then? True innovation.

Indeed it does. The whole point of bitcoin was to enable censorship resistant
financial transactions.

And it turns out, that the people who most want to have access to censorship
resistant transactions, are the people who are CURRENTLY being censored by the
existing financial system.

I have no idea how I'd be able to quickly send cash to someone on the other
side of the world, in a way that is censorship resistant. Bitcoin enables
this, though.

Obviously, though, if you don't care about fighting financial censorship, then
you wouldn't see the usecase for a toll that helps you make censorship
resistant transactions.

It is perfectly fine for you to not care about subverting financial
censorship. But you should be aware that there are other people with different
values, who care a whole lot about this cause.

------
goodroot
Please HN...

Can we approach this with an open-mind? These technologies have major promise.
I do not see the use in being reactively closed mind and negative towards the
efforts here. As an Ethereum - not a "crypto" \- supporter, I feel brow-beat
each time one of these threads comes up.

Let's take a deep breath, explore it, and try to help it realize its potential
instead of instilling doubt and criticizing. This is a community for
forwarding thinking builders...

~~~
barbegal
On a positive note sharding technology does have great promise because of the
problems that it can potentially solve. With Bitcoin you can only validate a
transaction by having all of the unspent transactions in memory [1]. With a
sharded system you only need to have a subset of those transactions in memory.

At a fundamental technological level you are increasing the size of available
memory by using the memory of other nodes. And there are ways you can query
other nodes about their subset of unspent transactions by using a merkle tree.
If you trust the root of a balanced merkle tree you only need O(log(n))
computations [2] to prove that a leaf belongs to it.

This sharding mechanism could be added to bitcoin where the root of the merkle
tree of the unspent transactions is added to each block. Nodes on the network
can store a sub-tree of arbitary size and validate part of each block. Whether
this scheme would be robust to dishonest miners is still an open question
though.

[1] [http://gavinandresen.ninja/utxo-uhoh](http://gavinandresen.ninja/utxo-
uhoh) [2]
[https://gist.github.com/maaku/41b0054de0731321d23e9da90ba4ee...](https://gist.github.com/maaku/41b0054de0731321d23e9da90ba4ee0a#inclusion-
proofs)

~~~
hobofan
> With Bitcoin you can only validate a transaction by having all of the
> unspent transactions in memory.

I'm somewhat sceptical of that claim, for the reason pointed out in the blog
post, and I think you can get away with much simpler solutions for quite some
time.

\- You can use many heuristics to decide which UTXO data to keep in memory.
LRU + deprioritizing addresses where the balance is too low to pay the current
transaction fees seems like a good start. Research shows that there is a high
percentage of stale data[0]. \- With a block time as high as 10min, you can
definitely spend 1ms on read latency for retrieving UTXO data from disk. \-
Even if you don't manage to retrieve UTXO data from disk quickly, that just
means that the transaction missed its chance to be included in the current
block. It could still be ready in time for the next one.

Overall, UTXO set growth is, of course, a problem since it increases
centralization, but I don't think the problem of _in-memory_ UTXO sets is as
big as it is often made out to be.

[0]:
[https://eprint.iacr.org/2017/1095.pdf](https://eprint.iacr.org/2017/1095.pdf)

------
hardwaresofton
I'm really really really not qualified but:

 __TLDR __, split up a blockchain into a bunch of smaller blockchains sharded
by something (let 's say address), and then a different tier of nodes which
only operate at the shard level can exist -- they call these "collations".
collation-internal transactions can be managed by only the nodes in the
collation, and they even lay out an initial proposal for how to do cross-
collation transactions, leaning on merkle proofs for verification. They also
propose making it transparent as a part of an auto-loadbalancing part of the
blockchain so developers wouldn't feel any difference.

This paper was actually pretty interesting. It's pretty thoroughly researched,
and the solution that they come up with (sharding) is actually pretty
promising as a solution to the centralization + throughput issues of the
blockchains up until now. I certainly learned a lot about blockchain systems
by reading it, and more importantly learned the current problems with the
state of the art and how this group plans to address them.

At this point, I also look at blockchain as a generic distributed database
problem -- so I get the feeling these finndings will benefit the field some
day. Surely something will come out of the research energy that's being
dedicated to blockchain right now that will be beneficial to future
researchers.

~~~
dlubarov
That's pretty accurate. One minor thing:

> developers wouldn't feel any difference

The details haven't been decided yet, but developers will need to deal with
some kind of locking or "yanking" [1] mechanism in order to communicate with
accounts on foreign shards.

[1] [https://ethresear.ch/t/cross-shard-contract-
yanking/1450](https://ethresear.ch/t/cross-shard-contract-yanking/1450)

------
barbegal
The section I particularly have a problem with is "How is the randomness for
random sampling generated?" Everyone seems to think this is a solved problem
but the reality is it is extremely difficult to solve. (The same problem
exists for proof of stake schemes)

There is lots of complex cryptography, game theory and economics involved in
working out if a robust solution really exists. The most commonly stated
solution is something called a "Threshold Signature Scheme". Threshold
signatures schemes (and threshold cryptography in general) have two main
problems.

Firstly they don't scale well. As you add participants you need lots of extra
rounds of communication and computation.

Secondly they aren't resistant to Sybil attacks. The proposed way around this
is to link participation to a stake of coins (yep back to proof of stake) such
that launching a Sybil attack requires lots of coins. This is not trivial
however because of the "Nothing at Stake" problem where stake can be used to
participate in many forks of a scheme at the same time.

~~~
drcode
Ethereum POS is already running on a test net, you're welcome to go over there
and see what happens when you try to use your stake on "many forks at the same
time". Spoiler: you won't be able to do so without losing your stake very
quickly.

~~~
barbegal
Well not quite. The test net is actually a hybrid of proof of work and proof
of stake [1]. Proof of work is used as the main decider in the case of a fork,
however if long forks are seen then checkpoints validated by proof of stake
are used.

The slashing function is in the contract [2] but there is actually no
incentive to duplicate votes on the testnet since there is only rewards for
voting (block rewards are given to proof of work miners).

[1]
[https://github.com/ethereum/casper/blob/master/IMPLEMENTATIO...](https://github.com/ethereum/casper/blob/master/IMPLEMENTATION.md)

[2]
[https://github.com/ethereum/casper/blob/master/casper/contra...](https://github.com/ethereum/casper/blob/master/casper/contracts/simple_casper.v.py)

~~~
drcode
Don't totally follow your argument, but if you're saying that we won't have a
full understanding of the POS economic guarantees until we have empirical
evidence from a "live network" with significant rewards at stake I agree 100%.

------
homakov
Interestingly Ethereum has hit in a sweet spot: tech crowd that learns EVM
quickly and enjoys playing with it (like a new raspbery pi) & has sizable
spare cash to invest in ether/erc20. Bitcoin is nowhere as fun to learn, the
VM is quite boring. I wonder if any other blockchain can hit into this sweet
spot.

------
ilaksh
I personally think that sharding is going to work great because for starters
100% trustlessness was, in my opinion, never a real requirement. If it was
then nothing in our society would work.

People are going to look back and say that the HBO show 'Silicon Valley' was
'prescient', but really they are just catching up to developments in Ethereum
and similar projects.

Long live the real PiperNet (well they have some significant differences but
also some similarities).

------
cocktailpeanuts
Good for them for nerding out with all the esoteric tech to solve the
decentralization problem.

But evidently all this means nothing if it's possible to mutate the blockchain
itself through social and political engineering.

~~~
spacenick88
But if that mutation needs majority (from miners or stake holders) how is it
any worse than a democratic country deciding to change a law or print money or
whatever?

~~~
gph
Wasn't the main part of the allure to blockchain/crypto to be better than
democratic societies and institutions? After all that's what they are trying
to replace right? Governments can't be trusted and all that?

------
boffinism
'FAQ' seems like the wrong term. I doubt that "What are some trivial but
flawed ways of solving the problem?" is frequently asked...

~~~
hobofan
Eh, having had quite some discussions on the topic, a lot of the "flawed ways"
come up as "why don't we just do this?" suggestions.

------
voidmain
I think maybe you could improve on this sort of approach by weakening the
invariants. Instead of trying to make sure that only valid "collations" get on
the main chain, let anyone willing to pay for it put collations on the main
chain but define the first honest one on the chain to be valid. You can't tell
which ones are valid at a glance but you can with O(shard history) resources.

~~~
DennisP
If you can figure out the details on that, they'd probably be interested.

------
mst
Having just attempted to read and comprehend this, my total conclusions are:
(a) I have a headache (b) I have absolutely no idea if this is going to work.

(if anybody has more useful conclusions, I'd love to hear them, most of the
threads seem to be alternating "blockchain yay" and "blockchain boo" comments,
neither of which tend towards being desperately informative)

~~~
oceanman888
I feel the same, the arguments about blockchain is one of the more fruitless
one about technology(scams and over-hype makes it worse). Meanwhile the doers
is trying to figure out how to make it useful. Once some infrastructure and
successful examples are done(For example block chain collection done right or
oracles that publish user or sensor data for other smart contract to use at a
cost). We will have much more stuff to start arguments from.

------
thisisit
I wonder what effect PoS will have on the Ethereum ICO system. If coins are
needed to stake and mine, large holders might have more incentives to hold
Ethereum instead of any assets?

------
decentralised
Sharding will enable Ethereum to scale without compromising on the
decentralised and permissionless nature of it's blockchain implementation.
Really excited about this!

------
Animats
What's the need for a high transaction rate? "10,000 TPS". Right. Nobody uses
cryptocurrencies for small transactions any more. It's just a speculative
investment.

Bitcoin being stuck at 7 TPS is embarrassing, but if they could get up to a
few hundred TPS, that would probably be adequate.

~~~
immad
Ethereum is also a DAPP platform.

In theory you could build a decentralized Twitter/Facebook using Ethereum but
without 10k+ TPS there is no practical way to get enough users using it.

~~~
stjohnswarts
Honest questions why would you need a blockchain for this? I can understand
for money transactions/contractual signing but for Facebook? Does anyone
really care that there is a ledger for Facebook?

------
DINKDINK
Sharding: Putting the trust in trustless

/s

------
markkat
TBH part of what makes me so bullish on Ethereum is how much of an emotional
reaction it produces, both for and against the technology, all the while it
continues to gain mindshare and utility.

~~~
doubledad222
I've had a couple friends try to enroll me into their ponzi schemes. They were
really emotional both during the sale and after I remained in "no". I think
emotions are just the communication of the base animal part of people. No need
to imbue it with prescience or exceptional intelligence.

~~~
markkat
Ponzi schemes are terrible. Many ICOs aren't much more. However, blockchain
enables digital scarcity. That's an interesting piece of technology right
there. Using it as a state-machine is a very interesting application of it.
IMHO many negative emotional responses come from scamming applications, many
positive ones from new possibilities created by the technology, some of both
are just due to the transformative nature of its application.

~~~
spookthesunset
> However, blockchain enables digital scarcity.

Eh? Scarcity? Ever looked at all the forks of bitcoin?

Blockchain absolutely does _not_ enable scarcity. This has been proven time
and time again...

~~~
tuesdayrain
You say that as if Bitcoin forks are unquestionable proof of increasing
supply, which many people will disagrewle with. I consider them to be
completely separate products, seeing as the forks are not backed by Bitcoin's
hashpower.

~~~
root_axis
It doesn't matter what someone "sees" them as, the fact remains that forks are
possible because the scarcity is an artificial illusion.

------
Density
Sorry for being ridiculously inappropriate, but the cryptocurrency topic calls
for me to leave this more appropriate title here:

Ethereum Sharting FAQ.

Have a nice day!

