
Entropy explained, with sheep (2016) - mlejva
https://aatishb.com/entropy/
======
PaulDavisThe1st
This was really great. It's how I was taught entropy at college (biophysics
and molecular biology) though without the sheep. "Statistical Mechanics" was
the name our professor used.

The only tiny change I'd like to make is to add a line or two near the end,
something along the following lines:

    
    
      There's a lot fewer ways to arrange water molecules so that they form an ice cube than there are to arrange them as a liquid. Most arrangements of water molecules look like a liquid, and so that's the likely endpoint even if they start arranged as an ice cube.
    
      The same is true of more or less any macroscopic object: the thing that we recognize and name ("chair", "table", "pen", "apple") requires the atoms to remain in one of a fairly small set of particular arrangements. Compared to the vast number of other arrangements of the same atoms ("dust"), the ones where the atoms form the "thing" are quite unlikely. Hence over time it's more likely that we'll find the atoms in one of the other ("random", or "dust-like") arrangements than the one we have a name for. The reason things "fall apart" isn't that there's some sort of preference for it - it's that there are vastly more ways for atoms to be in a "fallen apart" state than arranged as a "thing".

~~~
kgwgk
This kind of intuition can be dangerous:

"Students who believe that spontaneous processes always yield greater disorder
could be somewhat surprised when shown a demonstration of supercooled liquid
water at many degrees below 00 C. The students have been taught that liquid
water is disorderly compared to solid ice. When a seed of ice or a speck of
dust is added, crystallization of some of the liquid is immediate. Orderly
solid ice has spontaneously formed from the disorderly liquid.

"Of course, thermal energy is evolved in the process of this thermodynamically
metastable state changing to one that is stable. Energy is dispersed from the
crystals, as they form, to the solution and thus the final temperature of the
crystals of ice and liquid water are higher than originally. This, the
instructor ordinarily would point out as a system-surroundings energy
transfer. However, the dramatic visible result of this spontaneous process is
in conflict with what the student has learned about the trend toward disorder
as a test of spontaneity.

"Such a picture might not take a thousand words of interpretation from an
instructor to be correctly understood by a student, but they would not be
needed at all if the misleading relation of disorder with entropy had not been
mentioned."

[http://entropysite.oxy.edu/cracked_crutch.html](http://entropysite.oxy.edu/cracked_crutch.html)

~~~
jbay808
I'd rather they get more curious about the spontaneous crystallization, and
get inspired to look deeply into what's really going on when that happens.

------
andrewflnr
So one thing I've never understood is how you can "count" microstates, or bits
required to describe them, when the relevant physical parameters all seem to
be real numbers. For instance, a gas of N atoms is described by 6N real
numbers (3d position and velocity) regardless of how hot it is. The article
talks about quanta of energy, but that seems like a simplification at best: a
given interaction might be quantized, but quanta in general (e.g. photons)
exist on a real-number spectrum, so it's possible to have an uncountable
infinity of energy packet sizes right? (That's the point of a black body,
right?)

What am I missing? Is this a weird measure theory thing, where hot objects
have even bigger uncountable infinities of states and we get rid them all with
something like a change of variables? If you told me spacetime was secretly a
cellular automaton I could deal with it, but real numbers ruin everything.

~~~
IIAOPSW
I asked the same question to my professor when I took thermodynamics as an
undergrad. In that class we are told that a particle in a box of size 2 can be
in twice as many places as a particle in a box of size 1. But in real analysis
we learn there are just as many numbers between 0 and 1 as there are between 0
and 2. The answer I was given, in true physicists form, is hand-wave it.
There's an intuitive notion that twice as big means twice as many places to
be, therefore just accept it and let the mathematicians cry over our abuse of
the reals.

The true answer is that "quanta of energy" is _not_ a simplification. The idea
that physical variables like energy and position come in discrete units is the
quant in quantum physics. If you imagine the position of a particle in a box
of size 1 to be discretized into n states, then a box of size 2 really would
have 2n states. So all of your concerns are moot because quantum mechanics
replaces all the uncountable sets with countable ones.

But this still leaves us with the issue that Boltzmann did all this work
before quantum mechanics existed so there must be some useful notion of
"bigger uncountable infinities". The answer, as far as I know, is that you can
always approximate classical physics with finite precision variables so long
as the precision is high enough (replace the reals with floats). The idea of
counting states works for any arbitrarily precise (but still finite)
discretized variables, and as far as physicists care an arbitrarily precise
approximation is the same as the real thing.

~~~
anonytrary
> energy and position come in discrete units

This is not true; position/time are not quantized in the standard model.
String theory is not canonical. I think a better way to think about it is not
in terms of size, but in terms of time. A particle in a bigger box, on
average, can go on a random walk for longer without hitting a wall. It will
take longer for a particle to sufficiently (arbitrarily close) exhaust the
phase space of a bigger box.

~~~
matt-attack
I thought the Planck constant somehow tied in with the smallest unit of length
allowed. Sort of like the pixels of 3-space.

------
brummm
As a physicist, this is such a great explanation that's actually correct for a
change. Entropy must be one of the if not the most misunderstood physical
concept out there (along with Planck metrics like Planck energy or Planck
length). Entropy is commonly used and written about by so many people that
clearly lack the understanding of it that this blog post is a refreshing
change.

~~~
cinntaile
The author is a physicist as well.

[https://mobile.twitter.com/aatishb](https://mobile.twitter.com/aatishb)

~~~
brummm
Yeah, it showed in the way he explained things.

------
ramadis
I recommend you the story "The Last Question", by Isaac Asimov[0], one of my
favourite stories of all time. It's about how the question about reverting the
direction of the entropy keeps rising on people, throughout the life of the
universe.

[0]
[https://en.wikipedia.org/wiki/The_Last_Question](https://en.wikipedia.org/wiki/The_Last_Question)

------
layer8
That still doesn’t answer the question how, if the laws of physics are time-
symmetric, the universe as a whole can have a time-asymmetric evolution of
entropy. I.e., if something forces entropy to increase in the long run, then
that should hold in both directions of time. So what is it that causes entropy
to only increase in the direction of the future, but not in the direction of
the past, given that the laws pf physics do not distinguish between both
directions?

~~~
kergonath
We don’t have an answer to this question. I don’t want to discuss metaphysics
here, but there is a very interesting discussion on that subject here:
[https://youtube.com/watch?v=-6rWqJhDv7M](https://youtube.com/watch?v=-6rWqJhDv7M)

~~~
layer8
Let me rephrase maybe: Given the state of affairs I described above, I don’t
understand what is the convincing argument that entropy does indeed increase
in the long run. Any argument given should also work in the reverse direction,
given the symmetry of time, shouldn’t it? (And thereby create a kind of
reductio ad absurdum.) If not, why not?

~~~
PaulDavisThe1st
I think it collapses down to an even simpler, perhaps even tautological point
(though I think that it really isn't).

In a system with energy present, the system is continually and randomly
shifting between microstates.

We identify macrostates, and analytically can identify certain macrostates as
having more or less possible microstates.

Given the constant random movement between microstates (thanks, energy!), the
system's macrostate is most likely to be one represented by large numbers of
microstates.

The system isn't really "increasing its entropy" \- it's simply randomly
exploring all accessible microstates. If we could observe the microstates
directly, we'd probably never think of entropy at all. But since we observe
the macrostates, we also end up noticing that over time, systems with energy
tend toward macrostates representing large numbers of microstates.

If the opposite were true, you'd basically be saying "more probable things are
actually less probable", which is contradictory. Increasing entropy is really
just a different way of saying that "more probable things tend to happen".

Because of the relationship between microstates and macrostates, and our
cognitive biases towards certain macrostates, we tend to notice things moving
towards what we consider disorder. All that is really happening that things
just tend toward "more probable" macrostates, because they represent larger
numbers of possible microstates.

Again, if you were unaware of the macrostates, you'd see no asymmetry.

------
maxov
Loved this article. There are so many applications of entropy and statistical
physics in computer science, and I find it fascinating that the same general
properties are useful in such different contexts.

For example, there's a well-known phenomenon in probability called
concentration of measure. One of the most important examples in computer
science is if you flip n coins independently, then the number of heads
concentrates very tightly. In particular, the probability that you are more
than an epsilon-fraction away from 0.5n heads is at most around e^{-epsilon^2
n}. This is exactly the setting described in the article with the sheep in two
pens, and this inequality is used in the design of many important randomized
algorithms! A classic example is in load balancing, where random assignment
sometimes produces 'nice' configurations that look very much like the high-
entropy states described in this article (but unfortunately, many times random
assignments don't behave very well, see e.g. the birthday paradox).

The sum of independent random variables is well known to have concentration
properties. An interesting question to me is what sorts of other statistics
will exhibit these concentration phenomena. An important finding in this area
is the bounded differences inequality
([https://web.eecs.umich.edu/~cscott/past_courses/eecs598w14/n...](https://web.eecs.umich.edu/~cscott/past_courses/eecs598w14/notes/09_bounded_difference.pdf)),
which generally states that any function that doesn't "depend too much" on
each individual random argument (and the sum of bounded variables satisfies
this assumption) exhibits the same concentration phenomenon. There are some
applications in statistical learning, where we can bound the estimation error
using certain learning complexity measures that rely on the bounded
differences inequality. In the context of this article, that means there's a
whole class of statistics that will concentrate similarly, and perhaps exhibit
irreversibility at a macroscopic level.

~~~
pabo
There's a related anecdote about John von Neumann: he used to joke that he has
superpowers and can easily tell truly random and pseudo random sequences
apart. He asked people to sit down in another room and generate a 0/1 sequence
via coin flips, and record it. Then, generate another sequence by heart,
trying to mimick randomness as much as possible. When people finally showed
the two sequences to him, Neumann could instantly declare which one was which.

People were amazed.

The trick he used was based on the "burstiness" rule you describe: a long
enough random sequence will likely contain a long homogeneous block. While
humans tend to avoid long streaks of the same digit, as it does not _feel_
random enough.

So, all he did was he quickly checked with a glimpse, which of the two
sequences contained the longest homogeneous block, and recognized that as the
one generated via the coin flips.

~~~
maxov
That's a cool anecdote :-) I wouldn't say it uses concentration of measure
exactly, but I see how it is related. The anecdote is about asymptotic
properties of random sequences, and concentration of measure is about the same
too. In this case, I think you can show that homogenous blocks of length
log(n) - log log (n) occur at least with constant probability as n gets large.
In other words, the length of homogenous blocks is basically guaranteed to
grow with n. I suppose a human trying to generate a random sequence will
prevent homogenous blocks above a certain constant length from appearing
regardless of the length of the sequence, which would make distinguishing the
sequences for large n quite easy!

I think there is also a quite strong connection in this anecdote to the
information-theoretic notion of entropy, which takes us all the way back to
the idea of entropy as in the article :-) Information-theoretically, the
entropy of a long random sequence concentrates as well (it concentrates around
the entropy of the underlying random variable). The implication is that with
high probability, a sampled long random sequence will have an entropy close to
a specific value.

Human intuition actually is somewhat correct in the anecdote, though! The
longer the homogenous substring, the less entropy the sequence has, and the
less likely it is to appear (as a limiting example, the sequence of all 0s or
all 1s is extremely ordered, but extremely unlikely to appear). I think where
it breaks down is that there are sequences with relatively long homogenous
substrings with entropy close to the specific values (in the sense that the
length is e.g. log (n) - log log (n) as in the calculation before), where the
human intuition of the entropy of the sequence is based on local factors (have
I generated 'too many' 0s in a row?) and leads us astray.

------
leafboi
[https://www.youtube.com/watch?v=wI-
qAxKJoSU](https://www.youtube.com/watch?v=wI-qAxKJoSU)

In the video above you will witness a metal wire in a disorderly shape
spontaneously form into an organized spring shape when entropy is increased
(the wire is heated). There are no special effects or video rewinding trickery
here, the phenomenon is very real.

It is as if you were looking at water spontaneously form into ice cubes but
entropy is NOT reversing! It is moving forward.

The video is a really good example of entropy. It's good in the sense that if
you understand why entropy is increasing when the metal is heated than you
truly understand what entropy is... as the video gets rid of the notion of
order and disorder all together.

That's right. There are many cases of increasing entropy where the system
spontaneously progresses from a greater disorder to more organization.

Many people say that the when some system becomes more organized that means
entropy is leaving the local system and increasing overall in the global
system. This is not what's happening here. The wire is being heated. Atoms are
becoming more organized and less disorderly by virtue of MORE entropy entering
the system. If you understand this concept then you truly understand entropy.
If not you still don't get it.

------
xwdv
If the entropy of the universe were to suddenly go in reverse would we be able
to detect it or would our memory formation and perceptions being reversed make
it indistinguishable from what we experience now?

~~~
jbay808
This is a very good question.

Yes, time reversal would also reverse the process of our perception and
memory, so we would experience time "moving forward" even if time were
"moving" backward. We would experience entropy increasing even if entropy were
"decreasing with time". (And it's perfectly valid to call the past "+t" and
the future "-t"; the laws of physics don't care; if you do that, you'll see
that entropy decreases as "t" increases, but despite changing definitions
you'll still only have memories of a universe with lower entropy than the
present).

This is one reason why it might be more correct to say that time doesn't move
at all. We just perceive a direction of time wherever the universe has an
entropy gradient along "the time axis".

~~~
PaulDavisThe1st
Entropy decreasing is not equivalent to time reversal.

[Edit: to expand a bit: time reversal requires that some previous macrostate
is achieved again. Entropy decreasing merely requires that the system enters
_any_ macrostate represented by less microstates than the current one.

Put in more simple terms, the ice cube could reform but in a different shape.
Entropy would have decreased, but it would not "look like" time reversal - it
would just look like something very strange had happened. ]

~~~
jbay808
This is true! I oversimplified a bit, because it's much easier to concretely
discuss that scenario and its consequences, and it's more important from a
philosophical perspective. Just like it's much easier to catch ice cubes
forming on film by rewinding a tape than to run the camera waiting for
spontaneous entropy decrease.

Nonetheless, if the entropy of the whole universe were to consistently
decrease for a sufficient time, even along a different route than time
reversal, you should gradually start to notice that it's easier to form
predictions or "memories" of the future than to remember the past, due to
Landauer's principle. Before long, you should go back to considering the
future to be the state of higher entropy.

~~~
PaulDavisThe1st
I think you're priviledging microstates. There are plenty of entropy decreases
that are possible without the ice cube reforming. They're (extremely) unlikely
of course, but somewhat less unlikely than the new, singular microstate you're
asserting is somehow "less of a coincidence". It's vastly more likely (for
example) that 1% of the particle velocities reverse than all of them doing so,
and that could (depending ...) still lead to a decrease in the entropy of the
system (but not a reformed ice cube).

I don't see how a long term entropy decrease that does not follow the time
reversal route could be easier to form predictions in. There are many possible
higher entropy states for the water than the water sitting in the glass, and
if the universe enjoyed random entropy decreases, I see no reason why it would
tend to pick certain macrostates over others.

~~~
jbay808
Ah, I think we're just miscommunicating. I'm not saying that local time-
reversal is more likely to _happen_ than other kinds of entropy decrease; I'm
not saying that a system is more likely to retrace its past than to enter a
different state of low-entropy. Spontaneous local entropy decreases happen all
the time, of course (but are overpowered by entropy increases), and the
majority of those won't be exact reversals.

And I'm not saying that locking yourself in a refrigerator will allow you to
predict the future!

I'm mainly picking on time reversal because it's easier to concretely
communicate and reason about as an example of how a system behaves under
entropy decrease, and is philosophically important because depending on one's
definition of the sign of 't', you could actually view our universe as
undergoing entropy decrease _right now_. But redefining the arrow of time
won't let you form memories of the future -- you'll have memories of the lower
entropy universe, regardless of which way you call "the future".

The deeper point is, that's not a coincidence and doesn't depend on entropy
decreases specifically being tied to time reversal. If we lived in a universe
where the second law were somehow different and entropy "statistically always"
decreased with time the way it "statistically always" increases now, then...
well, it's hard to reason about such a universe because either it has very
different laws than our own, or it's just our own universe with exactly that
arbitrary t <\--> -t transformation. But in most logically-consistent
interpretations of that scenario, that universe's version of Landauer's
principle would also flip the arrow of time perceived by its local
inhabitants, and they'd end up with only memories of their "future".

If I'm not quite making my point clear, try asking yourself why, if the laws
of the universe are time reversible, why _can 't_ you remember the future the
way you can remember the past? This gets into the mechanics of how memory
works (as a general concept, not human memory specifically; it's easier to
think about computer memory).

(Edit): Why isn't this a trivial point? Well, if you imagine a universe
composed of a chain of of "linked states" that goes from (low entropy) - (high
entropy) - (low entropy), you'd find that any inhabitants of that universe
would perceive a universe with directional time that progresses in the entropy
gradient, even though that universe has no consistent directional time.

~~~
PaulDavisThe1st
>why can't you remember the future the way you can remember the past

this is getting a bit meta, maybe even off-topic. But I think this is fairly
simple to explain: you can't remember states you haven't been in. Generalized
memory implies some record of a state that has occured - states that have not
occured cannot be remembered. The problem is that memory of types that we are
familiar with (human, written, computer) all involve macrostates, not
microstates. And increasing entropy (aka "more probable things are more likely
to happen") mean that there is asymmetry at the macrostate level, and thus in
memory.

A memory system that only recorded microstates would, I suggest, have no
concept of time, and a much reduced notion of causality. There may be some
statistical patterns that could be observed and could perhaps be "strong"
enough to infer that "after state N we frequently end up in state P", but the
sheer number of states would likely interfere with this.

There's also the timescale problem. A memory system that operates on the
timescales of typical human experience will notice relatively constant change
in much of the world, as macrostates come and go. But a memory system that
operates at, say, geological timescales won't record many macrostate changes
at all, and will tend to indicate that almost nothing happens in the world.
All those macrostates ("tables", "chairs", "houses", "books") that came and
went without being noticed form no part of this system's memory of the world.
Of course, there are processes still taking place (new macrostates made up of
even more vast microstates), but these going to be even more
directional/asymmetric.

The one part of this view that leaves me a little confused is that at very
short timescales, the unchanging nature of many macrostates is echoed in
relatively unchanging microstates for most solids. The piece of metal that
makes your <whatever> isn't changing macrostates at any appreciable pace
(which is why its eventual wearing out forms an asymmetric experience of time
for us), but it also isn't changing microstates in any notable way either. I
find this confusing.

~~~
raattgift
> You can't remember states you haven't been in

If only memory worked that way!

False memories are commonplace. Where exactly did you put your keys? Even if
you give the right location, is that a true memory of the state you and the
keys were in when you separated, or is it a retrodiction ("I probably put them
in their usual storage place")?

> A memory system that operates on the timescales of ...

You mean one that measures parts of the world periodically?

ISTR we discussed this some exp(10^120) years ago[1] but I forget whether we
reached any conclusions.

\- --

[1] Dyson, Kleban & Susskind, 2002, [https://arxiv.org/abs/hep-
th/0208013](https://arxiv.org/abs/hep-th/0208013) eqn (5.2).

~~~
PaulDavisThe1st
I was commenting on a comment that specifically disconnected the word "memory"
from the specifics of human mental memory.

------
bobsoap
I love how the notion of entropy permeates into so many other things. It's
fundamental, universal, and at the heart of nearly every aspect of our
existence.

Take philosophy. If the ultimate state of everything culminates in chaos
(according to the theory of entropy), the human existence constitues the exact
opposite: controlling the chaos that surrounds us, and shaping it into
something useful and, in entropy-speak, progressively unprobable. Making ice
cubes out of water.

It follows that our existence can at least be described as a function of
entropy.

This describes life. Survival is the battle against entropy. Procreation is
the chosen weapon against chaos, a force of order in a universe that can't
help itself but fall into chaos -- and, undoubtedly, will ultimately prevail
in that fight. In the long run, life will lose.

Great article. Fascinating stuff.

~~~
antonios
Alternatively, all that attempts to control chaos and decrease entropy
actually results in _faster_ entropy increase overall on a systemic level. I
remember reading about a (Russian?) physicist that believed that life simply
happens as a result of the universe's attempt to increase entropy faster on
sufficiently complicated systems. If someone remembers his name I'd be
obliged.

~~~
0134340
Dorion Sagan (Yes, Carl Sagan's son) also covers this in his book Into The
Cool. That life is only a force multiplier in increasing it universally,
basically.

------
keenmaster
Great explanation. What are the top theories for why the universe began in a
low entropy state? The mere fact of that seems to contradict our current
understanding of entropy. That implies that there is something very
fundamental about the universe which we don't understand.

~~~
jbay808
> The mere fact of that seems to contradict our current understanding of
> entropy

What part of our understanding does it contradict? The second law of
thermodynamics says that entropy increases with time; this seems entirely
consistent with a low-entropy past.

One explanation for why the universe began in a low entropy state is that that
state has a very low description length. (This is a bit of a truism, since
description length is a measure of entropy). But basically, let's just imagine
that the universe is a simulation, with an initial state described by an
initialization routine that sets up the simulation to run. If the initial
state has high entropy, that initialization routine would need to be very long
and detailed to describe exactly the location of every electron, neutrino,
etc. If the initial state has very low entropy, that initialization routine is
very short. If there's a reason to think that a short program is "more
probable" than any particular very long program, then that would explain a
low-entropy initial condition.

Another explanation is that, if the universe random-walks through all possible
configurations, the "past" will still always look lower-entropy than the
"future", for any little life-form that occupies that universe, because that
life-form's memories will be much more likely to be correlated with the lower-
entropy state. (It would have been nice for the article to go into this
detail, but it's rarely discussed).

Still another explanation is provided by Many-Worlds interpretation of QM.
Again the "big bang" is akin to initializing the wavefunction of the universe
to something very simple and compact like a constant function, which as a
whole evolves unitarily; the complexity and increasing entropy arises within
particular branches of that wavefunction, where an observer requires an ever-
longer description length to identify their particular branch.

~~~
novaRom
Your reasoning is strange. Actually, higher entropy is what we may call "of
lower complexity" requiring ever-shorter description length.

~~~
jbay808
A higher entropy state has a longer description length.

For example, let's say I have a magic electron microscope that can scan and
record the exact position and velocity of each particle in some 1-cubic-micron
volume, to within Heisenberg uncertainty limits and some finite digitization
precision.

If my sample is a 1-cubic-micron volume of flawless monocrystalline silicon at
0 Kelvin, I can 'zip' my recording and transmit that description in a much
shorter sentence (in fact, I just sent it to you!) than if my sample is a
cubic micron of room-temperature saltwater (whose macrostate I just described,
but whose microstate I did not).

~~~
novaRom
Your example of monocrystalline silicon at (almost) 0 Kelvin has actually
higher entropy than your example of saltwater.

~~~
jbay808
Can you elaborate? And what if I used as comparison something like room-
temperature doped polysilicon?

------
rstuart4133
That's remarkably well done, but the underlying principle can be summarised as
"regression towards the mean" which is not so difficult to understand.

Now explain this: when the sheep are cooped up in one box you can extract
energy from the system. This becomes apparent if you imagine the boundary
between the boxes is a fan. As the sheep move from what is a high pressure on
one side to 0 pressure on the other they will move the fan. Attach a generator
to the fan, and you get energy out.

This configuration, with high pressure on the side low on the other, will
happen every so often through random chance. Admittedly not very often, in
fact so rarely it's useless to us. But in principle we have a perpetual motion
machine.

We don't, of course. But why not?

And more to the point is the 2nd law nothing more than a statement of
averages? By which I mean "order always tends towards disorder" is in fact
false. We will every so often see order arise from disorder. So the 2nd law is
not really a hard and fast law, any more then "you always loose when gambling
at the casino" is a hard and fast law.

------
boyobo
Great article. Can energy be explained in a similarly simple fashion? I'm
pretty comfortable with probability theory so this entroppy explaination makes
sense, but I still don't understand what energy is.

Also, who determines what a 'macroscopic variable' is? Why do there only seem
to be 3 for gassess (V, T, P?)

~~~
petschge
There is more macroscopic variables, for example the three components of the
bulk flow velocity or the density of the gas.

~~~
boyobo
But are those the only 6?

------
iandanforth
Great article, I had hoped it would also cover locally anti-entropic
processes. We do see liquids turn into regular solids after all:

[https://www.youtube.com/watch?v=caGX6PoVneU](https://www.youtube.com/watch?v=caGX6PoVneU)

~~~
trhway
>locally anti-entropic processes. We do see liquids turn into regular solids
after all:

Crystallization is exothermic process. It releases heat into environment. It
converts potential chemical energy into heat as result of bond making. Thus
the total entropy of the system "crystal forming liquid + environment" is
increased. Life is another famous process of local anti-entropy which is
driven by the increase of the total entropy.

------
paultopia
Alternative name: an explanation of entropy that isn't at all baaaaaaaad.

------
ba0bab
great explanation and visuals. but I do not quite get the way the arrangements
of sheep are treated. They seem to be counted in the standard "Unordered
Sampling with Replacement" fashion.

>Just as the sheep wander about the plots of land in the farm, these packets
of energy randomly shuffle among the atoms in the solid.

this would mean that any "packet" of energy is equally likely to be in any of
the buckets and the "packets" are independent of each other. so we have an
equal distribution on the product space with 6^6 elements.

but then later

> Now, let’s assume the sheep are equally likely to be in any of these 462
> arrangements. (Since they move randomly, there's no reason to prefer one
> arrangement over another.)

under the prior assumptions these arrangements would not be equally likely.
e.g. "all sheeps in plot 1" would be far less likely than "each sheep in a
different plot" am I missing something here?

in any case the same conclusions can be drawn in both cases, only that the
concentration around 3 is already more pronounced in the "6 sheep, 6 plots"
case using the product space model.

~~~
emdubbelyou
I think the confusion is in the way that sheep as a word can be both plural
and singular. Specifically, one sheep is as likely to be in any single spot
compared to any other single spot. It’s when you get to more than one sheep
that you see the distributions

------
djaque
Great article! I'm almost done with a PhD in physics and statistical physocs
is still the hardest thing for me to wrap my mind around. Even more so than
quantum and relativity. There's something about how unintuitive math becomes
in ultra high dimension and the old timey feeling way that it was taught to me
(steam engines and stuff) that makes it difficult to learn.

~~~
atrettel
Thermodynamics as a subject is taught in two entirely different traditions,
the engineering tradition and the physics tradition. I've taken courses in
both traditions and I personally find that physics tradition explains a lot
more.

When I took engineering thermodynamics, the focus was on solving "practical"
problems, and I never got a good understanding of what entropy was. The point
of the class was not to teach physics, though. The point was to get engineers
to solve thermodynamics problems (like heat engines and cycles). The primary
thing I remember from that course was looking up things in the tables in the
back of the book and converting between Btus and other units.

I only understood entropy after taking a statistic physics course. The
course's explanation of entropy is basically the same explanation as the
article. The point of that class was entirely different than the engineering
course, so the professor spent several lectures going into detail about what
entropy is and how it relates to everything else. I never had to do any table
lookups in that class too.

Side note: you can easily tell what tradition you were taught in based on the
sign convention in the first law of thermodynamics you learned:

[https://en.wikipedia.org/wiki/First_law_of_thermodynamics#Si...](https://en.wikipedia.org/wiki/First_law_of_thermodynamics#Sign_conventions)

------
pawelmi
Reminded me of Anna Karenina principle.
[https://en.m.wikipedia.org/wiki/Anna_Karenina_principle](https://en.m.wikipedia.org/wiki/Anna_Karenina_principle)

------
roimor
I don't really understand this "Entropy Is All About Arrangements" takeaway. A
fair coin has higher entropy than a biased one. What are the "arrangements" in
this case?

~~~
kgwgk
The coins don't have entropy, the sequences they produce do (in information
theory sense, this is not about physics).

For a long sequence (say 1000), the former will produce close to 500 heads and
500 tails. The latter, assuming one heads are three times as probable as
tails, will produce around 750 heads and 250 tails. There are many more
different sequences of the first kind.

~~~
roimor
I'm referring to the entropy of the Bernoulli distribution. If the coin is
fair, the entropy is 1 bit... if the coin isn't fair, then the entropy of the
distribution is less than 1 bit. I'm having trouble reconciling the
information theory way of thinking about entropy as a function of a
distribution, with how physicists tend to think of entropy of arrangements.

~~~
kgwgk
There is a relationship between the distribution of x_i and the sequences
generated from that distribution x_1, x_2, ..., x_n. If the coin isn't fair
there are less arrangements possible. If the coin has two heads there is only
one possible sequence.

------
jaspal747
excuse my naivete, but do black holes help reduce entropy by
capturing/engulfing things around them? Is that the cycle how universe keeps
creating and recreating itself?

~~~
adeledeweylopez
Nope, black holes have entropy proportional to the surface area of their event
horizon. So the more stuff they engulf, the more their entropy increases, and
thus they satisfy the 2nd law of thermodynamics just like everything else.

It's a very interesting and active area of physics actually:
[https://en.wikipedia.org/wiki/Black_hole_thermodynamics](https://en.wikipedia.org/wiki/Black_hole_thermodynamics)

~~~
WealthVsSurvive
Has this been observed, or is it more: here's some math that makes entropy
even possible because the alternatives sound unlikely? I'm betting on the
weird, things like the universe being generative on a macro scale and entropic
in local timespace, wherein dark matter is merely newly created matter not in
another universe, but in this universe, maybe some unknown interactions
between the unstoppable force of expansion and the immovable object of a black
hole's gravity. I'm probably blathering, I'm not a physicist.

------
aoowii
Order as humans understand it is different from physical order. We see order
as an array of ascending numbers or a house of cards. The universe sees order
more like a state of 'useful' energy, where it's possible to extract it, and
seemingly became ordered by putting energy in. This is why I think the
information theory definition of entropy is a bit misleading.

I like to see it as a radioactive atom, which starts as a useful structure
with potential energy that has an inherent timer until this energy is lost due
to the universe wanting to return to equilibrium. So it's statistically
extremely unlikely to get the atom back to its original energetic state by
nature, but not because because it's literally impossible, it's just
impossible to do this without putting energy back in and that's not something
that happens naturally.

A reversal of entropy is entirely possible, it's just called doing work. Of
course, we don't have the knowledge necessary to reverse each individual atom
in the melting process of an ice cube, but one day we might. It's
theoretically possible with enough work. Of course, there will always be a
loss of energy, but i'm pretty sure that's an entirely different thermodynamic
law.

