> When hydrocarbon molecules are dispersed in water, the water molecules rearrange to maximize the number of H-bonds they make with one another. They form a cage-like structure around each hydrocarbon molecule. This cage of water molecules around each hydrocarbon molecule is a more ordered arrangement than that found in pure water, particularly when we count up and add together all of the individual cages! It is rather like the arrangement of water molecules in ice, although restricted to regions around the hydrocarbon molecule. This more ordered arrangement results in a decrease in entropy. The more oil molecules disperse in the water, the larger the decrease in entropy. On the other hand, when the oil molecules clump together, the area of “ordered water” is reduced; fewer water molecules are affected. Therefore, there is an increase in entropy associated with the clumping of oil molecules —a totally counterintuitive idea!
What about with heavy gases and light ones like helium? Do they eventually mix completely in large atmospheres under a big gravity gradient?
I would think a mixture should gain kinetic energy as the heavier one settles more on the bottom, releasing photons from the heat which increases total entropy, but in a closed system where the photons are reflected back it would hit some equilibrium with heavier stuff at the bottom, but more kinectic and photonic energy which maybe both together give more degrees of freedom than a more evenly mixed mixture with less kinetic energy and higher gravitational potential energy.
> I would think a mixture should gain kinetic energy as the heavier one settles more on the bottom,
But this is not what happens, or there would only be CO2 (and a small ammount of the heavier atoms and molecules) in the lower part of our atmosphere.
> I would think a mixture should gain kinetic energy as the heavier one settles more on the bottom,
But this is not what happens, or there would only be CO2 (and a small ammount of the heavier atoms and molecules) in the lower part of our atmosphere.
Edit: Maybe you know this already, but in equilibrium the mixture does indeed have more kinetic energy (per unit of volume) as you go down because even though the temperature remains constant the pressure increases.
You’re right, the atmosphere is much more complex and dynamic than in these idealized models. But assuming the system is closed and in thermal equilibrium and that these are ideal non-reacting gases then gravity has an effect on the density and pressure but not on the composition.
I disagree. Assume it is a closed box reflecting all photons back inwards and perfectly bouncing the gas particles. Now assume here are only two gas particles in the box, a heavy atom and a light one. Assume the box is tall enough that in the presence of gravity there is not enough total energy in the system for either particle to reach the top of the box.
The light one will move faster than the heavier one on average when they come into contact, and in the presence of gravity it will have a higher average height. The same will hold as you add more particles, but there is a curve to it.
It is true hat the atmosphere is more complex, and has things like ozone layer causing temperature inversion due to different absorption characteristics, etc., but the general reasons that H and He are so much more prevalent in the upper layers is largely due to this kind of explanation using gravity.
It’s true, I don’t know what I was thinking. The barometric formula that gives the density gradient for each (ideal) gas depends on the molecular mass so the profile will be different and the composition of the mixture will vary with height.
I think this example is not fair, because it doesn’t compare the entropy of two states in thermodynamic equilibrium. The emulsion state is clearly not an equilibrium state.
In this situation I think of entropy as “lowest energy state”.
The oil and water have different densities, and are under the same force of gravity.
Any higher-density substance above a lower-density substance will have potential energy, that eventually must dissipate into thermal energy as the higher-density molecules descend and perturb other molecules.
Slightly OT, but I just saw this last night and thought it was hilarious:
You should call it "entropy", for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name. In the second place, and more important, no one really knows what entropy really is, so in a debate you will always have the advantage. — John von Neumann, suggesting to Claude Shannon a name for his new uncertainty function, as quoted in Scientific American Vol. 225 No. 3, (1971), p. 180.
And there's a story that Norbert Wiener used to wander around MIT telling anyone who would listen that information is negative entropy, including Shannon. (I'm having trouble finding it, will edit if I track it down)
Edit: ah, it was Fano who told that story (quoted in "The Cybernetics Moment"):
> Electrical engineer Robert Fano at MIT, recalled that sometime in 1947, Wiener "walked into my office ... announced 'information is entropy' and then walks out again. No explanation, no elaboration."
The definition of entropy from a statistical mechanics perspective, is roughly a count of the number of different (microscopic) configurations, \Omega, of a system. When all the configurations are equally probable, you get the formula
S ~ log(\Omega)
I don't think you see this definition in a physics education until you do a course on statistical mechanics? I feel like I certainly had teachers say "disorder". I think the connection being that if there are more configurations it's more "disordered" which isn't really a precise thing.
Looking back, I think rambling about ideal gases/pressure/volumes is a boring way to teach this subject (at least at late high school/first year uni). You should probably introduce it by talking about configurations and information. You can think of some very clear examples (configurations of a list of objects for example, it doesn't necessarily have to be "physical"). Then maybe you would go on to show that this is connected to macroscopic physical properties.
> I don't think you see this definition in a physics education until you do a course on statistical mechanics?
I'm over 60, and I only recent discovered the definition of entropy recently thanks to Leonard Susskind youtube videos. For 60 bloody years I only got the dumbed down version about "disorder" and it made absolutely no sense to me whatsoever. Now I see the same confusion in others - like here on HN were to people where duking it out over whether one arrangement of a pack of cards (all individually identifiable of course) had more entropy than the other.
This dumbing down does nobody any favors whatsoever. It certainly doesn't promote understanding of the basic principles.
So it's an article on entropy (as disorder) without one mention of Shannon[1]? This seems like a pretty big oversight.
In the context of information theory, entropy is absolutely a decay of information. I understand that this is physics (not math -- although the formulae are virtually identical), but there's a very clear, accurate, and formal use of "entropy" to mean exactly what the author purports it doesn't mean: chaos.
It also matches what we know: the less "uncertain" some amount of the information the less it is compressible.
However he does mention "chaotic" in one sentence, in evaluating the "constraints" imposed by the language when making the crossword puzzles:
"If the redundancy is too high the language imposes too many constraints for large
crossword puzzles to be possible. A more detailed analysis shows that if we assume the constraints imposed
by the language are of a rather chaotic and random nature, large crossword puzzles are just possible when
the redundancy is 50%."
I don't think it's an oversight. Shannon entropy, Von Neumann entropy and other information theory entropies are useful, but not physically fundamental in the sense of the second law of thermodynamics.
That said, I don't under whether Gibbs entropy or Boltzmann entropy is fundamental-- ?
> Shannon entropy, Von Neumann entropy and other information theory entropies are useful, but not physically fundamental...
I have to disagree, especially given that one of the great mysteries of our generation (the Black Hole Information Paradox) hinges on what happens to information -- sure, it's information about particles, whatever that might mean. But Jacob Bekenstein (among others) argues that Thermodynamic entropy is analogous to information (Shannon) entropy[1].
“As you have probably noticed, I didn’t say anything about information. That’s because really the reference to information in “black hole information loss” is entirely unnecessary and just causes confusion. The problem of black hole “information loss” really has nothing to do with just exactly what you mean by information. It’s just a term that loosely speaking says you can’t tell from the final state what was the exact initial state.“
I take your point, it might have been interesting to talk about how a lot of relativity and quantum mechanics can be thought of in terms of propagation and destruction of information. And for many students, talking about codes is a lot more interesting than pressure gradients in ideal gasses. A missed opportunity, perhaps.
Shannon / information entropy absolutely is fundamental to thermodynamics. It clears up a lot of confusion when you realize that the entropy isn't in the system itself, it's in your head (your lack of information about a system's exact state, knowing only summary statistics like its mass, volume, and temperature).
I would argue that's exactly the wrong way to think about thermodynamic entropy. Consider this: Energy isn't just in your head (you can use it to do work, and you have to know where it is precisely if you want to do General Relativity). Temperature isn't just in your head (you can measure it with a thermometer, and it tends to manifest as an average energy). The laws of thermodynamics then connect energy and temperature to entropy, which implies that the latter also can't just be in your head unless you add some mubo-jumbo.
Thermodynamics doesn't care about our ignorance of potential system configurations, but that these configurations are available for microscopic system evolution!
> Thermodynamics doesn't care about our ignorance of potential system configurations, but that these configurations are available for microscopic system evolution!
Thermodynamics equations are based on the macroscopic constraints on the system that we choose to measure and manipulate, implicitely admitting our ignorance about the potential microscopic configurations. Thermodynamic entropy, unlike energy, is not an intrinsic property of the physical system; it is a property of our macroscopic description of the system and the experimental processes that we decide to perform on it.
"Recognition that the "entropy of a physical system" is not meaningful without further qualifications is important in clarifying many questions concerning irreversibility and the second law. For example, I have been asked several times whether, in my opinion, a biological system, say a cat, which converts inanimate food into a highly organized structure and behavior, represents a violation of the second law. The answer I always give is that, until we specify the set of parameters which define the thermodynamic state of the cat, no definite question has been asked!"
Energy and entropy are quite different though; energy certainly is a property of the system itself.
Temperature, interestingly, is also in our heads. It isn't actually a measure of average energy, but actually is defined as a function of energy and entropy. (Entropy comes first and temperature is defined in terms of it, not the other way around).
T = dQ/dS
You can create systems in real life with infinite temperature, or even with negative temperature, but of course they certainly don't have infinite energy. One way to do so is to pump a system into having a population inversion, such that its entropy decreases as its energy increases, because the number of possible high energy microstates is small.
In general, only one microstate is ever "available" for a system to evolve into, which is the one that its current microstate will evolve into in accordance with the laws of physics. The trouble is that we don't know its current microstate, so we don't know its future microstate, so it seems to us that there are many available microstates, but that's all due to our own ignorance having only conducted summary statistical measurements and isn't an intrinsic property of the system.
I studied Shannon entropy like any CS student back in University, so I'm not an expert in all interpretations of Shannon entropy, but I don't think Shannon entropy tells you anything about what or how much information you are lacking about the system.
It may be used to this end, but it's not what Shannon entropy directly gives you.
There is no way to reduce the entropy of a totally random system by knowing more about it, unless your hypothesis is that true randomness doesn't exist in the universe
To connect them, the entropy of a system (such as an ideal gas) can be computed as the logarithm of the number of equally probable microstates in your probability distribution for that system.
(You need to define a sensible measure, because the state variables are continuous, but that's not important).
You can express this in terms of the message length or number of bits of information you would need to gain about that container of ideal gas in order to narrow it down to a particular microstate.
It's no different than the entropy of, say, a password.
> There is no way to reduce the entropy of a totally random system by knowing more about it
There absolutely is! For example, Maxwell's demon. Maxwell's demon will fail to reduce system entropy if it doesn't know the microstate to begin with, but otherwise it can use its knowledge to reduce the entropy of a gas by separating its parts.
> unless your hypothesis is that true randomness doesn't exist in the universe
I do also subscribe to this hypothesis, that like entropy, randomness also only exists in our heads as a reflection of our lack of information about a system, and isn't a property of any system itself. But I don't think I need to rely on this assumption.
With the entropy of a password, abcdefg has less entropy than kfbeksn because it is more likely to occur. Further, the calculation of entropy is based on an alphabet of symbols, shared by sender and receiver.
I don't see how that connects to the ideal gas example. Care to share?
I'm on mobile so it's hard to write a lot of detail. But here's probably the most straightforward way to clarify it.
Gasses are the same in principle but harder to visualize concretely, so let's think instead of a crystal lattice with impurities -- for example a doped silicon wafer.
I hand you a 28 gram disc of 0.1% n-doped silicon. It's got about 6e23 atoms in it, of which 0.1% are phosphorus.
That's a good statistical description of the crystal, but there are still a large number of possible configurations of phosphorus and silicon atoms that match that specification, but only one of them describes what you are holding in your hand. For whatever reason, you want to know the exact atomic arrangement of this particular wafer.
Fortunately, I've imaged this disc with an advanced electron microscope that has identified the identity of each atom in the crystal lattice, whether silicon or phosphorus.
How large is the data file that I have to send you? How much can I compress my electron microscope data?
Just like "abcdefg is less likely than kfbeksn", we have some prior information we can use to do compression. For example, dopant atoms are unlikely to cluster together, so Si-Si-P-P-P-Si-Si is less likely than P-Si-Si-P-Si-Si-P. That prior information is no harder to incorporate into the entropy calculation than it is in the case of passwords.
But such subtleties aside, we can start by assuming that any configuration is equally likely, compute the total number of configurations, and take the log base 2. That's going to be the minimum file size of the electron microscope scan. (I guesstimated 6e12 Tb but don't quote me on that). To do a proper job, we'd also need to measure and transmit the state vector of the lattice vibrations, but I'm ignoring that for now.
Ok, so that's for a crystal, which I chose because the electron microscope technology is easy to concretely visualize, and the state can be expressed as a string of Si's and P's. But an ideal gas is similar in principle, except its state variable consists of a string of positions and velocity values. We will have to discretize to some resolution to get discrete states, but that's fine. There is some datafile I could send you that would (quantum uncertainty aside) let you know the exact state of the gas, and with fine manipulators you could use that data to sort the gas molecules into different containers by molecular energy and reduce its entropy to zero. The entropy of the data you'd need to do that would be at least as high as the entropy of the gas, and upon clearing that data from your memory, the second law is satisfied.
The author's point is that "chaos" and "disorder" aren't well defined, usually.
The entropy content of a situation can't be computed without knowing the physics that would have gotten it there, just as I can't compute the entropy of a message (my surprise at receiving it, or the number of bits needed to transmit it) consisting of a single symbol without some knowledge of the distribution it was drawn from (the expectations of myself, or the receiver). (If it is part of a longer message I can use that message itself as a model, but that is only a first approximation.)
It seems to me that the "Typical vs. average" section of the article specifically covers informational entropy. The question seems to hinge on the difference between disorder and randomness.
It really doesn't. The little Enterprise thought-experiment and anecdotal impressions of what "looks" and "doesn't look" disordered is nonsense. From my source above:
> Shannon entropy is defined for a given discrete probability distribution; it measures how much information is required, on average, to identify random samples from that distribution.
It's carefully defined, and used in basically every piece of technology we own. Further, it's not even particularly tough to wrap your head around (a middle-schooler could understand Huffman encoding).
> The little Enterprise thought-experiment and anecdotal impressions of what "looks" and "doesn't look" disordered is nonsense.
I think you're misreading it, the author is trying to make the point that you can't meaningfully answer that question just by looking at the dots. It's a trick question, but it's used as a part of teaching how you compute entropy (which involves knowing the distribution). Yes, this is essentially information theory, but he's a physics teacher.
I can't tell you the "randomness" of a bunch of points any more than I can tell you the location of sunspots by looking at our star with the naked eye. That doesn't mean that there isn't a way of calculating the entropy, though (just how there is a way of making out sunspots with the right equipment).
Is that really the idea here? You're right: maybe I'm just not getting the point.
The "equipment" needed is a model of how the points came about. You may already get the point but permit me an example:
From an information theory perspective, consider 'apple' vs 'aaaaa'. I submit that if you have nothing besides the string itself, you'd conclude that 'aaaaa' is lower entropy than 'apple', but if your model is word frequency on the web, it'd be the other way around. There isn't anything inherent in the strings that makes one or the other true.
There are interesting cases in-between where you could derive a model with more math, e.g. the first million digits of pi has very low entropy, but in general exactly how much is a matter of the uncomputable Kolmogorov complexity.
With a physical system, if you know the configuration of the system and the laws of physics it obeys, you can compute this in a meaningful way, and it can tell you what direction the system "wants to go" to satisfy the second law of thermodynamics.
If you know the exact state of a system, then it has zero entropy regardless of what that particular state happens to be. In this case, you also don't need thermodynamics to model the system's evolution, because you can predict its future exactly.
When you only know summary, average statistics of a system (such as its temperature), its entropy is nonzero, because your information about the system isn't enough to isolate a particular microstate. Instead you just have a distribution of many possibile microstates, any of which would match the measurements you know. The more possible system microstates there are that would agree with your data, the higher its entropy.
I liked this article. Really clear. I found some of my own misconceptions. For instance, that uniformity and gradients are ways of characterising entropy.
The example of oil and water is great. While they are separated with a gradient, there is no latent tendency to flow together and mix.
But, it's only gravity that makes the oil and water issue hard to understand. With no gravity, they would not be evenly divided.
We still don't understand gravity, as it relates to entropy. Usually entropy makes things spread out, but gravity makes things come together. Weird.
I dunno. I always thought "entropy = disorder" was a big simplification, almost in the realm of lies to children. Entropy is what it is; the pattern detection algorithms in our heads are not always very good at judging global properties such as order.
I do have a degree in Physics, so perhaps that's the source of this belief I've held - probably from countless impromptu discussions with fellow physics geeks. You wanna know the entropy of this system? Calculate it, don't eyeball it.
I also liked the article. I lack theoretical knowledge to determine if the system that is affected by gravity can still be considered "isolated" as per 2nd law of thermodynamics. I think this is not really relevant here, as the article focuses more on the common mental model, that disorder always increases with time. This model is not only incorrectly labelled (as entropy), but also misleading when used to describe some processes in the world.
This oil/vinegar example is immediately useful for other areas too. We have oil and vinegar in the dressing. They eventually become separated and we need to put effort to mix them together. Now replace oil and vinegar with leave/remain, blue/red or whatever else...
The author seems to mix up Gibbs free energy with entropy. The remark that thermal equilibrium equals maximum entropy is simply wrong. All examples can be explained with minimization of the free energy:
dG = dH - TdS
dH == internal energy (chemical bonds) + pV
dS == Entropy term
T == Temperature
Where entropy is the number of different states the system can be in for a given internal energy.
A thermodynamic system in equilibrium will always tend to a state of lowest free energy.
Example: a perfect crystal would be a totally ordered state (and with the lowest entropy), with the internal energy minimized due to the highest number of chemical bonds. For T -> 0, this would be the thermodynamically most stable configuration (however not always kinetically accessible, e.g. supercooled water). As Temperature goes up, the Gibbs free energy of a more disordered phase (e.g. crystal defects or a liquid/gas phase) becomes lower and will eventually lead to the melting of the crystal.
For the oil/water example, the separated state minimizes the free energy since polar water molecules can form hydrogen bonds with themselves (lowering the internal energy), and push out the oil molecules, even though the entropy is lower than for a totally mixed state. At higher temperatures, this will change theoretically and the mixed-phase becomes favorable thermodynamically.
From this point of view, seeing higher entropy as more disorder seems absolutely fine, where more disorder is "lower chance of guessing the exact configuration of all molecules in the system".
Entropy is maximized for an isolated system (constant energy), free energy is minimized for a system in thermal equilibrium with a heat bath (can exchange energy with the environment). Gibbs free energy is minimized at constant pression and temperature, Helmholtz free energy is minimized at constant volume and temperature.
If you put now the oil/water system into a isolated container the equilibrium state won't change substantially. It remains separated in two phases and that's now the maximum entropy state.
Typical fallacy in entropy considerations is to consider system as isolated in the model while it is strongly connected to the environment in reality. The oil/vinegar (similar to the pennies and dimes
mentioned in the article - when shaken in a jar would "magically" separate with pennies on top) has exported the entropy to the environment through the work of gravitational force in full accordance with the 2nd law - the system transitions following the gradient and thus increasing total entropy of the whole system. A similar mistake for example is when "life" is stated to be "anti-entropic" in any sense while it is a really a form of matter organization which generates even more entropy than the same amount of dead matter would.
Isn't this a too narrow definition of entopy? Ok. Pure physical entropy was the first and original usage, but now informational entropy can be used at higher level, and it correlates well with disorder at that level.
If I sort an array its entropy will be lower - while due to thermodynamics the total entropy of the world will be increased.
So there is the only the high level component I'm interested in: I see less macrostate in a sorted array(one) than in an undordered(n factorial) even if the microstates in the world have increased(I don't care).
Same goes for the article examples, for me subjectively an ice statue can have many different forms (macrostates), if it melts it will have one form (a puddle) - so I see the entropy to decrease - of course the puddle have more microstates than the ice statue - just I do not differentiate between them...
True: in the sbove examples things became more ordered at high level (entropy have decreased), meanwhile physical entropy have increased - but that increase was global and in our local point of interest it is still can be said (according to modern usage) that entropy have decreased!
Of course the total sum have to be positive - that's the basis of how for example the Landauer principle is calculated, wich connects the above high level informational view with physical entropy...
Note that this is an article for physics teachers and the conclusion is “we can warn our students that this is not the meaning of the word “entropy” in physics.”
I would say, in itself the oil being separated from and being above the Italian salad is something that in itself decreases the entropy (for now I assume we can think of this as two gases of different density). However! It is overcompensated by other factors and in total the entropy is lowest in this case.
What factors? Imagine you could place the particles as you wished, and being encouraged by my first sentence you decide to mix it all up to achieve higher entropy. To make it all mixed up, you'd have to move the heavier, denser material towards the top, increasing the total potential energy. This would need the introduction of external energy into the system. We cannot do that with a closed system. We could only take the energy as heat from the salad, making it all colder. However, in making it colder, the motion of the particles becomes less uncertain, leading to a decrease in entropy. Ultimately the method of mixing it up fails to increase the entropy, showing that the separated configuration is indeed lower entropy.
However it is not because of the separation, but in a certain sense despite it. The separation helps bring the particles into a higher inner-energy state, because the potential energy decreases as the denser material moves down. Without paying attention to the micro situation the effect wouldn't make sense.
> (for now I assume we can think of this as two gases of different density)
What does that mean? If they were gases they would mix (as the atmosphere, where you don’t see layers of gasses of different densities but an essentially homogeneous mixture).
I think the point still stands that the separation contributes to a decrease in entropy, however it enables another effect which increases entropy even more.
If we were allowed to mix it up evenly by adding energy from the outside to lift the heavier material up (constant temperature), then we'd get an even higher entropy, agree?
I don’t understand what point you are trying to make. If you assume that’s the equilibrium state and therefore the highest entropy state given the constraints on the system then of course the entropy cannot spontaneously increase. And if it’s the highest entropy state despite looking “low entropy” under the flawed criteria of “looks ordered” then it’s obvious there is something else we’re missing that explains why the entropy is indeed maximal.
Anyway, to increase the entropy you don’t just need to add energy, it has to be in the form of heat (something you can’t extract work from later). If you move the “heavy” layer on top of the “light” layer the (potential) energy has increased but if the temperature remains constant the entropy doesn’t change.
I'm sorry, it is difficult to express my thought, although it is a simple thought that may be trivial to you.
I mean that ceteris paribus the layering (separation of materials) causes a decrease in entropy. However, the system still does this, because it gains more entropy through a different "channel".
If we could destroy the layering, thereby evenly mixing things up, but not changing anything else that affects entropy, we could increase the entropy. So it show that the layered version is lower entropy as long as all else is the same. The catch is, this would require adding extra energy into the system, because the mixed version has more potential energy. We could also imagine a mixed up configuration that has the same total energy, but then the temperature would be lower since more of the energy would be used up as potential energy, leaving less for heat. In that case the entropy would be lower.
Main point: the layering, just the fact that it is layered, this property, if we know nothing else, is still something that is a low entropy thing. So our intuition is not broken that order corresponds to low entropy.
> I mean that ceteris paribus the layering (separation of materials) causes a decrease in entropy.
But it doesn’t, which is the reason why it happens. Because that intuition of “oh, there are two phases, this means order and entropy is disorder so the entropy is lower” doesn’t correspond to the reality of the physical concept of entropy (the thing that is maximized for a thermodynamic system In equilibrium). That’s the point of the article.
Note that the discussion about potential energy is a red herring, phase separation does also happen in absence of gravity (you get droplets which don’t coalesce as easily, but it happens nevertheless).
That's why I tried to shift focus to density based layering rather than the more complicated oil water interaction.
And the reason it happens is because it doesn't magically get the corresponding external energy. However if we artificially, like gods, arranged everything to be non layered but the same temperature, the entropy would be higher than if layered and same temperature.
No, the entropy would be lower and the phases would separate spontaneously because that would increase the entropy. The only reason you get layers in the first place is that oil and water don’t mix! Water and alcohol also have different densities.
> When hydrocarbon molecules are dispersed in water, the water molecules rearrange to maximize the number of H-bonds they make with one another. They form a cage-like structure around each hydrocarbon molecule. This cage of water molecules around each hydrocarbon molecule is a more ordered arrangement than that found in pure water, particularly when we count up and add together all of the individual cages! It is rather like the arrangement of water molecules in ice, although restricted to regions around the hydrocarbon molecule. This more ordered arrangement results in a decrease in entropy. The more oil molecules disperse in the water, the larger the decrease in entropy. On the other hand, when the oil molecules clump together, the area of “ordered water” is reduced; fewer water molecules are affected. Therefore, there is an increase in entropy associated with the clumping of oil molecules —a totally counterintuitive idea!