
Spikey Spheres - ColinWright
http://www.solipsys.co.uk/new/SpikeySpheres.html?HN_20161120
======
jakobegger
A teacher of mine told us, "Don't try to imagine 4D space. You can't. If you
think you can imagine a 4D hypersphere, you're wrong. It's just not possible."

And everytime someone talks about how to imagine higher-dimensional space, I
have to think back, and I'm convinced that my teacher was right. Our intuitive
understanding of 2D and 3D just doesn't generalise to higher dimensions.

A hypersphere is nothing like a sphere, just like a sphere is nothing like a
circle, which is again nothing like a line segment.

If you talk about multidimensional geometries, stick to the math, and don't
try imagining it. Analogies to 2D or 3D objects (it's smooth, but like a
spike) are pointless and don't lead to new insights. We really need to stick
to precise, mathematical language if we want to work with higher dimensional
geometry.

~~~
jacobolus
However there is a 4-dimensional sphere (3-sphere) that you can easily access:
the possible rotations of any object in 3-dimensional space is topologically
and metrically isomorphic to a (half of a) 3-sphere, in the same way as the
rotations of an object in 2-dimensional space are isomorphic to a 1-sphere
(circle).

Think of an object’s current orientation as the “north pole” of our abstract
4-dimensional sphere. Rotating the object around any axis pushes you toward
the equator, which you reach once you have rotated the object halfway around
(the equator is made from all 180° turns, and is topologically a 2-sphere).
Continuing to rotate the object takes you towards the “south pole”, which is
where you get if you rotate your object 360°. To get back to the north pole,
rotate by 360° again.

Cf.
[https://en.wikipedia.org/wiki/Versor](https://en.wikipedia.org/wiki/Versor)
[https://en.wikipedia.org/wiki/Quaternions_and_spatial_rotati...](https://en.wikipedia.org/wiki/Quaternions_and_spatial_rotation)
and
[https://en.wikipedia.org/wiki/Rotation_group_SO(3)](https://en.wikipedia.org/wiki/Rotation_group_SO\(3\))

* * *

One other way you can get at the 3-sphere is by taking a stereographic
projection into all of 3-dimensional space (plus a point at infinity). If you
have some 3-dimensional shapes drawn on the surface of your 3-sphere, these
will get distorted under the stereographic projection, but locally angles will
be preserved, just like a stereographic projection of the 2-sphere onto a
piece of paper.

~~~
JadeNB
I think that your terminology "3-sphere" (which is standard) as synonymous
with "4-dimensional sphere" (which is not) may be confusing. Of course the
3-sphere is 3-dimensional, hence the name; it is just that it is canonically
embedded in 4-dimensional Euclidean space (and not in 3-dimensional, flat
Euclidean space).

~~~
jacobolus
Fair enough. Hopefully nobody was confused. I was trying to meet the
grandparent poster halfway. His comment was “[...] Don't try to imagine 4D
space. You can't. If you think you can imagine a 4D hypersphere [...]”

------
srean
High dimensions are indeed very, what shall I say, non-intuitive. Bellman
introduced, or perhaps popularized, the term 'curse of dimensionality'. High
dimensions may also comes with its blessing, 'concentration of measure' a
phenomenon where randomness concentrates into events of certainty in high
dimensions. I checked out their Wikipedia pages, while the former has a lucid
description, the page on the latter is quite technical.

Both these notions play a significant role in topics like machine learning,
statistics, function approximations etc. Its sometimes a delight sometimes
sheer frustration to see these two phenomena play it out.

------
ot
One other mindblowing fact is that if you project the mass of the sphere onto
a line that goes through the origin, you get a Gaussian distribution as the
dimension goes to infinity.

If you're interested in more technical details about the weird geometry of
balls, I can't recommend enough the first few pages of "An Elementary
Introduction to Modern Convex Geometry" [1]. Ironically, the author's last
name is Ball.

[1]
[http://library.msri.org/books/Book31/files/ball.pdf](http://library.msri.org/books/Book31/files/ball.pdf)

~~~
rcthompson
It should be clarified that this fact refers to a ball of volume 1 in N
dimensions. Since the volume is held constant, as the dimension goes to
infinity, so does the radius.

~~~
TeMPOraL
Wait, isn't it the other way around? I.e. if you want to keep a volume of the
ball constant as dimensions rise, don't you have to _reduce_ the radius?

(I'm a noob at this whole dimensions thing.)

~~~
legolas2412
Volume of a hyper-sphere of radius R reduces with increasing dimensions!
[https://en.wikipedia.org/wiki/Volume_of_an_n-
ball](https://en.wikipedia.org/wiki/Volume_of_an_n-ball)

To visualize, take a hyper-cube of length 1. It's volume is 1 unit for any
n-dimensional hyper-cube. Scribe a hyper-sphere touching all the sides. How
about its volume with increasing dimension? That decreases! (with respect to
the cube). To visualize, think about the distance of the cube vertex from the
centre, it increases with increasing dimension.

In other words, Most of the cube's volume is closer to its vertex, not in the
ball in the centre!

~~~
TeMPOraL
Thanks! I'll need to spend some time trying to visualize it though, I'm
nowhere near grokking it.

~~~
rcthompson
You can start by working out the ratio of volumes between a sphere of radius 1
and the enclosing cube in 1, 2, and 3 dimensions (i.e. a line segment, a
circle, and a sphere, respectively). You'll see that the ratio starts at 1 and
decreases as the dimension increases.

~~~
JadeNB
> You can start by working out the ratio of volumes between a sphere of radius
> 1 and the enclosing cube in 1, 2, and 3 dimensions (i.e. a line segment, a
> circle, and a sphere, respectively). You'll see that the ratio starts at 1
> and decreases as the dimension increases.

Be careful using low-dimensional phenomena as a guide to the asymptotic
behaviour! If you take a sphere of radius 2, rather than radius 1, then the
analogous ratio _increases_ for a while (until n = 5), but then decreases to
0.

------
tel
Another counterintuitive result in this vein is to think about randomly
sampling uniformly distributed points in a sphere. As the dimension rises, the
chance of finding a sample within, say, r/100 of the surface goes to infinity.
In high dimensions, all of the mass of the sphere is concentrated at the edge.

~~~
pakl
Exactly! Now consider that deep networks that classify images are tasked with
getting reliable statistics in very high dimensional spaces. Conv nets are
being forced to map from high dimensional spaces down to a very low
dimensional categorical decision. This is why weird classification errors you
see in "adversarial examples" keep popping up. The learned classification
boundaries are very spiky, and it's easy for the world to fall in between the
spikes. More data can't solve it (at least not practically) because there are
way too many gaps between the spikes.

A more tractable approach is to learn to use dynamics for perception rather
than ("just") statistics. The dynamical physics of a ball rolling is much
simpler (lower dimensional, more tractable) than a statistical view of
millions of differently illuminated pixels hitting a camera.

(A colleague of mine has a blog post on this issue of "statistics and
dynamics" at [http://blog.piekniewski.info/2016/11/01/statistics-and-
dynam...](http://blog.piekniewski.info/2016/11/01/statistics-and-dynamics/))

~~~
js8
Your point about dynamics kinda reminds me of Chomsky's critique of
statistical approaches to AI, for example here:
[http://norvig.com/chomsky.html](http://norvig.com/chomsky.html)

~~~
pakl
Yes, but in direct contrast to Chomsky (who would say there's not enough
data/time for kids to learn from) I am saying that there is a ton of rich
dynamical data in the world around us all the time. Plenty to learn from.

Just plug a webcam into an adequate system and allow it to learn dynamics by
trying to predict what it will see next.

Chomsky is almost right for the wrong problem: there won't ever be enough
human-labeled data for good generalization. ;)

~~~
js8
I think what Chomsky was saying in that particular debate was that purely
statistical methods do not lead to true understanding, as opposed to more
phenomenological theory.

------
js8
I am not sure I would interpret this observation as spheres that are getting
more spikey. Maybe what gets more spikey is the actual box, so the sphere has
it easier to touch or even extend outside of it.

~~~
JadeNB
> I am not sure I would interpret this observation as spheres that are getting
> more spikey. Maybe what gets more spikey is the actual box, so the sphere
> has it easier to touch or even extend outside of it.

I think that this is not correct. In n dimensions, the cross section of a unit
n-cube near the edge is a unit (n - 1)-cube, just as your intuition tells you.
However, the cross section of a unit n-sphere near its intersection with the
unit n-cube is a tiny, tiny (n - 1)-sphere (the radius of a cross section at
height z = 1 − 𝜀 goes to 0 as n goes to ∞); this is the 'spikiness' that the
author is discussing.

~~~
js8
On the other hand, it seems to me (maybe I am wrong) that the solid
n-dimensional angle of each corner of the n-cube is progressively smaller (as
a ratio of full n-dimensional angle). So in that sense, each n-cube gets more
spikey, and there is "less space" in each corner for each n-sphere that
enclose the central n-sphere.

------
SonOfLilit
For a more rigorous treatment (and many other gems on the border of highly
practical and highly theoretical), see Richard Hamming's great book "The Art
of Doing Science and Engineering", available for download from Bret Victor's
website:

[http://worrydream.com/refs/Hamming-
TheArtOfDoingScienceAndEn...](http://worrydream.com/refs/Hamming-
TheArtOfDoingScienceAndEngineering.pdf)

------
snippyhollow
I did a quick "visualization" hack a while ago
[http://nbviewer.jupyter.org/urls/gist.githubusercontent.com/...](http://nbviewer.jupyter.org/urls/gist.githubusercontent.com/syhw/9025964/raw/441645b476a2a997f27f5993e4da2988febe1ef3/SpikeySpheres)

------
musgravepeter
Fun.

I love that weird intuition trap you mention - the volume of a N dimensional
sphere of radius R sphere goes to zero as N -> infinity.

[https://en.wikipedia.org/wiki/Volume_of_an_n-
ball](https://en.wikipedia.org/wiki/Volume_of_an_n-ball)

~~~
jacobolus
This comes down to the definition of (hyper)volume. We use a (hyper)cube as
our unit, but that’s somewhat arbitrary.

We could alternately define (hyper)volume using a standard _n_ –simplex as the
unit, then we would say the volume of an _n_ –ball → ∞ as _n_ → ∞.

------
pakl
To see some concrete examples of how this sort of phenomenon causes surprising
failures in deep networks classifying images, see "Intriguing properties of
neural networks"[1]

[1]
[https://cs.nyu.edu/~zaremba/docs/understanding.pdf](https://cs.nyu.edu/~zaremba/docs/understanding.pdf)

------
benjismith
Very interesting! I've done a lot of work with higher-dimensional spaces in
NLP and Machine Learning (creating feature vectors with tens of thousands of
dimensions, and then calculating cosines between vectors to estimate document
similarity), but these ideas about spikey geometry are new to me and very
insightful. Thanks!

------
nkurz
_So one visualisation of a sphere in very high dimensions is not something
smooth and round, but something that is somehow simultaneously very
symmetrical, and yet also very spikey._

It's unlikely that this will help anyone, but I recently learned that the
spiky protrusions a naval mine are called "Hertz horns":
[http://www.popsci.com/blog-network/shipshape/terrible-
thing-...](http://www.popsci.com/blog-network/shipshape/terrible-thing-waits-
under-ocean)

So if for some reason you are trying to trying to make a loose analogy to the
spikiness of hyperspheres to an older military mariner who has never seen a
hedgehog, you could clarify by saying "you know, like the Hertz horns on a sea
mine?"

------
sbierwagen
[https://en.wikipedia.org/wiki/Curse_of_dimensionality](https://en.wikipedia.org/wiki/Curse_of_dimensionality)

