
Researchers: It takes 1.5 MB of data to store language information - lelf
https://medicalxpress.com/news/2019-03-approximately-megabytes-language-brain.html
======
wrinkl3
I haven't read the paper itself, but the article makes it sound like they
simply counted the _minimum_ number of bits required to represent the English
language. It mentions no neuroscience-specific insights, so the "brain" part
of the title is quite misleading.

~~~
rob74
Aside from that - is there really a way to express neural connections in the
brain in terms of bits and bytes?

~~~
jerf
The problem isn't whether there is "a" way; the problem is there's a
multiplicity of ways and we don't know what the best one is. Part of the
reason we don't have any idea what the best is is that we can't even construct
any of them from a real brain, that is, the "brain scanning" technology the
techno-rapture-style Singularity expects to develop does not even remotely
exist right now. So we have nothing to experiment with or gain any experience
with. The only ones we'd even remotely have access to right now would be some
rather brute-force encoding of neurons and connectivity strengths and we
pretty much _know_ that such a thing must be incredibly redundant, but we have
no ability right now to remove that redundancy. And we can't do it on a very
large neural set.

It has been done on small models, though, at least to some significant extent:
[https://en.wikipedia.org/wiki/OpenWorm](https://en.wikipedia.org/wiki/OpenWorm)

~~~
aperrien
Brain scanning techniques and simulation software have a long way to go, but
there have been significant advances made:

A full fly connectome:
[https://www.nature.com/articles/s41684-018-0183-8](https://www.nature.com/articles/s41684-018-0183-8)

Architecture of the Mouse Brain Synaptome:
[https://www.cell.com/neuron/fulltext/S0896-6273(18)30581-6](https://www.cell.com/neuron/fulltext/S0896-6273\(18\)30581-6)

So we are past just a simple worm simulation now.

~~~
jerf
Thank you for the better links. I just meant that as an example.

------
MrQuincle
This describes how to summarize a language in a computer system.

I think the storage requirements for the semantics is in particular
questionable. If you can refer to a particular concept somewhere in the brain
you only need a pointer to it. However, single pointers might not be
sufficient

The apparatus to learn a language might also not so easily distinguished from
the structure to store the language or the structure to quickly recall the
language. In that case taking about storage is like talking about compressing
Plato's world of concepts.

~~~
coldtea
> _This describes how to summarize a language in a computer system._

It describes how to summarize a language from a information theory standpoint
(which is something different).

There's a good reason to believe evolution would use something close to the
most information theoretic efficient implementation (and even if not, it's a
good lower bound)

~~~
madhadron
> There's a good reason to believe evolution would use something close to the
> most information theoretic efficient implementation

Do you have a citation? I would be very interested to see it.

I hate to make analogies between computing and biological systems, but I
immediately think of denormalization in databases, error correcting codes, and
database replication as examples of optimizations that take you farther from
information theoretic efficiency.

Information applies very straightforwardly to the study of a spike train of a
synapse (source, channel, and receiver). I remain unconvinced that it systems
like memory make sense as information channels.

~~~
pmayrgundter
Shannon observed this in his work towards information theory. Humans are able
to predict next letters in a stream of natural language with a probability
approaching optimal. The original paper is here:

[https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf](https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf)

The summary at wikipedia:

[https://en.wikipedia.org/wiki/Entropy_(information_theory)#E...](https://en.wikipedia.org/wiki/Entropy_\(information_theory\)#Entropy_as_information_content)

Notably, PPM, the best compression algorithm (at the time), was not able to
predict as well. So that's a quantitative demonstration of optimality in
language performance that includes a faculty for understanding syntax and
semantics and presumably is linked to phonology, pragmatics etc.

Chomsky discusses optimality in a 1998 talk here:

[https://youtu.be/7Sw15-vSY8E](https://youtu.be/7Sw15-vSY8E), at about 1h 7m
an onwards.

He says language evolved rather quickly for such a significant function
(estimating it arose quickly ~200kya). He says though many things in evolution
are pretty messy, language is close to optimal design. He doesn't provide
references, but I think (given the time and academic associations; anyone
know?) he may be referring to Optimality Theory:

[https://en.wikipedia.org/wiki/Optimality_Theory](https://en.wikipedia.org/wiki/Optimality_Theory)

Separately, our knowledge of anatomy of the language faculty is that it is
highly localized in the Wernicke and Broca areas of the brain. This also
suggests a small modification of an existing brain structure.

Additionally, optimality in biological systems has been a productive research
direction in a few areas: \- Optimality Principles in Biology -
[https://link.springer.com/book/10.1007/978-1-4899-6419-9](https://link.springer.com/book/10.1007/978-1-4899-6419-9)
\- Near optimal energy use in the metabolism of cell division -
[https://arxiv.org/pdf/1209.1179.pdf](https://arxiv.org/pdf/1209.1179.pdf) \-
[https://www.quantamagazine.org/the-math-that-tells-cells-
wha...](https://www.quantamagazine.org/the-math-that-tells-cells-what-they-
are-20190313/)

~~~
madhadron
I think we may be talking about different things. You're talking about the
language itself. I was talking about the neurological structures that produce
the behaviors that are language.

I poked through the optimality literature in biology pretty thoroughly a few
years back. There is a great deal of interest, but except in a few cases where
a simple piece of natural history is nicely described by an evolutionary game,
the choice of criterion to optimize is generally arbitrary. Without an
evolutionary mechanism and observations of forces driving that mechanism there
is no reason beyond just-so stories to apply any particular criterion.

~~~
pmayrgundter
I see the distinction but I am following Chomsky, that they're aspects of the
same thing; language is a faculty in a specific, recently evolved neurological
structure. That's a bold theory, but it seems to have held up better than
approaches where language is an ability learned through a general cognitive
intelligence mechanism. Chomsky says in that lecture that it's separate (and
compact and recent) and somehow coupled to the general intelligence and
performative areas located elsewhere.

I think we're at a point with language and its neural mechanism where by
analogy it looks like the phenomenon of flight being closely determined by the
exact form and use of a wing. Almost any shape will not do, so there is a form
closely (efficiently) following function. Of course there were preadaptations
for both which provide very inefficient flight or communication abilities, but
something flips and the adaptive utility of the function rapidly sculpts and
elaborates the form in that direction.

About optimality, generally agree though I think following thermodynamic
efficiency seems a deep inductive step as that allows measure of what is
arbitrary or not, e.g. in the work of England that I cited.

~~~
madhadron
> I am following Chomsky, that they're aspects of the same thing; language is
> a faculty in a specific, recently evolved neurological structure.

That is a bold hypothesis, certainly, but not one that you could use as a
basis for argument at this point in time. I also deeply doubt it when viewed
in the context of Wittgenstein's language games. A human and a sheep dog can
very effectively engage in a language game, even subtle ones such as naming.
Nor is it pure behavioral conditioning, as what makes various dog breeds
better at particular things is selection for incidence of particular
behaviors.

Then you go to a gorilla or a parrot, which can engage in quite abstract
language games. Language games with parrots always make you aware of
Wittgenstein's lion, though.

For Chomsky's hypothesis to be true, you would need to have a particular
inflection point where language games become language, and for that inflection
point to be reified as the evolution of a specific neural structure instead of
exaptation of lots of aspects in the brain.

It feels too much like Watson and Crick's nonsense "one gene = one protein,"
which could have been discarded with a modest amount of thought before it was
allowed to damage biology for decades.

> I think following thermodynamic efficiency seems a deep inductive step as
> that allows measure of what is arbitrary or not

Until you look at, say, bower birds and realize that conspicuous waste is a
signalling mechanism for fitness. There you have an evolutionary game that
rewards thermodynamic inefficiency.

------
cs2818
I haven't had a chance to look at it yet but here are the links to the paper
and associated Open Science Framework data if anyone is interested:

Paper:
[https://royalsocietypublishing.org/doi/full/10.1098/rsos.181...](https://royalsocietypublishing.org/doi/full/10.1098/rsos.181393)

OSF Page: [https://osf.io/ga9th/](https://osf.io/ga9th/)

------
LKAndrew
How would it be possible that 40,000 words translates to 400,000 bits?! Am I
missing something here?

~~~
simias
Compression I suppose. Like storing the word "compression" is easier if you
already know the familiar "com-" prefix as well as the noun "pression". Which
is itself easier to remember if you know the verb "to press".

I've been actively learning Portuguese and Russian lately, it's impressive how
much faster I can pick up Portuguese vocabulary vs. Russian. And that's even
for words that don't have an obvious cognate in languages I already know. The
structure of the words, the various building blocks are just so much more
familiar in Portuguese. A word like "atrever" (to dare) doesn't have any
obvious cognate in languages that I know but it just "looks right" it a way
that, say, "atverer" or "aterver" wouldn't. Those last two words sound
distinctly un-Portuguese (I might say, un-Romance) to me. That makes it a lot
easier to remember the spelling.

Eventually as I grow my Russian vocabulary I start making similar connections.
Волноваться is pretty tricky to memorize on its own but it becomes easier when
you know that ~ся is the reflective, ~ть is the common verb ending, волна
means wave and ~овать is a very common building block for Russian verbs.

~~~
mrec
"Compression" applies very much to grammatical variants of words. It's why the
only bizarre irregular verbs in a language tend to be the ones that get used
_all the time_ \- be, do, go etc - because for anything more obscure the brain
just forgets the special case and applies the general rule.

Steven Pinker's book _Words and Rules_ is a great layman-oriented read if this
sort of thing interests you.

~~~
coldtea
It also makes sense why the most common verbs are irregular in most languages:
to have us pick the direct word quickly, instead of the slower way of deriving
it from a rule.

So, they're like "constants" vs calling a function to calculate a value.

~~~
mrec
Not sure I follow. If (big if) there were really a measureable advantage to
having us _" pick the direct word quickly, instead of the slower way of
deriving it from a rule"_, it doesn't follow that irregularity makes that
easier. I could memorize "goed" just as easily as I can memorize "went".

~~~
simias
I took it the other way around: it's easy to memorize "went" because you use
it all the time. If on the other hand a much less common verb like "to
satiate" had a very irregular conjugation then it would regularize pretty
quickly because nobody but ultra-pedants would bother to remember the
exception.

I think a decent real example of that is fiancé/fiancée, those are french
borrowings and have, at least originally, kept the French grammatical gender
inflection. However nowadays I often see people using either spelling in a
gender-neutral way since most people don't bother to learn French grammar for
this one word.

~~~
coldtea
> _I took it the other way around: it 's easy to memorize "went" because you
> use it all the time._

That still wouldn't explain the why of having it like "went" vs "goed".

Sure, it's easy to memorize because we use it all the time, but why have it to
memorize it in the first place, versus something like "goed".

So, this theory (I tried to convey above) said that it being irregular placed
ensured we don't slow down try to derive it from regular rules, but instead
have fast access to a memorized form.

Couldn't we just memorize "goed"? If it's just "frequency of use" that
mattered, "went" and "goed" would work just as well.

But the extra idea is that "goed", being regular, would be too easy for us to
confuse with thousand of other regular verbs, and not use our "fast recall"
mechanism, regardless of that verb being needed all the time.

Not sure if correct - read it years ago. This seems to be related to that:

[https://en.wikipedia.org/wiki/Regular_and_irregular_verbs#Li...](https://en.wikipedia.org/wiki/Regular_and_irregular_verbs#Linguistic_study)

~~~
simias
Your point is interesting but I think you're falling for the same type of
fallacies people often have regarding evolution (if you keep going into water
for a long time over many generations you'll eventually grow gills!).

Natural languages are not designed, they evolve. The irregular nature of the
conjugation of "to go" might just be a remnant of some archaic form and
nothing else, in the same way some argue that the plural of "octopus" is
"octopi" or "octopodes. Does it serve any linguistic purpose? I don't see how,
but it won't stop some people for saying it. Look at the use of the
subjunctive in "if I were you", which is one of the only occurrences of the
subjunctive outside of set-phrases in day-to-day modern English. Is it really
necessary? If one was to say "if I was you" would it lose some additional
nuance or meaning in practice?

I often see people trying to rationalize some language features (such as
arbitrary genders of nouns in many languages) as error correction or some
"optimization" but I'm generally unconvinced. Maybe "I went" is just the
linguistic equivalent of a platypus, some weird byproduct of a very long
evolution with no other intrinsic purpose in the grand scheme of things.

~~~
coldtea
> _Natural languages are not designed, they evolve._

But that's part of my point. I don't say irregular verbs were designed to be
effective that way, but that they were evolved to be effective.

Hence the link with "language acquisition" being involved -- when regular and
irregular verbs developed that wasn't a known theory some "language designer"
could consciously follow. Just something innate that could develop because of
a evolutionary advantage.

In fact, if someone merely designed, they'd probably go for all regular,
rather than regular + irregular, as it's "cleaner".

> _I often see people trying to rationalize some language features (such as
> arbitrary genders of nouns in many languages) as error correction or some
> "optimization" but I'm generally unconvinced._

Tons of language features are indeed optimizations for different things. Cold
climates for example have languages with less vowels (keeping your mouth
closed more).

------
adrianN
I believe it if they provide a program that can speak English in under 1.5MB
total size.

~~~
tobyhinloopen
What does it mean to "speak English"? Does a program that plays a recording of
the word "I" count as a program that speaks English?

~~~
adrianN
I think you know the answer: think about how you find out whether a human
speaks English or not.

------
mannykannot
Of the various estimates that go into this number, I am most skeptical of the
semantics one. I doubt that one can draw a clear line between our semantic
appreciation of language and the totality of our remembered experiences.

------
messe
Makes me want to write a retrocyberpunk setting featuring languages loaded off
floppy disks.

~~~
segfaultbuserr
Do it!

Preferably, the floppy disks should be the 5-1/4 inch 1.2 MiB high-density,
double-sided type, 3-1/2 inch disks sounds good too ;-)

------
werber
This seems very low. Just thinking of all of the different ways you can greet
someone. The inflection and modifiers based on circumstance and audience alone
seems like it would be higher than 1.5mb

------
oeuviz
So, all of it fits on a floppy? Nice.

~~~
timw4mail
Not quite. Considering the most common floppies stored a bit less than 1.44MB.

~~~
segfaultbuserr
One feature of floppy disks is that the recording technology and the format is
independent from the physical medium, the limitations are mostly imposed by
the disk drive, the encoding and the file system, not the medium itself.

Well, this "feature" creates significant compatibility issues as all the
vendors created their own proprietary formats, sometimes with patents. As a
result, together with the economics of scale, often, the standard format is
not the best design, a simpler design that serves as the lowest common
denominator.

On the other hand, due to this feature, it was already possible to store 1,760
KB of data on a 3-1/2 inch HD floppy on a 1986 Amiga. And in 2000, it was
possible to store 32 MiB of data on a standard 1.44 MB floppy disk, by using a
SuperDisk LS-240 drive (although random write is sacrificed, the entire floppy
must be rewritten if a change is needed, like CD-RWs).

I believe the advancements in magnetic recording technology in the past 20
years allows one to achieve even higher capacity on a standard floppy, and it
can be as cheap as early floppy drives if mass produced, only that it doesn't
make sense to do so.

~~~
MisterTea
I always wanted to know how far you could push the capacity of an old standard
density floppy drive if one hacked on modern control electronics. Things like
adding more tracks and variable sectors per track. Though I think you'd be
limited by the magnetic density of the media and physical head size. Still, it
would be fun to see just how much more data you could squeeze onto an old
1.44MB floppy.

------
admax88q
1.5 million what? Fix your title.

~~~
valerij
reminds me of that "hacker steals millions of internet data" joke

------
wmnwmn
Haha wrong

