> a chunk of data that can be operated on as a single unit by the hardware
I didn't realize that this concept was referred to by the term "byte". But this clarifies how you are using that term.
Given that usage, I'm not sure why 8-bit bytes are an issue at all for a language like Python that is supposed to be hardware-agnostic; the whole point is that programs should not care about how the data is physically represented in the hardware. Python 3 got rid of the distinction between int and long for the same reason. The only reason the "bytes" object exists at all is to represent binary data, for example data that gets read from/written to files, network sockets, and other I/O, and AFAIK bytes in such data are always defined to be 8-bit chunks of data. (If you really insist on having Python representations of hardware data chunks, there's the ctypes module.)
> I didn't realize that this concept was referred to by the term "byte".
It is common to see "byte" used to mean 8-bit-byte, and also to see the term used for wider units of binary data. If you want to be unambiguous, the word "octet" is commonly employed to mean 8-bit byte.
> a language like Python that is supposed to be hardware-agnostic
That is exactly why you want wide bytes. If you want to represent an integer >255 as fixed-width binary data and you don't have wide bytes, then you have to choose an endianness convention. If you have wide bytes then you can be endianness-agnostic.
> AFAIK bytes in such data are always defined to be 8-bit chunks of data
This is usually true, but not always. For example, UTF-16 and UTF-32 data is specified as 16-bit and 32-bit bytes. This is why you need a BOM.
> there's the ctypes module
Yes, this is why I said it's not a serious problem.
> If you want to represent an integer >255 as fixed-width binary data
But Python 3 doesn't represent integers that way, because Python 3 doesn't put any size limit on integers. There's no way to represent integers of potentially unbounded size as fixed-width binary data.
This is one of the things I meant by "hardware agnostic"--I didn't mean "able to represent the different register sizes of different hardware"; I meant "having a representation that doesn't have any concept of register size at all".
(Python 2 does have a separate int type, which IIRC is either 32 or 64 bits wide depending on the platform. AFAIK this is just a wrapper around the underlying C int, so it isn't affected by any 8-bit byte restriction.)
> UTF-16 and UTF-32 data is specified as 16-bit and 32-bit bytes. This is why you need a BOM.
Um, no. UTF-16 and UTF-32 data is specified as 16-bit and 32-bit code points. You need a Byte Order Mark (BOM) to specify in which order the 8-bit bytes that make up a 16-bit or 32-bit code point appear.
From the Unicode FAQ[0]:
"Q: What is a UTF?
"A: A Unicode transformation format (UTF) is an algorithmic mapping from every Unicode code point (except surrogate code points) to a unique byte sequence. The ISO/IEC 10646 standard uses the term “UCS transformation format” for UTF; the two terms are merely synonyms for the same concept."
Further down, under the table listing the UTFs and their properties:
"In the table <BOM> indicates that the byte order is determined by a byte order mark"
And in the answer to the next question after that:
"UTF-16 and UTF-32 use code units that are two and four bytes long respectively."
Yes, obviously. Variable-length bytes is a feature that Python does not have. (AFAIK the only language that has variable-length bytes is Common Lisp.)
> There's no way to represent integers of potentially unbounded size as fixed-width binary data.
Yes, obviously. To be excruciatingly precise I should have said: if you want to represent a binary data type with N instances where N>255 (e.g. an integer in a range from 0 to N where N>255, or from -N to N where N>127) then you either need bytes wider than 8 bits, or you need to worry about endianness.
> UTF-16 and UTF-32 data is specified as 16-bit and 32-bit code points.
No. Unicode characters are specified as code points, which are just numbers with no specific representation. UTF-16 and UTF-32 are encodings of unicode which use 16-bit-wide and 32-bit-wide bytes respectively to encode those code points. It is only when you serialize those encodings to octets that you need a BOM. When UTF-16 and UTF-32 are used internally to one machine you don't need a BOM.
There are also, as I pointed out before, many algorithms (particularly in cryptography) that are specifically designed to operate on fixed-width binary data wider than 8 bits.
> Variable-length bytes is a feature that Python does not have.
That wasn't my point. My point was that, as far as integer objects are concerned, there is no concept of "byte" at all, not even a fixed length "byte" of 8 bits. There is no concept of "chunk of data operated on as a single unit by the hardware", and there is no concept of "unit of data of the underlying storage for this object". There are just objects representing integers of any value you like.
The only kind of object in Python for which a concept of "byte" in your sense exists at all (if we leave out modules like the ctypes module) is the byte string (str in Python 2, bytes in Python 3). For those objects, you are correct that "byte" is always 8 bits in Python, and there is no way to vary it. But that has nothing to do with the underlying hardware; it is just the data model that Python has chosen for binary data.
> if you want to represent a binary data type with N instances where N>255 (e.g. an integer in a range from 0 to N where N>255, or from -N to N where N>127) then you either need bytes wider than 8 bits, or you need to worry about endianness.
Only if these things are exposed to you at all. In Python, you don't have to worry about this for integer objects because you don't see the underlying storage at all; things like what endianness is used to store integers > 255 are implementation details of the interpreter. Frankly, I find that to be a good thing: if I wanted to worry about stuff like that, I'd be programming in C, not Python. But of course different people will have different needs and preferences.
> UTF-16 and UTF-32 are encodings of unicode which use 16-bit-wide and 32-bit-wide bytes respectively to encode those code points.
I understand that you prefer to use this terminology; but my point in that particular comment was that it is not the terminology that is used in the Unicode documentation, which is the closest thing we have to an "official" terminology for Unicode. As is shown by what I quoted, that documentation clearly uses the term "byte" to mean 8 bits; it refers to UTF-16 and UTF-32 as encodings that use 2 or 4 "bytes" to encode a code point. It does not use the term "byte" to refer to 16 or 32 bit chunks of data. So it seems misleading, or at least potentially misleading, to use the term "byte" to refer to anything other than 8 bits when talking about Unicode encodings, since the Unicode documentation does not give the term "byte" that meaning.
Let me try to re-state the point I was trying to make: the only reason there is anything special about 8 bits as a unit of data is because hardware is built to operate on units of 8 bits at a time. Whether you call these things "bytes" or "words" doesn't matter. You can call them florbs for all I care.
The fact that florbs are 8-bits wide is mainly for historical reasons. There is no particular reason to make the magic number be 8 (except that it's a power of 2), and indeed there have been historical examples of hardware with florbs of other sizes. The Intel 4004 had 4-bit florbs. The PDP-8 had 12-bit florbs.
Modern hardware is built to natively handle florbs of multiple sizes, typically 8, 16, 32 and 64 bits, but sometimes other sizes as well (e.g. 80 bits for extended precision floats, or 128 for SSE5 instructions). Because of this, many algorithms are designed to operate on binary data in these sizes, and because of that it can be helpful to have binary data in these sizes exposed in your programming language. Python exposes 8-bit florbs, but not other sizes of florbs. Common Lisp has arbitrarily sized florbs. If you're doing certain things (like crypto) having access to florbs larger than 8 bits can be useful. That's all.
> the only reason there is anything special about 8 bits as a unit of data is because hardware is built to operate on units of 8 bits at a time.
More precisely, hardware at the time that "the natural unit of data" was becoming a standardized convention was built to operate on units of 8 bits at a time. As you say, the 8-bit convention is mainly for historical reasons, and hardware today can handle larger chunk sizes.
> Python exposes 8-bit florbs, but not other sizes of florbs
To the extent it exposes florbs at all, yes. But that does not mean you are stuck with 8-bit operations in Python in all cases. For example, operations on Python floats use the IEEE standard data chunks, which, as you note, are not 8-bit. Operations on Python integers are not restricted to using 8-bit CPU instructions; they will use whatever register sizes the hardware allows (more precisely, the hardware the Python interpreter was compiled for--if you run a 32-bit Python interpreter on a 64-bit machine, you won't be using 64-bit register operations). It is true that the programmer has no control over the data chunk sizes that Python uses; they are hard-coded into the interpreter.
> Common Lisp has arbitrarily sized florbs.
Can you give an example of how this capability in CL is used?
The best example I can think of is implementing certain cryptographic algorithms, particularly symmetric ciphers and cryptographically secure hash functions, which are often defined in terms of 32 and 64-bit chunks and mod 2^32 and mod 2^64 arithmetic.
Why not just use C? Because the C language spec has "features" that allow a C compiler optimize away parts of your code that you really don't want optimized away in crypto code. For example, if you do this:
int secret_key[SIZE];
...
for (int i=0; i<SIZE; i++) secret_key[i]=0; // Clear secret key so attacker can't read it
A C compiler is allowed to get rid of that code if it can prove that it doesn't contribute to the final result (the "as-if" rule). Forcing a C compiler to not do such things is generally very tricky.
Not really. Just put the clearing function into another translation unit.
clear_memory(secret_key, sizeof secret_key);
(And don't enable any wacky whole program link-time optimizations. Those violate ISO C by continuing semantic analysis into translation phase 8. ISO C makes it clear that tokens are "syntactically and semantically analyzed" in phase 7, as translation unit. Then in phase 8, only references are resolved to link. Doing any semantic analysis to optimize things at link time violates the conceptual model thereby given. GCC has support for this but it has to be explicitly enabled; moreover, there is more to using link-time optiizations than just passing options.)
The current translation unit must really call clear_memory and really pass it the pointer to the secret_key, whose contents have to be settled. The writes performed by clear_memory really have to take place, because the caller depends on it; clear_memory has no idea that the object is dead (having no next use) in the calling translation unit.
The main problem is not getting the clearing not to be elided, but with stray copies of the data being elsewhere. The C programmer doesn't have visibility and control over all the storage areas where a datum may end up. If secret_key is really cleared, is that enough?
I didn't realize that this concept was referred to by the term "byte". But this clarifies how you are using that term.
Given that usage, I'm not sure why 8-bit bytes are an issue at all for a language like Python that is supposed to be hardware-agnostic; the whole point is that programs should not care about how the data is physically represented in the hardware. Python 3 got rid of the distinction between int and long for the same reason. The only reason the "bytes" object exists at all is to represent binary data, for example data that gets read from/written to files, network sockets, and other I/O, and AFAIK bytes in such data are always defined to be 8-bit chunks of data. (If you really insist on having Python representations of hardware data chunks, there's the ctypes module.)