Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

> "UTF-16" on Windows usually means UCS-2

Wikipedia says it's UTF-16 since Windows 2000: https://en.wikipedia.org/wiki/Unicode_in_Microsoft_Windows

Supporting Windows NT4 or Windows 95 in 2019 is what I would call "very special needs".




UTF-16 is only a convention for file names on Windows. This is why we have things like WTF-8: https://simonsapin.github.io/wtf-8/


It’s works exactly the same way on Linux. Neither kernel nor file system changes the bytes passed from userspace to kernel, regardless whether they are valid UTF-8 or not.

Pass invalid UTF-8 file name, and these exact bytes will be written to the drive. https://www.kernel.org/doc/html/latest/admin-guide/ext4.html says “the file name provided by userspace is a byte-per-byte match to what is actually written in the disk”

Also try this test: https://gist.github.com/Const-me/dcdc40b206fe41ba200fa46b2e1... Runs just fine on my system.


It's not exactly the same on Linux, because Linux doesn't have duplicate pairs of system calls for one-byte character strings and wide strings. Linux system calls are all char strings: null-terminated arrays of bytes. It's a very clear model. Any interpretation of path-names as multi-byte character set data is up to user space.


> Linux doesn't have duplicate pairs of system calls for one-byte character strings and wide strings.

Neither is Windows. These “DoSomethingA” APIs aren’t system calls, they’re translated into Unicode-only NtDoSomething system calls, implemented in the kernel by OS or kernel mode drivers as ZwDoSomething.

Windows system calls all operate on null-terminated arrays of 16-bit integers. It's a very clear model. Any interpretation of path names as characters is up to user space.


I don't know what you're talking about. You said Windows uses UTF-16 and pointed to Wikipedia. I'm only pointing out that that's only true by convention. Windows, even today, does not require that its file names be UTF-16.

Whether Linux analogously does the same or not (indeed it does) isn't something I was contesting.


The file system / object manager is only one part of the whole, though. Object names and namespaces in general will have that restriction, but in user-space there's a lot of Unicode that's treated as text, not a bag of code units. And those things are UTF-16.


Wikipedia is wrong on the technical details.

E.g. the filesystem accepts any sequence of WCHARs, whether or not they're valid UTF-16: https://docs.microsoft.com/en-us/windows/desktop/FileIO/nami...

> the file system treats path and file names as an opaque sequence of WCHARs.

The same is true more generally, there's no validation so anything goes.


From your comment I was replying to:

> "UTF-16" on Windows usually means UCS-2, so you risk losing information if you reencode.

On Windows, you normally call this API to convert UTF-8 to UTF-16: https://docs.microsoft.com/en-us/windows/desktop/api/stringa... As you see, the documentation says it converts to UTF-16, not UCS-2, so no information is lost re-encoding.

And the article you’ve linked says “file system treats path and file names as an opaque sequence of WCHARs.” This means no information is lost in the kernel, either.

Indeed, kernel doesn’t validate nor normalize these WCHARs, but should it? I would be very surprised if I ask an OS kernel to create a file, and it silently changed the name doing some Unicode normalization.

Linux kernel doesn’t do that either, https://www.kernel.org/doc/html/latest/admin-guide/ext4.html says “the file name provided by userspace is a byte-per-byte match to what is actually written in the disk”


I'm sorry if I was unclear but my point was that when you receive a string from the Windows API you cannot make any assumptions about it being valid UTF-16. Therefore converting it to UTF-8 is potentially lossy. So if you then convert it back from UTF-8 to UTF-16 and feed it to the WinAPI you'll get unexpected results. Which is why I feel converting back and forth all the time is risky.

This is one reason why the WTF-8[0] encoding was created as a UTF-8 like encoding that supports invalid unicode.

[0] https://simonsapin.github.io/wtf-8/


> I would be very surprised if I ask an OS kernel to create a file, and it silently changed the name doing some Unicode normalization.

Doesn't OS X do that? AFAIK files names are in NFD there.


Yes, Mac normalizes and decomposes. It's weird.




Consider applying for YC's Fall 2025 batch! Applications are open till Aug 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: