Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

One of the classic FAQs about Lua is:

"Why doesn't Lua have unicode support?"

I mean, it does, but in the same fashion as ANSI C: you need to either pick a convention for multibyte characters or use a third-party library.

And that's not an accident. One part of Lua's embeddability design is that the standard implementation is written in pure ANSI C, and pure ANSI C ... doesn't support unicode.



I don't understand how writing it in "pure ANSI C" prohibits Unicode support. Python, for example, is written in "pure ANSI C" and does support Unicode. Though doing so correctly and completely is hard, and requires code and data which bulks up what is otherwise a compact embeddable implementation.


Basically, it's as you guessed: Lua's string implementation is pretty much a transparent skin over C strings. The main difference is that Lua checks the boundaries and collects garbage.

It's a real turnoff for lots of people. In terms of "things new to Lua find unattractive", it's second only to 1-based indexing.


Standard C has wchar_t, which is used for Unicode on platforms supporting Unicode.


wchar_t is implementation-specific.


Sure. char is not specified to support ASCII either but it's not really a problem in practice :)

If you start avoiding all the things that are not fully specified by the standard, your C programs will end up doing very little indeed.

Now, it might be that some environments targeted by Lua have very spartan C runtimes: consoles. Anyone have a pointer to the C runtime docs for Xbox 360 and PS3?


wchar_t should never be used in any new code


Just out of curiosity: What is its replacement? If I remember correctly it began its life as the char type for multi-byte character sets and later found use for UTF-(16|32) code units.

Should Unicode strings forcibly be UTF-8 in all programs and use char again (thus suffering from the same problem that Python 2 had where bytestrings and Unicode strings weren't sufficiently distinguished, often leading to programmers using the former for text)?

Or should you use short/int arrays instead and have no support from the stdlib at all for handling them (not that the stdlib is particularly good at handling text)?


If you're using UTF-8, use either uint8_t* (preferred) or char. If you're using UTF-32, use uint32_t. If you're using UTF-16, don't.

As far as library support goes, there are a few good Unicode libraries for C, most notably ICU4C.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: