
Text Editing Hates You Too - panic
https://lord.io/blog/2019/text-editing-hates-you-too/
======
pcr910303
As a person living with the CJK languages (I’m specifically a Korean), I find
that some of these problems are prominent, even in the 21th century.

There are an excessive amount of programs that conflate key presses & text
input, and ones that don’t consider input methods.

I use macOS & Linux, and while the default text handling system called Cocoa
Text System in macOS handles input methods well, almost all applications that
implement it’s own, like big apps like Eclipse and Firefox, don’t get this
right.

On Linux, it’s terrifying; I’ve never seen any app that allows input systems
to work naturally, and after a week of use you get used to pressing space &
backspace after finishing every Hangul word. The Unix-style composability they
want (apps should work whether or not input methods are used - and looks like
Linux users that use Latin characters don’t use any input methods (opposed to
macOS where Latin characters are input by a Latin input system), so looks like
this state will persist.

About the emoticons, I’m not that concerned with that since most (if not all)
users won’t really input the color modifier separately (or even encounter
files that have a separate one), so you can just select a sensible behavior
like the #2 or 3 or 4. Users who understand the color modifiers, and other
Unicode fiasco will understand what is happening under the hood, and ones that
don’t will just think the file is broken and none of the behaviors will make
sense, whatever you do.

~~~
SiVal
For decades I've been saying that _all_ text input should go through OS-level
IMEs (input method editors). For more complex scripts, the need is obvious,
but even writing in English, we can get great benefits from a system that
expands abbreviations, replaces easy-to-type sequences with proper Unicode
chars, runs little scripts and inserts the output, gives you quick
dictionary/thesaurus lookups, gives you emmet-style powers, etc., whenever
you're writing and in any app.

Of course, everyone's preferences will be different, so you'll get default
IMEs with default configs, but with the idea that you can reconfigure or
replace them entirely with systems that work the way _you_ want to work
everywhere you input text. There have been utilities that do this sort of
things for decades, but they've always been treated as clever hacks rather
than standard text input.

In other words, instead of powerful text input methods being the exception,
they would be the rule, and apps that didn't use them would be the exception.

~~~
tempguy9999
> we can get great benefits from a system that expands abbreviations, ...
> emmet-style powers

I can't agree here (excepting perhaps the unicode char replacement which a
non-english speaker needs to comment on the viability of, vs actually having a
non-english keyboard (excepting huge alphabets like chinese where that is used
already AFAIK)).

Writing is fundamentally a thinking process, the text entry for a touch typist
is relatively quick. If you are to propose expanding abbreviations, how much
time do you expect to save? I mean, actually measured it?

> runs little scripts and inserts the output

What is the purpose of this?

> gives you quick dictionary/thesaurus lookups

I use those perhaps once a week or less. If you use it say 3X a day, you'd
save little time, perhaps a couple of minutes.

Dunno what emmet powers are though.

MS word & libreOffice does some of what you want and the first time I install
them, I spend several minutes tracking down each setting and turning them off
- they drive me bonkers. They think they know what I want but they don't.
Touch typists can hit many keys a second and kind of pipeline their typing.
Having the input modified automatically is rarely useful IMO.

Your idea may be good but like many other ideas such as graphical programming,
except in restricted cases they don't work. Perhaps if you measured it I'd be
convinced but I can't accept it now as an obviously great benefit.

~~~
TeMPOraL
> _Writing is fundamentally a thinking process, the text entry for a touch
> typist is relatively quick. If you are to propose expanding abbreviations,
> how much time do you expect to save? I mean, actually measured it?_

Not quick enough. I think faster than I type, and I type fast. It's fine until
I start getting impatient with myself. Call it micro-impatience, a flash of
irritation in which you're suddenly conscious of not having finished typing in
the thought. It's distracting.

Honestly, I like parent's idea; a good chunk of Emacs's awesomeness and the
reason people like me use it as an operating system is because of that -
unified, fully configurable and expandable text-based interface. I often wish
to have something like it system-wide, because standard UIs are very far from
optimum ergonomy-wise. But then again I wouldn't trust Apple or Microsoft to
do it right; they'd quickly find a way to dumb it down, or restrict the
extensibility in the name of security.

~~~
lioeters
> the reason people like me use [Emacs] as an operating system is because of
> that - unified, fully configurable and expandable text-based interface. I
> often wish to have something like it system-wide..

I can totally imagine that. Underneath all the GUI layers, every operating
system and application has (or has the potential of) a fully text-based
interface. There's just no standard or integration, and tools that allow that
(like a system-wide middleware) haven't caught on, I guess. Maybe in an
alternate historical timeline, such a feature could have been a fundamental
layer of an OS.

From the grandparent comment:

> a system [with powerful text input methods] that expands abbreviations,
> replaces easy-to-type sequences with proper Unicode chars, runs little
> scripts and inserts the output, gives you quick dictionary/thesaurus
> lookups, gives you emmet-style powers, etc., whenever you're writing and in
> any app.

Yes, yes - and the last point: in any app. I picture it like how TCL can
script other programs, even ones that weren't designed to be "remote
controlled".

~~~
madhadron
> Underneath all the GUI layers, every operating system and application has
> (or has the potential of) a fully text-based interface.

Why do you say this? There is nothing fundamental about text. Is there
something fundamental about text in Smalltalk? Or AmigaOS?

> There's just no standard or integration, and tools that allow that (like a
> system-wide middleware)

COM on Windows. Scripting interface to apps on MacOS. They're there.

~~~
lioeters
Yeah, I had some vague doubts while I was writing that comment. I guess I
meant "text-input based", or maybe better to say "keyboard based" with a
system-wide/application-agnostic middleware of some kind.

~~~
madhadron
"Accessible via a programming language" perhaps?

~~~
TeMPOraL
Not that. More like, UX paradigm more fixed and forced on applications, but
also being customizable and user-programmable _externally_ to any given
application. So that e.g. you could have a system-wide autocomplete/code
completion, whether you're in a code editor or text editor or in a dialog box
of some other program somewhere; that system-wide autocomplete would be
configurable and trivial to extend or replace wholesale with another widget.

This is a reality within Emacs (which really is a 2D text OS running lots of
applications inside, including a text editor), and being text-based does play
a role. When it's very hard to draw arbitrary pixels on screen and most of all
apps deal with text, it's easy to make a large set of very powerful interface
tools, and it's easy to pull data out of an app and put data into it, whether
the app intended it to happen or not.

In the back of my mind, I sometimes wonder how something like Emacs could be
made with modern browser canvas, to enable cheap rich multimedia, while
retaining the ability for inspection and user-programmability. Introducing
arbitrary GUIs is hard, because next thing you know, half of the stuff is
drawing to canvas directly and it's all sandboxed away from you.

~~~
lioeters
> user-programmable externally to any given application

I think this is why it reminded me of TCL, specifically the "expect" command
that can script apps that know nothing about it. From the Wikipedia page, the
TCL Expect extension: "automates interactions with programs that expose a text
terminal interface".

So how I imagine this "Emacs as an OS" paradigm you're describing, is that it
mediates interactions with any and all apps that expose a text input/edit
interface, to allow programmatic customizations.

Like I'd love to script my own shortcuts for Firefox (or other apps) -
possibly with multiple steps, taking input from some config file, or sending a
link to another app.. Or, as you mentioned, Emmet-style expansions that work
in any input field or textarea..

------
dvt
This post reminds me of one of Jon Skeet's seminal SO answers[1] on
subtracting dates. Things we take for granted are hard. Dates are hard, text
is hard, graphics are even harder. We stand on the shoulders of giants. It's
important to stop and smell the roses every now and then.

[1] [https://stackoverflow.com/questions/6841333/why-is-
subtracti...](https://stackoverflow.com/questions/6841333/why-is-subtracting-
these-two-times-in-1927-giving-a-strange-result)

~~~
ggm
I know its not the answer, but I always convert time into time in seconds
since the epoch and perform the arithmetic.

epoch time understands negative numbers:

% date -ur -28501440617 Mon Oct 29 05:09:43 UTC 1066 %

~~~
eesmith
It's easy for me to write something in Russian, I just hire a Russian
translator for me.

That is, "convert time into time in seconds since the epoch" is the hard part.
All you've done is make it someone else's problem, while the question Skeet
answered asked specifically about the time difference between 1927-12-31
23:54:08 and 1927-12-31 23:54:09 in Singapore.

~~~
ggm
Yes. This is beautifully expressed. And 'hiring a russian interpreter' is
almost always the right way to fix this kind of problem. POSIX would seem to
me to be the russian interpreter of choice here. Is java a language which has
no word for the colour green? _in the sense where the eponymous russian
interpreter has to say: "concept which cannot adequately be expressed in
russian" or something_

The POSIX spec exists for this reason. This feels like a variation of "omg
utf-8 is hard" which is why people embed utf handling in the language python2
-> python3

~~~
eesmith
It seems less than graceful to respond to articles which point out the
existence and difficulty of problems by saying it's fixed. The entire point is
to show how it's more difficult than many think, not elide the issues.

What does POSIX have to do with anything? Eg, the tz database isn't POSIX.

~~~
ggm
The POSIX spec of date time specified conversions into a neutral form. Maybe
it's C centric thinking but it felt like a problem which has existed and been
solved by writing a 'russian interpreter' solution: convert the values on the
edge to a canonical form and use functions which manipulate the canonical
form. The TZ files i only mention because within the limit of modern epoch
Times and dates they do a pretty good job of handling the minutia of what a
local time means in any given period.

I'm sorry if I came across as churlish. Do you feel this is a problem of
concept in abstract, or in java, or in java implementation or what? I feel
it's a problem which I solve, by standing on the shoulders of smarter people
and using the code they wrote. I don't reimplement the wheel because I
distrust my own wheelmaking skills.

I'm sure there are heaps of non-trivial corner cases I don't have to deal with
and I do not mean to minimise work done to explain a problem.

~~~
eesmith
dvt's top-level comment was "Things we take for granted are hard."

Your response took things for granted.

dvt originally wrote "We stand on the shoulders of giants."

You commented "I feel it's a problem which I solve, by standing on the
shoulders of smarter people and using the code they wrote."

It came across as totally missing dvt's point.

I picked Russian because I was thinking of an essay by Douglas Hofstadter.
Quoting myself at
[https://news.ycombinator.com/item?id=16289783](https://news.ycombinator.com/item?id=16289783)
:

> I think it was in Godel, Escher, Bach where he talks about the problem of
> translating a line from Russian, along the line of "He lived on B____
> street." It's possible from the story, which takes place in a real city, to
> figure out which street that was. Let's say it was "Main Street". Does the
> English translation keep the original Russian word and initial? Does it
> translate to "Main Street" and replace the "B" with an "M"?

You can use a Russian-English translator to get a translation into English.
But that doesn't help understand the hard problems involved, which was the
point of Hofstadter's essay. (See, for example, his "The Shallowness of Google
Translate" in The Atlantic; HN comments at
[https://news.ycombinator.com/item?id=16267363](https://news.ycombinator.com/item?id=16267363)
and in dupes.)

~~~
chrisweekly
This reminds me of Milan Kundera's "Testaments Betrayed", which expresses the
profound challenge (near impossibility) of translations doing justice to
original works.

------
alkonaut
How much is poor text input eroding the use of non-latin script worldwide?
I.e. how many just give up instead of using poor input?

As an example: I use a Swedish keyboard. If I'm in a situation where ö is
missing from the input (because the app is stuck in an en-US keyboard layout
say, so I have to hit a modifier ¨+o to type an ö) then there is a very good
chance my communication with my colleagues would just naturally be in english
instead. I'd sigh and just give up using it. Rather than using an o for my ö
I'd just type it in english instead.

Is this a thing in e.g. asia, israel, or the arab world? Do kids that speak
english communicate more in english in cases where the input doesn't let them
communicate easily using their preferred script? Are there new "hybrid"
languages popping up in electronic communication where languages that use non-
latin scrpit are written in latin in e.g. text messages? (You could argue that
emoji is just that but the other way around I suppose)

~~~
babuskov
> I use a Swedish keyboard

AFAICT, Swedish keyboard has a dedicated key for ö. You just press it. I don't
understand how an app can get stuck on keyboard layout if you set the system
keyboard layout to Swedish. Could you give an example?

Or are you're actually using US keyboard with Swedish layout and there is no ö
key on it?

~~~
kstenerud
It happens to my wife's German USB keyboard in Windows 10. One day it's
working fine, the next day pressing Z outputs a Y and Ä outputs a semicolon.
I've had tons of problems with Japanese input in Linux and Mac (but not in
Windows).

Multilanguage support is a function of how many customers demand it, and aside
from English and Japanese, there hasn't been enough economic incentive to take
it seriously.

~~~
LaxisB
the windows issue might also be caused by accidentally pressing the hotkey
(alt+shift i believe) to switch layouts. it's super easy to mis-trigger

~~~
raverbashing
This is the correct answer, if you have two keyboards configured it's not
uncommon to switch between them accidentally.

------
defanor
While spending a lot of time with monospace fonts and mostly ASCII characters
(programming and writing, terminal emulators, IRC/mail/MUDs/feeds in Emacs)
and working on a hobby project involving text rendering and selection (with
potentially proportional fonts and Unicode), I keep wondering whether it's
even worth all the trouble.

In addition to what's mentioned in those articles, on larger documents
rendering speed also matters; word cache (in addition to mentioned glyph
cache) helps somewhat, but complicates other things even further, and
calculating total document height (for scrolling) still requires to calculate
all the positions, wrap the lines, etc, which in turn requires to render all
the text first. Perhaps one can introduce yet another mechanism to only
estimate that at first, and then elaborate the estimation in background, but
that's yet another opportunity for bugs to creep in (and additional
complication, of course). On the contrast, the programs that just use
character grids (and perhaps don't handle bidirectional texts well) can be
both faster and simpler, while still are quite capable of rendering texts,
even with various styles/decorations and occasional images, in many languages.

There seems to be plenty of accidental complexity even in handwriting, but
once it is used for computing and text manipulations are involved, the task
appears to be much harder than it has to be.

~~~
pcr910303
> While spending a lot of time with monospace fonts and mostly ASCII
> characters (programming and writing, terminal emulators, IRC/mail/MUDs/feeds
> in Emacs) and working on a hobby project involving text rendering and
> selection (with potentially proportional fonts and Unicode), I keep
> wondering whether it's even worth all the trouble.

Yeah... and that's why FireFox & Eclipse still doesn't get CJK character input
right. The percentage of people that must use characters not included in ASCII
is much bigger than people who do. It is worth the trouble, please consider
users outside of the US & England.

~~~
j-pb
I think it would have been better to have every body just learn english as
their second (computer usage) language. I'm german and my english was horrible
until I switches my computer and my websites to english ones.

Imagine a world where everybody speaks at least one common tongue, we could
have so much more peace and understanding.

English becoming the esperanto of the world is an accident but I don't care
which language it is, as long as it has less than 255 letters. (Hawaiian would
have fit into a nibble which is cool)

~~~
tralarpa
> better to have every body just learn english as their second (computer
> usage) language

Why not Chinese? It would have many advantages:

\- Text layout would be much easier (Chinese fonts are mostly mono-space).

\- In addition to the first point, Chinese characters can be easily used for
vertical text. Looks much nicer in narrow, tall buttons than latin letters.

\- Chinese characters are (mostly) pronunciation independent. No
they/there/their anymore.

\- It has been shown that dyslexic children have less problems recognizing
Chinese characters than latin letters.

\- Chinese characters are part of the culture heritage of billions of people
(both Chinas, Japan, Korea) and not just something they learn at the
university.

\- Old-school C enthusiasts can now have meaningful variables names while
keeping their self-imposed three-character limit (or was it four?)

\- Finally, everybody will use wchar_t

\- Edit: But there is one disadvantage. There is no Chinese character for "/s"
which some people here seem to need

~~~
j-pb
Logographic writing systems are just a pain in the ass. You have to create
huge fonts, you waste ridiculous amounts of space, and you need phonetic
descriptions of new words anyways.

Like I said, Hawaiian would be cool since it fits inside a nibble with 16
characters, although the words tend to get long.

~~~
K0SM0S
Considering how the brain seems to process words when reading ('glancing in
context' more than analyzing each letter), I'm pretty sure the length of words
does not make reading much longer. Writing is another matter but with auto-
predict you could probably shove half the typing off on average.

As a software engineer and cousin of _Stitch_ , Hawaiian gets my vote!
:handclap:

------
danShumway
I've lightly advocated for a while that emoji shouldn't be part of the Unicode
standard at all. I'm sure there are things some advantages, I'm sure there are
other considerations I'm not thinking of, but it just seems like a really bad
idea to stuff the Unicode standard.

I don't know the official name or who came up with it, but I use Slack's entry
format exclusively in every application.

:thumbsup: :pelipper_blushing: :angry_cat: :apple:

If the application can detect that as an emoji and swap it out, fine. If it
can't, I don't change my format. My preference would be if applications left
emoji in that format, and just rendered them differently at display time.

The advantage of having emoji just be a purely clientside rendering feature,
and behind the scenes all fall back to normal text is:

a) they can be easily aliased across multiple languages (:cat: :gato:)

b) if you paste an emoji into an application that doesn't support them, you
don't get an unrecognizable character. Progressive enhancement!

c) it's accessible when copied and pasted into a pure UTF-8 text format. It's
just better blind-accessible in general.

d) it's more forward compatible. I can use :cthulhu: right now without waiting
for it to get added to the standard.

e) get rid of modifiers. Like, seriously, just get rid of them. Emoji aren't
programmatically generated, you still need to draw one image for each modifier
combination, and you still need to program support for each one. So, what's
the advantage of using modifiers over just adding multiple glyphs? They're
just there to save space in the character list, which is only a problem
because emoji are in Unicode. :smile: :fake_smile:

f) better support for custom emoji in general. Basically every platform has
custom emoji, and it's _weird_ because half of your emoji are standardized and
half aren't. And then whenever new emoji get added to the standard, if they
conflict with your custom emoji your app breaks.

I would hesitate to standardize emoji at all, beyond having a consortium that
says, "this is what :thumbsup: means, you can extend on top of this as you see
fit."

It feels like extra complexity for no benefit other than, "we need a
standard".

~~~
philplckthun
I think this works great for apps like Slack, in user-land so to speak, but
isn't realistic for the Unicode standard, not only because these entry formats
are in English.

Modifiers and combinators aren't exclusive to emojis, but apply to all kinds
of glyphs in other languages and writing systems as well. Arabic script even
has some common ligatures for common expressions.

A lot of complexity simply doesn't stem from emoji in Unicode, a lot of the
complexity comes from all the writing systems that Unicode supports.
Admittedly, emoji are kind of an oddball addition to Unicode, but they're by
far not the most complex part of it.

~~~
derefr
And even in Slack/IM apps, custom emoji codels only “work” because people
aren’t often trying to 1. interoperate with external services using, or 2.
parse archived logs of, arbitrary message text.

If either of these were common (e.g. slack bots that tried to parse semantic
meaning from regular text rather than responding to commands; or Slack logs of
OSS communities being public-access on the web) then you’d see a lot of people
up-in-arms around the fact that these custom codels are used.

But since text in these group-chat systems is private, ephemeral, and mostly a
closed garden, it never bubbles up into becoming an issue anyone else has to
deal with.

(Though, on a personal note, I wrote my own VoIP-SMSC-to-Slack forwarder
because Slack is a much better SMS client than any of the ones built into VoIP
softphone apps, and I’m irritated every day that Slack auto-translates even
Unicode-codepoint-encoded emoji from a source postMessage call, into its own
codels in the canonical message stream. I don’t _want_ to send my SMS contact
“:thumbs_up:”, I want to send them U+1F44D!)

~~~
danShumway
Does Unicode make this any better?

Let's say Matrix supports custom emoji, and it wants to interoperate with
Discord.

Does representing those user-added emoji as custom Unicode entries instead of
plain text make it easier or harder to interop with Discord?

~~~
derefr
Think of Unicode like HTML. What’s better for interoperation and machine-
readability: a custom SGML entity (like you could use up through HTML4); a
custom HTML tag; or a normal HTML tag with an id/class attribute that applies
custom CSS styling?

One way to encode a ‘custom emoji’ would be encoding it as a variation of some
existing emoji. Use an as-yet-unused variation-selector on top of an existing
emoji codepoint, and then “render” that codepoint-sequence on receipt by the
client back to an image (but in a way where, if you copy-and-paste, you get
the codepoint-sequence, not the image. In HTML, you’d use a <span> with
inline-block styling, a background-image, and invisible content.) This is
pretty much what Slack was doing with the flesh-tone variation-selectors,
before Unicode standardized those. But you can do it for more than just
“sub”-emoji of a “parent” emoji; you can do it to create “relatives” of an
emoji too, as long as it’d be semantically fine in context to potentially
discard the variation selector and just render the base emoji.

Or, if your emoji could be described as a graphical (or _more_ graphical)
depiction of an existing character codepoint, you could just use the “as an
emoji” variation-selector on that codepoint.

Or, rather than a variation-selector, if you have a whole range of “things to
combine with” (i.e. the possibilities are N^2), you could come up with your
own private emoji combining character for use with existing base characters.
The “cat grinning” emoji U+1F639 could totally have been (IMHO _should_ have
been) just a novel “face on a cat head” combining-character codepoint, tacked
onto the regular “face grinning” emoji codepoint. Then you could have one such
combining-character for any “head” you like! (And this would also have finally
allowed clients to explicitly encode “face floating in the void” emoji vs.
“face on a solid featureless sphere” emoji, where currently OSes decide this
feature arbitrarily based on the design language of their emoji font.)

And, I guess, if all else fails, you could do what Unicode did for flags
(ahem, “region selectors”), and reserve some private-use space for an alphabet
of combiner-characters to spell out your emoji in. That way, it’s at least
clear to the program, at the text-parsing level, that all those codepoints
make up _one_ semantic glyph, and that they are “some kinda emoji.” Custom-
emoji-aware programs (like your own client) could look up _which one_ in a
table of some kind; while unaware programs would just render a single unknown-
glyph glyph.

I don’t suggest this approach, though—and there’s a reason the Unicode
standards body hasn’t already added it: it’d be much better to just take your
set of emoji that you’re about to have millions of people using (and thus
millions of archivable text documents containing!) and just send them to the
Unicode standards body for inclusion as codepoints. Reserving emoji codepoints
is _very_ quick, _because_ the Unicode folks know that the alternative is
vendors getting impatient and doing their own proprietary thing. Sure, OSes
won’t catch up and add your codepoint to their emoji fonts for a while—but the
goal isn’t to have a _default_ rendering for that character, the point is to
encode your emoji using the “correct” codepoint, such that text-renderers 100
years from now will be able to know what it was.

So, please, just get your novel emoji registered, then polyfill your client
renderer to display them until OSes catch up. Ensure your glyph is getting
sent over the network, and copy-pasted into other apps, as the new Unicode
codepoint. Those documents will be _correct_ , even if the character doesn’t
render as anything right now; if the OS manufacturers think the character is
common (i.e. if it ever gets used in text on the web or in mobile chat apps),
they’ll provide a glyph for it soon enough. And, even if the OS makers never
bother, and you’re stuck polyfilling those codepoints forever, there’ll still
be entries in the Unicode standard describing the registered codepoints, for
any future Internet archaeologists trying to figure out what the heck the
texts in your app were trying to communicate, and for any future engineers
trying to build a compatible renderer. (Consider what Pidgin’s developers went
through to render ICQ/AIM emoji codels. You don’t want to put engineers
through that.)

~~~
paulddraper
> So, please, just get your novel emoji registered

Yeah I'm not gonna register my company icon.

------
NickGerleman
I until very recently worked on Microsoft Word. The whole problem gets even
more complicated when you add support for richer content like formatting,
images, comments, etcetera. Sprinkle in requirements for things like three-way
merge, simultaneous editing from multiple authors, undo behavior on top of
that, and the amount of cross-cutting complexity for something seemingly
simple can be absolutely astonishing.

~~~
edwinyzh
Hey Nick, I'm glad to hear about that you are in the Word team:) I'm working
on a document manager and static site builder based on Mirosoft Word
([https://docxmanager.com/](https://docxmanager.com/)) (The upcoming version
will have a more user-friendly tabbed UI)

~~~
pitaj
He specifically said that he is not longer on the Word team.

------
be5invis
Let me count all the features that a proper rich text box should support:

    
    
      · Input methods, including...  
        · Composing IME  
        · Speech  
        · Handwriting  
        · Emoji picker  
      · Complex text rendering, including...  
        · Bidi  
        · Shaping  
        · Complex-script-aware editing  
        · Complex-script-aware find, replace, etc.  
      · Accessibility
    

And what they can choose to support:

    
    
      · Advanced typography, including...  
        · About line breaking...  
          · Hyphenation  
          · Optimized line break (Knuth-plass, etc.)  
        · About microtypography  
          · OpenType features (ligatures, etc.)  
          · OpenType variations  
          · Multiple master fonts  
          · Proper font fallback ← It is not a simple lookup-at-each-character process  
        · Advanced Middle East features, including...  
          · Kashida  
        · Advanced Far East features, including...  
          · Kinsoku Shori  
          · Auto space insertion  
          · Kumimoji  
          · Warichu  
          · Ruby  
        · Proper vertical layout, including...  
          · Yoko-in-Tate  
      · Inline objects and paragraph-like objects, including...  
        · Images  
        · Hyperlinks  
        · Math equations ← This is really hard.

------
pcr910303
For people who are interested in the HN post mentioned in the article: here it
is.

Text Rendering Hates You[0]

[0]:
[https://news.ycombinator.com/item?id=21105625](https://news.ycombinator.com/item?id=21105625)

------
saagarjha
I once wrote a simple LaTeX renderer for a project, and boy was it hard. And
all I had to do was support a subset of the whole thing! Even with clear
biases towards left-to-right languages and just a subset of the LaTex, it was
a nightmare: you'd do something, but realize that it wouldn't render correctly
in a certain situation, rewrite the code to include more context or do another
layout pass, and hit another issue somewhere else. It was slow work,
complicated to debug, and utterly stymied many refactoring efforts. I can't
even imagine how much work must go into doing this quickly and correctly for a
language that's much more complex…

------
lykahb
I did quite a lot of hacking around the text selection when extending CKEditor
for XML editing.

In my opinion there are several reasons why text input and selection are so
difficult:

1\. The behavior is complex.

2\. The behavior resists being formally defined. It differs across the user
interfaces, such as URL bar, rich text editor, terminal, etc.

3\. The requirements are evolving. The recent big change - the emojis - is
still not universally supported.

4\. The reasons above tend to add accidental complexity to the API that will
only grow over time.

I feel like the best approach would be to create a mathematical theory
describing the text input semantics. This approach worked quite well for other
complicated areas in CS, such as concurrency or memory management.

------
indentit
This is the first time I've ever seen it explained how input methods allow
users to write in languages like Chinese using the Latin letters A-Z:
phonetics. I had always wondered about this, it seemed like some arcane magic
to me lol, how people knew what to type.

It's now clear that for someone like me - whom only understands a few Latin
based languages - a super comprehensive tutorial would be required if one
wanted to understand enough about other languages to be able to work on a text
editor etc. I guess ideally all teams working on such projects would have an
experienced team member whose native language isn't Latin based.

~~~
hyh1048576
Chinese here. To be a little picky, phonetics is just one of the several
methods to input Chinese. (Pinyin is the dominant one, and yes it is
phonetics, other ways uses shapes of the characters and maybe a lot faster,
see
[https://en.wikipedia.org/wiki/Wubi_method](https://en.wikipedia.org/wiki/Wubi_method)
for example.)

Speaking of input methods I always with there are good English input methods,
it will be useful too! For example if user enters "compre" the input methods
goes 1\. comprehension 2. comprehensive (with an order depending on the
conditional probability -- there is some interesting math behind input
prediction, and with their help input can be a lot smooth, however I only see
it used on phones not computers.)

~~~
pbhjpbhj
>however I only see it used on phones not computers //

IDEs do prediction for input.

OpenOffice used to predict words longer than say 5 characters, IIRC, it might
have been back in the Star Office iteration. Come to think of it I write in a
widower so little now I've only just realised that this feature might have
been disabled as a default (or removed).

------
dandare
Tangential: Microsoft's automatic selection of whole words and sentences is
the single biggest productivity drain in my professional career (all my
attempts to disable it failed so far).

~~~
spongeb00b
For anyone having to deal with this on Microsoft Word on the Mac it can be
disabled through Word’s preferences.

~~~
dandare
The setting does not apply to Outlook when writing/editing emails, only when
reading emails. Then, from my experience, the setting will be reverted with
the next Word update :(.

------
jakear
Not mentioned in the article, but an interesting point of difficulty in
editors is the character transposition command (Ctrl-T on most MacOS
applications, stemming from emacs I imagine):

TextEdit and VSCode are the only editors I know of to handle transposing emoji
in a reasonable way. VSCode will separate the emoji from the color modifier.
TextEdit keeps them together, other editors I know of corrupt the data by
slicing surrogate pairs or just no-op the command.

~~~
Pxtl
Wait, there are people who use the ctrl-T command? That thing is the bane of
my existence because I have muscle-memory for "new tab" in firefox but when I
do it in a text editor for "new file" out of habit it just jumbles text.

~~~
jakear
I’m a pretty bad typist, so it’s helpful for when I need to fix a word and I
can do it in one keystroke. Also my new tab binding is command-t.

------
Thorentis
Interesting! I hadn't thought this deeply about text editing before.

I disagree that the "Bad #3" example in the "Emoji Modifiers" section is
actually bad though - it's the outcome I would expect of an editor.

~~~
Mathnerd314
Another option would be that deleting the 'a' doesn't completely delete it but
instead replaces it with a zero-width space or zero-width non-joiner, so it
looks like Bad #1 (but is Unicode-compliant) and hitting delete again gives
Bad #3.

The whole example is a bit contrived though, nobody is going to enter in a
skin tone modifier character by hand in daily use. They'll select an
appropriately-colored emoji.

~~~
jakear
IMO, having delete trigger a zero-width space insertion would be the worst
option. I mention in a sibling comment that VSCode gets around this by having
two separate logical carat positions combined into a single visual position.
So the byte offset of the cursor changes as expected, while still maintaining
"Unicode correctness", for whatever thats worth.

------
aasasd
Every time I hear someone is implementing even a part of text editing, I begin
to snigger. (Ahem Workflowy cough.)

I'll gladly put these two articles in my reading list, to go over some evening
while reclined comfortably and sipping wine—so I then can laugh and slap my
knees even harder when a new text editor is mentioned.

------
tomaszs
Some years ago i was working on a project to autocorrect text with high level
written text recognition. I have used standard text editor. Man, i thought 1
day about rewriting text field. But angels stopped me from it. People dont
know how hard it is!

------
phil9987
This is incredible, thanks for sharing. I had the exact same thoughts the
author is describing in the beginning - how hard can it be? It turns out super
hard. And one shouldn't forget that it is one of the few input methods to our
computers. With voice not working properly and drawing on the mouse pad /
touchscreen being too slow I would argue it is still the number one input
method. So the expectations towards UX are extremely high and there should be
no faults. I admire anybody who works on some kind of text input mechanisms
from this day on.

------
knolax
The problem the author states in Vim is overstated. All you need is to write
two functions that switch the input method to English when entering normal
mode and back to your original input method when entering insert/replace mode.
It's only 24 lines of vimscript including whitespaces and comments.

Out of all the TUI programs out there, Vim with it's modal design is probably
the one program least affected by a conflation between keypresses and text
input.

~~~
big_chungus
This is not really cross-platform. This is also not cross-application, so it's
often necessary to do os-level, hence prior problems. At least for Spanish
there are a limited number of non-ascii chars, so I jus bind directly to a
modifier.

------
greggman2
a common issue in browsers are pages looking for the ESC key. I'm in some
modal dialog on a page. I'm typing CJK in the IME. I press ESC to exit the IME
or to cancel a conversion, the IME exits but the page gets the ESC key and
closes the model dialog. Any text I had entered to that point is lost.

------
bandrami
Personally I've always thought RTL switches in an _editor_ were a mistake. An
editor displays and lets you edit a character stream, and those characters do
not need to be in their final spatially rendered position for editing.

------
jen_h
Really appreciate this post. But I’ve gotta admit that I CTRL-Fed it for
ligatures in bidirectional text and Czech accents and found no result...and
couldn’t figure out whether I was happy or sad about that.

------
azhenley
This is one of the best things I have seen on HN in a long time! It really
makes me want to go work for a company with a text editor product or to teach
a course on implementing simple text editors.

------
6510
On c64 the text is mono-spaced and empty areas are filled with spaces. I
didn't appreciate the genius of that until now.

------
derefr
> Our carets will need an extra bit that tells them which line to tend
> towards. Most systems call this bit “affinity”.

Is it just me, or does the author disprove themselves with the figure in the
same section? I feel like the best possible solution is the one depicted: that
you just see the cursor split between both lines.

~~~
greenshackle2
It's not a split cursor, it's two cursors, one on each line.

~~~
derefr
I know, but two cursors _sort of_ looks like a split cursor, and you could
visually tweak cursor rendering to make “one cursor split between two lines” a
visually-distinct case from having actual multiple cursors (because some text
editors do indeed support multiple cursors.) What I’m saying is that I’d
prefer a text-edit control that gives you a visual indicator for “one cursor
split between two lines”, to one that pretends the cursor is on one line or
the other, when it really will act with the navigation semantics of being
split between two lines.

~~~
greenshackle2
Ah right, I misunderstood you. Yeah that sounds like a sensible solution.

------
billconan
these type of things seem not to be documented anywhere. If I want to start
from scratch, what can I use as a reference?

------
throwaway_n
I once tried to customize a rich text editor to support SVG text effects (like
[https://www.smashingmagazine.com/2015/05/why-the-svg-
filter-...](https://www.smashingmagazine.com/2015/05/why-the-svg-filter-is-
awesome/)) and damn near lost my mind.

If I had known that you can write an entire book on just text layout I
wouldn't have bothered: [https://www.oreilly.com/library/view/svg-text-
layout/9781491...](https://www.oreilly.com/library/view/svg-text-
layout/9781491933817/)

------
arpa
To be honest, I can't help but think that we're overcomplicating things for
ourselves. I mean: think about a typewriter, hell, even typesetting. And now
let's consider modern text editing. It's crazy! And yet, the only thing I
_need_ is something even simpler than vi: ability to move between the lines
when doing cat - > file.

That being said, i do appreciate the effort the good people of text editing
make.

~~~
saagarjha
Right, but you probably mostly deal with a documents written in a language
with a Latin script. Those who don't or can't do this need better.

~~~
eadmund
> Right, but you probably mostly deal with a documents written in a language
> with a Latin script. Those who don't or can't do this need better.

So, I absolutely _love_ different scripts, and I think it's awesome how many
different ways to write mankind has invented over the centuries.

But if Latin-style scripts are so much easier to input into computers, and if
computers are not just the wave of the future but the present as well, maybe
it would make sense for more languages to adopt Latin-style scripts. After
all, we stopped using fractions everywhere in favour of decimals, because
decimals are in many cases (not all!) better. Ditto our switch from Roman to
Arabic numerals.

~~~
munmaek
"Just drop your language's writing system in favor of the latin alphabet!"

"It's a hard problem (and doesn't affect me) so let's ignore it altogether!"

Must be easy to make remarks like this when your native language happens to
use the latin alphabet...

How about we use our knowledge to build computer systems that respect
different writing systems, even if it's hard, instead of enforcing a latin
hegemony upon the rest of the world.

Comparing scripts to numeral notation is comparing apples to oranges. The
latin syllabary does not work for languages that have more sounds than are
represented by 26 characters!

------
Mikhail_Edoshin
"Hate" is a strange choice of word to describe what is merely a complex
subject. Lots of different models (emoji + skin tone modifiers is not a bad
model in isolation, very much like a letter + combining accent) are developed
independently and then when we test them with the rest of the players on the
field, they don't integrate too well. Yes, they don't, but what would you
expect? :) And of course decisions made by different developers will vary in
logic, consistency, and completeness. This is normal state for the integration
part.

~~~
TeMPOraL
It's a phrase. You approach a seemingly simple problem that reveals itself to
be a fractal of complexity - at some point you start to feel as if the problem
itself had agency and it wanted to make you miserable.

------
poooood
This guy went to the Harvard of the Midwest. I feel it in my soul.

------
poooood
This guy seems p bright. Prolly went to Harvard (of the Midwest).

------
z3t4
I've naively implemented text rendering, selection, etc from scratch in a text
editor, and if you only support monospace, it's pretty simple. The hardest
part in software is to not implement features. A trick is to only implement
the complex features for the users that need it - in another VCS branch. Or
you make an abstraction layer. But abstraction layers are hard, and if they
are not air tight (non leaky) you are worse off, because then the developer
need to know both layers. Then there is the third strategy, which I would call
an anti-pattern, that when enough features have built up, in order to get rid
of the relics you rewrite it from scratch, preferably in a new framework. I
also believe most advancements in science are accidental, like penicillin, and
lexically scoped modules - so if we do not do things, like new frameworks, and
re-invent the wheel, the CS field wouldn't advance.

------
Aperocky
As almost an exclusive vim user (large scale java doesn't work very well on
vim), I find it funny when text editing is a huge pain - if there was pain for
me that's about a month of overall time sunk into getting really good at vim.
After the transition text editing has mostly been a joy.

~~~
wonnage
The article isn't about how to use a text editor

