
Picking Apart the Crashing iOS String - Manishearth
https://manishearth.github.io/blog/2018/02/15/picking-apart-the-crashing-ios-string/
======
jrochkind1
What I'm really curious about is how bugs/errors in the iOS typesetting
algorithm result in a crash, rather than just wrong or nonsense typesetting --
and how the last time this happened, they appearently just fixed the specific
case, but not the ability of bugs/errors in the typesetting algorithm to crash
the system.

I am _not_ surprised there will be errors/bugs in the typesetting algorithm,
as the OP demonstrates, this stuff is extraordinarily complicated to do for
every language/all of unicode. But if you have something so complicated that
bugs/errors aren't surprising -- you'd think you'd want to make sure they
didn't cause hard crashes.

~~~
pkaler
The crash seems to be in CoreText. CoreText is embedded/linked in Messages,
Spotlight, Springboard, etc. CoreText is written in C.

The fix would be to rewrite CoreText in a memory safe language like Swift.
This would be “hard”. Or put CoreText in an XPC container. This would both be
“hard” and result in terrible performance.

For more details on how hard C, memory management, systems programming, and
operating system development is please refer to your local copy of Modern
Operating Systems by Andy Tannenbaum.

~~~
mapmap
I'm guessing C is how they get the performance they need. Re-writing in Obj-C
or Swift would likely have speed tradeoffs.

~~~
pjmlp
C code only got fast thanks to 40 years of optimizer improvements, taking
advantage of UB.

~~~
zxxon
Huh? C is fast (compared to Swift) because using it doesn't imply sprinkling
lots of sugar (like ARC) into the resulting machine code.

Simpler languages like Fortran can turn into even faster code than a C
implementation. UB optimizations aren't that relevant for real-world
performance.

~~~
pjmlp
Code generated by C compilers is fast in 2018.

Code generated by C compilers for C64, Spectrum, Atari, Atari ST, Amiga, Mac,
CP/M, MS-DOS, Windows 3.x, Nintendo, MegaDrive,... systems meant many times
the code would be 80% like this:

    
    
        void some_func(/* params */) {
          asm {
             /* actual "C" code as inline Assembly */
          }
       }
      
    

Lots of Swift sugar also gets optimized away, and there is plenty of room to
improvement.

The code that current C compilers don't generate, many times is related to
taking advantage of UB.

They also generate extra code for handling stuff like floating point emulation
though.

Just as an example, IBM did their whole RISC research using PL/8, including an
OS and optimizing compiler using an architecture similar to what LLVM uses.

They only bothered with C, after making the business case that RISC would be a
good platform for UNIX workstations.

~~~
zxxon
Why bring these ancient home computer platforms into play? Those were totally
different to program for. Why not compare a C compiler from 1998 to one from
2018, on x86 (no SSE of course)? C compilers _have_ gotten better, but not
spectacularly.

>> The code that current C compilers don't generate, many times is related to
taking advantage of UB

Compilers are really smart in optimizing things that aren't relevant to the
real world.

For example, this code would reduce to "return 32" in most modern compilers:

    
    
      int return32(){
        int x=1;
        for (int i=0; i<5; i++){
          x*=2;
        }
        return x;
      }
    

Does that make impact in real-world code? Almost certainly not, it's a
contrived case. Most UB cases fall into the same category.

>> They also generate extra code for handling stuff like floating point
emulation though.

Not necessarily.

~~~
pjmlp
> wWhy bring these ancient home computer platforms into play? Those were
> totally different to program for. Why not compare a C compiler from 1998 to
> one from 2018, on x86 (no SSE of course)? C compilers have gotten better,
> but not spectacularly.

To clear up the myth among young generations that C compilers always generated
fast code, regardless of the platform.

As for something more modern, in 1998, C code quality was still at a similar
level to other system's languages, before they started to fade away thanks to
the increase in UNIX, Linux and BSD adoption

For example, given that Delphi and C++ Builder share the same backend, their
generated code was quite similar, even if it would require disabling some of
the Delphi's security checks.

> Not necessarily.

Sure, it all depends on the CPU being targeted.

------
Diggsey
My guess would be that it's some aspect of _measuring_ the text that is
causing the crash: when you click in an editable text box, there is code to
track down where the cursor should be placed. This is done by measuring
various sub-strings of the whole line.

If measuring the sub-strings gives surprising results (sub-strings being
visibly longer for example), this could cause the algorithm to fail in any
number of interesting ways: for example if a binary search is used to locate
the cursor position, it could break the invariants of the binary search.

~~~
Manishearth
Well, the crash occurs for Spotlight without me clicking anything or having
any cursors anywhere.

But yeah, this is one of my theories about it. One of the previous crashes had
to do with an Arabic string which got _longer_ when you truncated it, which
made snipping it to display in a notification have bugs.

It's interesting to see it's causing a _segfault_ , i'd expect measuring bugs
to cause clean assertions or shitty rendering. Which is why I'm also wondering
if it's actually a disagreement on the number of "characters" in the rendered
things.

> If measuring the sub-strings gives surprising results (sub-strings being
> visibly longer for example), this could cause the algorithm to fail in any
> number of interesting ways: for example if a binary search is used to locate
> the cursor position, it could break the invariants of the binary search.

Cursor positions are based off of grapheme clusters -- there's a defined
algorithm for that. Though different parts of the system may disagree on the
specifics of the algorithm causing such a crash.

However, that doesn't gel with the fact that it's only specific consonants
causing this, all versions of UAX 29 do not consider any differences between
Indic consonants for a single given script.

~~~
rlanday
UAX 29 doesn't really describe how to handle Indic text very well.

[http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries](http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries)

> Grapheme clusters can be tailored to meet further requirements. Such
> tailoring is permitted, but the possible rules are outside of the scope of
> this document. One example of such a tailoring would be for the aksaras, or
> orthographic syllables, used in many Indic scripts. Aksaras usually consist
> of a consonant, sometimes with an inherent vowel and sometimes followed by
> an explicit, dependent vowel whose rendering may end up on any side of the
> consonant letter base. Extended grapheme clusters include such simple
> combinations.

> However, aksaras may also include one or more additional prefixed
> consonants, typically with a virama (halant) character between each pair of
> consonants in the sequence. Such consonant cluster aksaras are not
> incorporated into the default rules for extended grapheme clusters, in part
> because not all such sequences are considered to be single “characters” by
> users. Indic scripts vary considerably in how they handle the rendering of
> such aksaras—in some cases stacking them up into combined forms known as
> consonant conjuncts, and in other cases stringing them out horizontally,
> with visible renditions of the halant on each consonant in the sequence.
> There is even greater variability in how the typical liquid consonants (or
> “medials”), ya, ra, la, and wa, are handled for display in combinations in
> aksaras. So tailorings for aksaras may need to be script-, language-, font-,
> or context-specific to be useful.

For example, in Chrome, we added an extra rule to not allow grapheme clusters
to be split after Indic virama characters, but later had to modify the rule to
not apply to Tamil viramas:

[https://chromium-
review.googlesource.com/c/chromium/src/+/84...](https://chromium-
review.googlesource.com/c/chromium/src/+/843461)

I don't know the exact cause of this crash, but I can see why Apple might be
running into trouble with their logic for these languages. I suspect their
algorithm for computing grapheme clusters has a bug causing an inconsistency
somewhere.

~~~
Manishearth
Yeah, I'm aware, I've been arguing for UAX 29 to handle consonant clusters for
a while. The current draft has handling for it:
[http://www.unicode.org/reports/tr29/tr29-32.html#Virama](http://www.unicode.org/reports/tr29/tr29-32.html#Virama)

However, given that some Brahmic scripts prefer explicit viramas (Malyalam,
also Thai I think), this will probably be restricted to Brahmic scripts where
joining is always preferred (even if not possible).

I'd been testing UAX 29 stuff out before and Apple seems to follow the spec.
For example, Chrome and Firefox seem to do special handling for e.g. flag
emoji (distinguishing between regional indicator pairs that render as a flag
vs those which don't -- i.e the ones which don't correspond to a country
code). But Apple follows the spec rigidly. In particular it does not consider
joined consonants to form a single EGC.

I could be wrong on that, though.

~~~
jcheng
Nit: it's "Malayalam". I noticed the same typo in your blog post.

~~~
Manishearth
lol I keep making this mistake. Thanks.

------
torstenvl
I cannot trigger this crash on iOS 10.3.2, despite repeated attempts. I can
reliably crash friends' phones by iMessage for all friends on iOS 11.

That suggests to me that Apple made changes to CoreText, and _did not perform
adequate regression testing._

All software has bugs. I understand that. But I suspect a large part of power
users' and developers' growing frustration with Apple is that they keep
introducing severe, kernel-panicking, root-exposing bugs in software that
previously did not exhibit the problematic behavior.

Honestly, how do you _not_ have a stringent regression testing requirement for
changes to the "Core" of the operating system?

~~~
arvinsim
> Honestly, how do you not have a stringent regression testing requirement for
> changes to the "Core" of the operating system?

By not managing expectations and deadlines properly. Especially if schedules
are overriden from a manager who is not aware of the ramifications of changing
something in the core.

~~~
zatkin
And this is about to change, according to some recent news:
[https://www.bloomberg.com/news/articles/2018-02-12/how-
apple...](https://www.bloomberg.com/news/articles/2018-02-12/how-apple-plans-
to-root-out-bugs-revamp-iphone-software)

------
donarb
Very interesting article. It gives a bit more perspective to the problem than
the simplistic view that iOS has a problem displaying a certain character
onscreen.

This particular bug shows the problem with multibyte characters (terminology
may be wrong, perhaps multi glyph is better) where certain parts of the
character become left or right associative based on context.

~~~
Manishearth
You want "multi-codepoint graphemes" probably. Or really "things involving
combining characters". There isn't really a term for this.

~~~
bitwize
I believe the Unicode term is "grapheme cluster".

~~~
Manishearth
Not exactly, no.

क्र is currently not a grapheme cluster, for example (there is a proposed
change to UAX 29 to make it such, but that doesn't handle all such cases).

The strings in this situation are three grapheme clusters according to the
spec even though they usually render as a single "thingy".

------
kens
Not directly related to the crash, but I have a question about Telugu and
similar scripts: How do their speakers think about the structure of the
script? Do they consider each vowel and consonant a separate "thing" that just
happens to get written as a complex grapheme, or is the grapheme the unit you
think about and it just happens to be made up of smaller parts? I.e
[https://en.wikipedia.org/wiki/Telugu_script#Consonant_Conjun...](https://en.wikipedia.org/wiki/Telugu_script#Consonant_Conjuncts)

Also, do you learn the large number of graphemes separately? Or are they the
"obvious" way to write the consonant and vowel? Or is there a set of rules you
learn?

~~~
Manishearth
I can answer your question for Devanagari:

\- You normally think of clusters as single "letters". The word for them in
Marathi is "joined letter". This notion of "letter" may not be totally in line
with what English speakers would expect.

\- You still think of clusters as having component parts that make it up.
However, क्ष/ज्ञ and sometimes त्र are thought of as their own "fundamental"
consonants even though they're clusters. I think folks think of things like
क्र as being both -- a letter of its own, that is built up from component
letters.

\- Most of the clusters in Devanagari are predictable. So for any given pair
you usually know how to make a "half form" for the first one and join it on,
or you know how to stack them. Usually whilst reading it's obvious because you
see the components; and whilst writing it's usually fine to try what makes
sense.

\- For unpredictable clusters, you just eventually get used to them. No
different from learning that some words are spelled weirdly in English,
really. Though I very much dislike ह्म , ह्य, and द्य since they're kinda
confusing.

I think Bengali has a lot more unpredictable conjuncts; I stumbled across this
a lot when I tried to learn the script a year ago. I suspect folks just get
used to those.

~~~
Someone1234
As someone who can read both Devanagari and English/Latin script, I'm just
curious, when you read Devanagari does the text size need to be larger for
good comprehension?

There seems more subtlety with the word structure, like tiny little flicks
which seem to have significant meaning. While Latin-like script has characters
like the comma/full-stop, and modifiers like the dot on the i, these are far
rarer than Devanagari's modifiers from what I have seen.

Another related question is, because Devanagari's information density seems
higher, does that mean shorter text for the equivalent information?

PS - Sorry if anything I said was offensive, that wasn't my intent. I am just
an ignorant idiot wanting to learn someone's experience with a different
language script, nothing more.

~~~
Manishearth
Not really, I can read at normal text sizes. But it is somewhat annoying (much
like how tiny English can be readable but annoying to read) and used to have
my default font size bumped up by one in my browser. I currently have it
bumped up quite a bit for Chinese because I _really_ have trouble with the
"tiny little flicks" problem in Chinese.

The consonants of Devanagari are easy to tell apart; they are pretty
different. In most cases the "similar" ones are basically with an extra line
(प/ष, ब/व) or with a little loop (य/थ, ट/ढ) . (The line/loop has no semantic
meaning, these letters are just random letters that look similar) . The only
super annoying one is घ vs ध -- the thing up top is a loop in the second one,
but depending on the font/handwriting can be rather unclear.

The vowels are also easy to tell apart. There are 12 main ones, and they're
made up from some really basic components which are quite distinguishable.

So you can easily distinguish consonant+vowel combos.

Consonant clusters can get tricky, like I said there are a bunch of ambiguous
ones. Fortunately you end up realizing which is which and then it's nbd. But
like, द्म and ह्म can be infuriatingly similar and you just get it from
context.

Bear in mind, at one stage you start sight-reading words, so the actual
details of the word matter less.

> Another related question is, because Devanagari's information density seems
> higher, does that mean shorter text for the equivalent information?

Visually? Yeah. Words can be really small. But it ends up roughly being the
same number of code points (and probably more bytes in UTF-8).

~~~
imron
> I currently have it bumped up quite a bit for Chinese

It really annoys me when websites/apps keep the same font size for Chinese
that they use for English. 8pt Chinese is awful to read.

~~~
Manishearth
If you use Firefox or Chrome you can change default font sizes in preferences
on a per-language basis. You can also change the default font used. It's
pretty useful, lets me replace shitty system fonts with better ones and also
bump up the size for Chinese.

It does not apply to text using absolute font-sizes however, just text that
uses things like font-size:medium (or inherits the document default font size,
which is also medium)

------
0x0
Fonts are hard. I don't think CoreText is worse than others, most vendors seem
to have a history of horrors, and at least it's not rendering fonts in a
kernel driver like Windows did(!?) - see for example the Project Zero series
of "one font vulnerability to rule them all" \-
[https://googleprojectzero.blogspot.com/2015/07/one-font-
vuln...](https://googleprojectzero.blogspot.com/2015/07/one-font-
vulnerability-to-rule-them-all.html)

------
kibwen
Has anyone yet traced this to a specific syscall? Seems like the perfect
opportunity for whipping up a fuzzer to spit arbitrary unicode into the system
and see what else crashes.

~~~
enzanki_ars
Would that even be possible on iOS for non Apple employees?

My idea would be just to create a program that continually creates Unicode
strings and display them on screen with a pause before creating and displaying
the next. Record the screen and see if any thing exciting happens. Would never
end, if there was a SETI @ Home style program for it with insentives for
running it would be cool.

~~~
thephyber
> Would that even be possible on iOS for non Apple employees?

iOS emulator (part of XCode) might be enough.

~~~
enzanki_ars
Ah. Forgot that existed for some reason. (Windows/Linux user that gets screwed
out of programming for my own phone. Hate that....)

~~~
gsich
Use Hackintosh or a VM.

~~~
enzanki_ars
Given that Apple does not design their OS to work on anything but their
hardware, the pain I would incur trying to set up either option is just not
worth it. Unless someone has published a stable and updated pre-built VM for
macOS, it just seems like nothing but trouble.

~~~
userbinator
Not so long ago I was pleasantly surprised by how easy it was to setup a
Hackintosh VM in VirtualBox --- all I did was create a VM with the default
settings, add the all-important SMCDeviceKey, insert a completely stock El
Capitan ISO, and it booted up and installed on the first try. All on hardware
that was as non-Apple as it gets (a ThinkPad.)

The only thing that I found somewhat confusing and perhaps a bit un-Apple was
the fact that the installer would not prompt me to partition and format the
HDD first, but was perfectly happy to let me try to select and then fail to
install onto the install media itself (with an odd "not enough space"
message); I had to use the Disk Utility to do that _before_ going back to the
installer. I've always wondered why --- even Windows' installer includes
partitioning and formatting as one of its steps.

~~~
charrondev
Like because there isn’t a single current official device with a user
replaceable hard drive.

------
ahstilde
People interested in this bug may also be interested in "An Introduction to
Writing Systems & Unicode." Part 3[1] in particular discusses complex scripts.

[1]
[https://r12a.github.io/scripts/tutorial/part3](https://r12a.github.io/scripts/tutorial/part3)

------
bawana
I went to the Apple Store in Burlington, MA yesterday (2/16/2018) None of the
'geniuses' there knew about this bug or how to fix it. They reminded me of
microsoft drones following a troubleshooting script. 1) Reboot the phone 2)
reset the phone 3) do a dfu restore..

After they did this my phone would not recognize its sim card. You see my
phone originally had AT&T service. Then I switched to sprint some months back.
I guess the dfu restore ended up relocking my phone to AT&T. At least that's
what the Apple geniuses said. Then they said I had to go to a sprint store.

This is the first time I was disappointed with Apple's human help. I remember
past experiences going in there with a frayed power supply cord - they gave me
a new one. On other occasions they replaced my phone when they couldn't fix
it. They never spent more than 30 mins trying to fix a problem and always
fixed it. Their knowledge was deep. This time their opinions were
confabulatory - for example one guy said 'your phone does not have the
antennae to work with sprint' EVEN THOUGH I reminded him I walked into the
store with a working iPhone using sprint. Another one said that sprint
jailbroke my phone and that's how they got it to work. WRONG. Another one said
that the phone was 'soft unlocked' and relocked itself after the DFU restore.
I felt sad for Apple because I know that's not what they were about.

Then I suggested they replace my phone, the manager lady's eyes bugged out,
her voice became stern and said ' There is no scenario where you walk out of
here with a different phone'. This is the first time they wasted 5 hours of my
time and actually caused me to have more trouble than what I walked in with.

Galaxy here I come. or maybe pixel2

------
brian-armstrong
The broader question is, I think, how can any textual sequence cause a crash.
Text should be regarded very skeptically by the text renderer. What could this
be doing that it invokes a crash? Where else is this team not being careful?
It reeks of process problems.

~~~
spaceywilly
My question was, why was there not a unit test for this? It seems like it
would be trivial to step through every character combination for each language
they support and make sure it doesn't cause a crash.

~~~
nikanj
Unicode allows for 17 planes, each of 65,536 possible characters (or 'code
points'). This gives a total of 1,114,112. This crashing sequence is five
characters long.

Your unit tests would have to go through 1.71650179e30 sequences to be
guaranteed to catch this one. At a test rate of 1 millisecond per sequence,
that's just 4×10^9 × the age of the universe, according to wolfram alpha.

~~~
zokier
> Unicode allows for 17 planes, each of 65,536 possible characters (or 'code
> points'). This gives a total of 1,114,112.

Allows for 17 planes, but only small portion of those are actually used.
According to Wikipedia[1], currently Unicode has 148944 codepoints + 128k
private use ones (which might, or might not make sense to include in unit
tests). So your time estimate is off by mere 5 orders of magnitude.

~~~
nikanj
(148944 ^ 5) milliseconds to years = 2.324 quadrillion years .

Doesn't really change the result, imho.

------
calvinbhai
@Manishearth

Thats a great explanation (though quite a bit of it was too much for me to
understand).

The only correlation I can think of, is that the Kannada script keyboard and
full support for the script rendering in the UI didn't exist in iOS until iOS
11.0. So it is crashing for Indic scripts that were supported pre iOS 11.
Probably Apple tweaked something in iOS 11 for the languages already supported
and broke something.

~~~
Manishearth
Huh, that's an interesting theory.

------
ollin
Does anyone have a link to a thorough reverse-engineering of the bug from
someone decompiling/source-diving?

I tried replicating it on High Sierra out of curiosity (single text file with
the broken character, opened in TextEdit to cause a crash) and noticed that
the crash doesn't happen in consistently the same way each time (I got 8
crashes in a calloc call in CoreText's OTL::CoverageBitmap::Reset, 4 crashes
in a malloc in Foundation with no CoreText in the stack trace, plus 10
KERN_INVALID_ADDRESS errors from various places, as well as one time it opened
successfully with no immediate problems). Seems like a memory corruption error
somewhere earlier, but I'm not sure how to figure out where–I tried rooting
around in CoreText with Hopper for a bit but didn't get very far.

~~~
Manishearth
[http://www.cloudscan.me/2018/02/coretext-crash-
osx-10133-lld...](http://www.cloudscan.me/2018/02/coretext-crash-
osx-10133-lldb-voltron.html)

------
svat
(I should probably try this myself, but…) The equivalent programs on other
systems are HarfBuzz (open-source) and Uniscribe / DirectWrite (Windows). Did
you consider checking what they do with the equivalent text (presumably they
don't crash)?

~~~
mappu
No crashes on Debian harfbuzz.

------
foota
Funny, this is the same person that wrote the new str find functionality in
rust 1.24

~~~
Manishearth
Turns out I like doing unicodey things :)

------
komali2
Not sure why I'm not able to trigger this crash on any Apple devices in the
office - saw on twitter about how it was messing with iphones even if it was
an SSID, unable to replicate here though.

------
chris_wot
Would fuzzing help with this?

------
gatkinso
shoulda used JS

------
blackflame7000
Does anyone else think that this could be related to Auto-Correct having
problems with the malformed expressions?

~~~
Manishearth
This happens whenever the system is asked to render text; it doesn't have to
be editable.

(it's just easier to trigger in text boxes because you paste and that's it)

~~~
blackflame7000
Oh, gotcha. Yea not sure auto-correct could cause that. I initially read it
was only related to text-entry and thought auto-correct was likely since it
was affecting multiple apps indicating an OS issue.

