
iPhone update for Meltdown-Spectre: before/after performance benchmarks - justdutch
https://melv1n.com/iphone-performance-benchmarks-after-spectre-meltdown-update/
======
eridius
This article doesn't make any sense. The author is doing generic phone
benchmarks. But the Spectre fix is a _webkit fix only_ , not an OS-wide fix.
If they're seeing performance regressions across the whole OS because of
fixing Spectre, something's seriously wrong with their benchmark methodology.

Edit: The author upgraded from iOS 11.1.2 to iOS 11.2.2. This isn't just a
test of the Spectre fix. The most likely explanation here is upgrading to iOS
11.2 caused their iPhone 6 to start throttling due to battery wear (11.2 added
throttling to iPhone 7, and it's plausible that it changed the conditions for
throttling on iPhone 6). It's also possible that this is instead caused by the
Meltdown patch, but these numbers are still way out of line with what was
expected for Meltdown on iOS, whereas they're very much in line with what
we've been seeing with battery throttling.

~~~
runeks
I just did a 11.2.1/11.2.2 benchmark of my iPhone 7:
[https://browser.geekbench.com/v4/cpu/compare/6303880?baselin...](https://browser.geekbench.com/v4/cpu/compare/6303880?baseline=6303560)

The result is that 11.2.2 is slightly faster than 11.2.1 (around 2%).

And here’s one of my iPad Pro 10.5”:
[https://browser.geekbench.com/v4/cpu/compare/6304146?baselin...](https://browser.geekbench.com/v4/cpu/compare/6304146?baseline=6303715)
(0.9% increase in performance).

~~~
jlmn
I did this same test with an iPhone X and iPad Pro 10.5" as well as a several
year old Mini 2 and didn't see a difference. In fact, I saw what you did, my
scores got better in most categories.

------
bluedino
>> a significant decrease in performance on the iPhone 6 up to 50%

Something's up. I updated my 6S yesterday and have noticed zero in performance
changes or battery loss. Still plenty of bugs though, I was reading an email
and the 'flag/file/trash/reply/new' bar totally went away. At least the
touchscreen hasn't gone unresponsive, causing me to have to hit sleep/wake to
toggle it back on. Maybe they finally fixed that.

~~~
ponchoalv
same behaivor here... no perfomance / battery hit.

~~~
oatmealsnap
6 or 6s?

~~~
delish
I have a 6S+. I noticed a performance gain going from 11.2.1 to 11.2.2.

Before: 2384.

After: 2541.

------
viraptor
Ok, I'm sceptical about the results. The reason is that there doesn't seem to
be a massive difference between the tests. Since this fix is about speculative
exec, why would it affect crypto code which is very register based and
branchless as much as sqlite which is full of branches and memory/storage
based? Why would it affect AES which is hardware accelerated as much as
integer processing which is not?

I'm not saying this is impossible - maybe there's something that I'm missing.
But it just doesn't add up at the moment. I'd love a more detailed /
repeatable test.

~~~
scott_s
I can't speak about these benchmarks (the site is down). But Meltdown is _not_
about speculative execution, it's about out-of-order execution. The fixes for
Meltdown involve getting kernel data out of user virtual memory, and flushing
the TLB if the processor does not have the pcid instruction. So context
switches - including system calls - will tend to be more expensive, and if the
processor does not implement pcid, _way_ more expensive.

See [https://meltdownattack.com/](https://meltdownattack.com/). I haven't yet
read the Spectre paper, but I can say that the Meltdown paper is very readable
and walks readers through everything. I anticipate the Spectre paper is as
well.

~~~
jokr004
This is about Spectre, not Meltdown, which _is_ about speculative execution.
From the article:

> Apple released iOS 11.2.2 update to address Spectre security issues.

I'm not sure why Meltdown was added to the HN title...

------
FBISurveillance
I've went through Denial, Anger, Bargaining, and now in an Acceptance stage
after taking up to 31% performance hit on some of services managed by my team.
Worst case has been Elasticsearch so far with our load pattern, taking that
31% hit.

Oh well, too bad, enjoying the ride.

~~~
euroclydon
What about using dedicated hardware and not running the patches? At attacker
has to run malicious code on your server to exploit the vulns anyhow, right?

~~~
FBISurveillance
I usually don't trust people. More than one have we found things improperly
secured and/or exposed to the outer internet, or otherwise at risk. We also
run black-box software from third party vendors that provide compiled JAR
files which is directly related to this fuss. Last thing I'd like to have is
another attack vector at home.

------
enstyled
I made similar benchmarks recently on my iPhone 6S, running iOS iOS 11.1.2 vs
11.2.1, before and after replacing the battery:

[https://twitter.com/enstyled/status/948892447765131264](https://twitter.com/enstyled/status/948892447765131264)

TLDR: It's 11.2.1 that is throttling the older iPhones, because of the battery
wear.

------
schneidmaster
Sample size of 1, but I Geekbench'd my iPhone X before and after upgrade.

11.2.1: Single-Core 4137, Multi-Core 9315

11.2.2: Single-Core 4039, Multi-Core 9876

Anecdotally as well, I haven't seen a noticeable difference in performance. So
your mileage may vary substantially based on what device you have.

------
ucha
I just ran GeekBench 4 on my iPhone X on iOS 11.2.2 vs 11.2.1.

Single core: 4239 vs 4241

Multi-core: 10081 to 10203

So no difference.

~~~
TheSithMaster
The test done here was comparing 11.2.2 vs 11.1.2

~~~
dogma1138
Isn’t 11.2.1 the battery patch?

~~~
TheSithMaster
You are correct, it appears to be that "patch".

Source:
[https://discussions.apple.com/thread/8211392](https://discussions.apple.com/thread/8211392)

~~~
ucha
So basically, the difference in performance is likely almost entirely due to
the battery patch and has nothing to do the Meltdown-Spectre fix.

------
qrohlf
I'm curious what the delta is between perf hit on an iPhone 6 and an iPhone 8.
I'm guessing that, given the deadline apple had to get this out, most of the
attention & optimization went to current devices versus near-EOL devices like
iPhone 6.

------
sschueller
Is the Samsung S8 CPU also affected? I saw that some ARMs are not affected.

EDIT: It appears not the be on the list of affected ARMs [1]. The S8 Exynos 9
Octa 8895 is based on the Cortex-A53

[1] [https://developer.arm.com/support/security-
update](https://developer.arm.com/support/security-update)

EDIT2: The Snapdragon 820 and 835 do appear to be affected which are the CPU
in the US version of the S8

~~~
JoachimSchipper
Assume that [EDIT: high-end - dogma1138 is right] smartphone-class processors
and above are affected. Raspberry Pi and microcontroller-class ARMs are fine,
though (but much slower!)

~~~
dogma1138
Most of them aren't affected mid range ARM and below aren't vulnerable,
basically if you have a flagship product you should worry.

If we take the snapdragon example the 6XX's aren't affected while the 8XX are.

------
bitmapbrother
Google cache:

[http://webcache.googleusercontent.com/search?ei=jXNWWojdC4vc...](http://webcache.googleusercontent.com/search?ei=jXNWWojdC4vcjwSj2ozgCg&q=cache%3Ahttps%3A%2F%2Fmelv1n.com%2Fiphone-
performance-benchmarks-after-spectre-meltdown-
update%2F&oq=cache%3Ahttps%3A%2F%2Fmelv1n.com%2Fiphone-performance-benchmarks-
after-spectre-meltdown-update)

------
tn_
This might sound dumb but.. with these Meltdown + Spectre bugs would an
attacker be able to penetrate to gain full access only if a user downloads a
native app or would it be possible through the browser/js?

~~~
neilsimp1
This post, which I saw on HN yesterday, may answer some of that for you:

[https://webkit.org/blog/8048/what-spectre-and-meltdown-
mean-...](https://webkit.org/blog/8048/what-spectre-and-meltdown-mean-for-
webkit/)

Short answer: yes.

------
Negative1
Haven't seen much coverage on how this affects game consoles like the Xbox and
Playstation (which use AMD CPUs). Does anyone know if they've talked about
patching it and how this would affect game performance (which is a pretty big
deal for gamers who expect a consistent experience)?

~~~
Richs99
Mike Ybarra has said there's no need for them to patch the Xbox One against
these attacks:

[https://www.trueachievements.com/n31065/xbox-one-consoles-
un...](https://www.trueachievements.com/n31065/xbox-one-consoles-unaffected-
by-spectre-and-meltdown-chip-issues)

------
AuroraBob
I will repeat these benchmarks on my iPhone 6 when I get home tonight...but in
the meantime, the benchmark numbers in this article look almost exactly like
the ones I had as a result of battery throttling. I bet the actual patch-
related performance hit is pretty minor.

------
mrmondo
I’ve more than a little skeptical of these results, I too am very disappointed
in major CPU vendors (especially their PR and management teams), but these
results seem more than a little smelly.

I am what I’d consider a heavy iOS user on multiple devices and I don’t
believe I’ve honestly noticed any difference at all. If these results were
correct, I believe I’d notice at least a 10-15% decrease in performance - but
no. What I haven’t looked at is battery life so I cannot comment on that.

------
euph0ria
I wonder how much extra revenue this will bring to AWS/GCP etc selling more
instances to cover the performance loss on servers?

~~~
perfmode
It's energy providers that will win the most.

~~~
toomuchtodo
All major cloud providers are buying renewables for their operations. Yay?

At least the additional renewable generation capacity built out will exist for
decades (while processors should become more efficient as these security
issues are addressed and new hardware rolls out).

~~~
hueving
>All major cloud providers are buying renewables for their operations. Yay?

Not yay. This is an overnight massive bump so they are going to consume from
the same existing pool of renewables, forcing utilities to use more non
renewables to make up for the demand difference.

Also, even when they do build out the renewable generation to make up the
difference, it's still environmentally a problem due to the manufacturing of
the renewable generation (solar panels, windmills, transformers, etc).

We have no surplus of renewable energy to absorb this kinda stuff so more
energy usage is always bad at this point. It's just slightly less bad when
they commit to paying to fund renewable.

~~~
codazoda
I've read a few articles saying that there is so much Solar adoption in the
U.S. that there is a surplus at some times (summer days I believe). It use to
be that there was too much use from home AC systems so power was expensive
during the day. It seems to be switching to night now as so much is generated
during the day (with just 3% adoption of rooftop Solar).

Here's an example of one article on the matter, which I just skimmed a little.

[http://www.latimes.com/projects/la-fi-electricity-
solar/](http://www.latimes.com/projects/la-fi-electricity-solar/)

I'm guessing that CPU's use a whole lot less electricity than things like AC
systems, so they probably don't affect total electricity used by that much. I
was able to surmise that a 3 ton central air conditioner uses around 3500
watts per hour while an Intel I7 (whole system) uses around 150 watts per
hour.

My theory, then, is that this won't affect total power use in the U.S. all
that much.

------
sWallo
tested with GeekBench 4 on iPhone 8 Plus if anyone interested: \- iOS 11.2.1:
Single-Core 4257, Multi-Core: 10187 \- iOS 11.2.2: Single-Core 4259, Multi-
Core: 10287

Looks like no performance impact at all.

------
paws
My iPhone X has been on 11.2.5 since the day it came out, and I haven't
perceived any slowdowns.

Have not done any benchmarking, and I'm not saying performance is the same as
before, but anecdotally I haven't seen a difference. Just one guy's opinion.

~~~
jackson1way
the latest iOS version as of today is 11.2.2

I don't think 11.2.5 exists

~~~
taspeotis
It has existed in beta for a while.

[https://www.macrumors.com/2018/01/09/apple-seeds-
ios-11-2-5-...](https://www.macrumors.com/2018/01/09/apple-seeds-
ios-11-2-5-beta-4-to-developers/)

------
apohn
I just updated my 6+ and ran Geekbench before and after. No major difference
in scores and the single and multi-core benchmarks are consistent with the
comparison numbers for other 6+ phones. Battery is in good health.

------
2bitencryption
this leads me to a couple questions, hopefully you smart folk can answer:

1\. I thought Spectre was "Intel-only", and Meltdown was the general case,
which is less severe but effectively nearly everywhere? If so, how is an
iPhone susceptible to Spectre?

2\. Beyond that, I thought meltdown/spectre was an x86 problem. So why all
this trouble on phones, with ARM?

3\. I've read the first, simplest variant of meltdown, and it is so
beautifully simple. Is this "speculative execution + cache timing" thing an
entirely novel exploit, or have we seen incarnations of this before?

~~~
JoachimSchipper
1: Spectre is the general case, Meltdown is the Intel-only evil cousin.

2: Spectre applies to all modern processors with speculative execution, which
includes smartphone-class ARM processors. Raspberry Pi's and microprocessors
("toasters") are not affected.

3: There's some prior work - Spectre didn't fall from thin air - but using
speculative execution as the basis of an exploit makes Spectre the first bugs
in, I believe, a new class. (Meltdown, on the other hand, is just a silly
mistake that shouldn't be repeated.)

------
cdolan
Can we get a before/after with a new battery vs a battery with 1k, then 10k,
cycles on it?

------
moepstar
I benchmarked my 8 Plus yesterday, updated from 11.2.1 to 11.2.2.

No differences noticeable under same circumstances, maybe even slightly faster
(Compute benchmark, Geekbench 4).

------
bitmapbrother
Something isn't right with his benchmarks.

My iPhone 6 benchmarks:

 _Geekbench test taken on Dec 18, 2017 with iOS 11.2_

Single Core: 1566

Multi-Core: 2697

 _Geekbench test taken on Jan 10, 2018 with iOS 11.2.2_

Single Core: 1551

Multi-Core: 2675

 _Percentage slowdown:_

Single Core: 0.96%

Multi-Core: 0.82%

~~~
jasonlotito
Do you have Geekbench test with iOS 11.1.2 instead of 11.2?

~~~
mkagenius
Shouldn’t be needed if you go by absolute numbers - OP had 9xx instead 15xx ..
so, clearly something is not right

------
dotdi
Can't wait to see before/after benchmarks for Android. Something tells me it's
going to be even worse than iOS.

~~~
happycube
Depends on the phone. Lower end phones use cores like the A53 which don't have
enough speculation HW to be affected at all.

------
gok
This is...highly questionable. 11.2.2 only updates WebKit, which shouldn't
affect GeekBench.

------
fancyfacebook
I keep getting bad gateway from some awful CDN, is there an alternative link?

~~~
marcc
That's Cloudflare and it's not sending the bad gateway, it's telling you that
the upstream (the OP server) is returning a bad gateway.

~~~
jasonlotito
Except Cloudflare is saying this: "However, because the site uses Cloudflare's
Always Online™ technology you can continue to surf a snapshot of the site"

It shouldn't be telling you that the host is failing.

------
dijit
Aaand it's down. Anybody have a mirror?

~~~
cmg
archive.org does:
[http://web.archive.org/web/20180110193140/https://melv1n.com...](http://web.archive.org/web/20180110193140/https://melv1n.com/iphone-
performance-benchmarks-after-spectre-update/)

------
rconti
Why does Cloudflare tell me I'm able to browse a snapshot while the site is
offline, but there is no snapshot?

~~~
Pfhortune
This is because, humorously, the snapshot is actually of the 502 page.
Apparently Cloudflare kicked in too late.

~~~
chinhodado
Is that how is's supposed to work? Because any time this happened in the past
to a site where it is overcrowded and the Cloudflare banner is showing, the
502 page is showing as well, so I thought it's part of the Cloudflare error
page?

------
brador
Is this a "bug" we can recover from (in future hardware or software) or are we
going to collectively take a lifetime -18 hit to our Moore's Law progress and
lose the gains from this technique forever?

~~~
perlgeek
At least for Meltdown there are CPU microcode updates out there that should
mitigate it, which means the OS level mitigations can be rolled back. And I
haven't measured it yet, but I suspect the microcode mitigates are faster.

Regarding Spectre, I think it's too early to tell.

------
StringyBob
Wow. After ios 10.2.1 halved performance on my iphone 6 due to the battery
issues, I next avoided 11.0 as I accepted responsiveness would be slower on
the older iphone6 model.

Now if we want these security fixes that's another -40%... Yikes!

However, it looks like ios 11.2.2 and a new battery is still slightly faster
than ios 10.3.3 and an old battery! (geekbench single/multi core score from
article of 924/1616 vs my 844/1379)

~~~
intopieces
I don't understand this logic at all. The fixes are being made for a reason...
avoiding the update just means you are getting more performance for a less
secure system. Are you betting on a law of averages to break in your favor?
Hoping you're not a target of hackers? what am I missing with this strategy?

~~~
qeternity
All of these issues require local execution. It’s perfectly fine for many
single user, performance critical use cases to make this type of security
trade off.

~~~
mkagenius
> local execution

Delivered via JavaScript

~~~
euroclydon
So don't browse the web from your server. Why are we updating servers?

~~~
pdpi
It wouldn't be the first time that a buffer overflow somewhere led to
arbitrary code execution. What we're patching is a mechanism that allows
escalating from unprivileged execution to reading kernel memory, and a
mechanism that defeats KASLR

