
The Sad State of Logging Bugs for Apple - fireflies_
https://www.corbinstreehouse.com/blog/2019/03/the-sad-state-of-logging-bugs-for-apple/
======
ridiculous_fish
Disclaimer, I worked with Corbin for a decade, and have tons of respect for
him. I agree with the diagnosis here, and I have one additional treatment
suggestion:

The single easiest and most effective thing Apple could do to improve its SWE
org is to invest in Radar.

Radar's importance within Apple cannot be overstated. It subsumes what would
be multiple tools in other orgs. As an Apple SWE you spend a massive amount of
time in it. And yet Apple treats Radar as a cost center, developed by an
outsourced offshore team. It's slow to search, supports only plain text, is
hard to script, and is missing obvious features, e.g. automatic duplicate
finding.

Hire five good SWEs, give them a mandate to serve the needs of the org, and
you will massively increase the effectiveness of every other engineer.

(My experience is now rather stale and I hope they've already formed such a
team!)

~~~
saagarjha
> developed by an outsourced offshore team

I thought there were some engineers who worked on Radar, but then I heard that
they had moved development offshore. Is this a new thing?

> is missing obvious features, e.g. automatic duplicate finding

Last I heard, I think someone was working on this in their free time. I don't
think anyone has tried to add this as an official feature, though.

------
ThemalSpan
Anecdotally, I didn't find the situation internally to be much better. Many
bugs internally go unanswered because there is not enough time allocated to
fixing core systems and designing better replacements. The truth is, I know
personally of several teams that aren't able to get through the queue of
internally filed and scheduled bugs.

To me, it feels like Apple hasn't resourced core pieces of infrastructure and
engineering teams in line with upper management's plans for growth. While many
teams are relatively sequestered, once you start talking to folks elsewhere in
the company it becomes clear that many teams are struggling to stay above
water. More still, everyone shrugs about it because it's not clear exactly
what is wrong. The best description I've heard is in many cases engineers are
willing to offer hacks as a solution to meet management's demands, and
management is either willing to accept those hacks or doesn't know better.
That is far from a full picture, and its an example drawn from a small slice
of an enormous company. But it seems telling to me. I find it completely
reasonable to imagine that most teams in a place to deal with customer facing
bugs don't have adequate time to do so. Not to their satisfaction, never mind
customers.

At the same time, I think it can be hard to appreciate the ways in which Apple
is ahead of the curve as far as the categories of software projects it
tackles. So I don't mean to imply that anyone is really to blame per se. And
it's also a shame because Radar was the greatest bug tracker I have ever used.
It is unclear to me if Apple doesn't prioritize menial things like handling
external bug reports, or if problems like that are not visible to / perceived
by those in the company who could do something about it.

~~~
DoofusOfDeath
I wonder if it's as simple as this: a company that's making so much money just
has no incentive to improve. After all, if the end goal of a company like
Apple is to make money, they're already doing it.

~~~
arvinsim
Improving processes and hiring new people shouldn't be a big dent to their
massive revenues though.

~~~
andrekandre
probably not, but from my anecdotal experience with upper managers (i don’t
work at apple), every little bit counts towards “hitting the targets”...it
seems unfortunately almost nowhere is immune to this pathological way of
thinking...

------
satisfice
I was on the team that wrote the requirements for the original version of
Radar, in 1988. At that time each department was using it's own flat file
database for bugs. Also at that time, there was a very large testing group.
All testers at Apple were together: more than 400 testers. We had a strong
culture of testing. We had pride in ourselves and our company. This was pretty
much thrown away when the group was disbanded in 1989. Professional testing
culture was replaced with the attitude that only developers mattered. Hurray
for "Total Quality Management" and later Agile, I imagine. I was gone by '91.

We originally designed Radar so that bugs would be verified as closed by the
person with most interest in seeing this happen: the tester assigned to that
part of that project. Then management swooped in with an edict that bugs must
be verified as closed by whomever originally reported them. This is a stupid
idea, because it creates the perverse incentive that no one should report a
problem if they are outside the team (because then you are committing to
verify the fix, which just means more work for you that has nothing to do with
any of your main responsibilities).

When I pointed out that the system would now discourage people on different
teams from helping each other, the sponsoring director said "that's what pink
slips are for." Direct quote. Soon after that I resigned from the design team.

Without reasonably skilled and principled leadership, you just don't get
quality software. And "quality is everyone's job" is just an empty and
childish slogan. Excellence is not transmitted through slogans and wishful
thinking. You have to assign responsibility, provide resources and time (which
means lowering velocity of new development), and follow-up.

The fundamental reason why it doesn't happen is the technology market is not
efficient. Quality is, in fact, not as important as career testers wish it
were. You can get away with doing terrible work and not lose your job. The
fact that Apple pays no significant penalties for having buggy products
insulates it from our slings and arrows.

~~~
microtherion
I disagree. The person who wrote the bug is in the best position to decide
whether the problem was actually fixed.

~~~
redwoodmonkey
I could easily see it go either way; it really depends on the bug. Some
obvious issue, like "this button should do X but it does Y" can be verified by
almost anyone. But some issues need the attention of the original author to
really verify the bug. Maybe what needs to be done is someone in QA needs to
attempt to verify the bug, or "pre-verify", and then it goes back to the
originator for final verification, who can also verify it, or simply close it
if they feel like QA did a good job.

~~~
microtherion
That sounds like a good idea. As long as the originator has the option of
having the last word, internal attempts to verify can only help.

------
makecheck
When a process is annoying and you do nothing, people eventually _do_ give up
and leave. When it reaches that point, _they’re not coming back_ even if you
finally wake up and fix what bugged them.

Apple software quality is in serious danger precisely because of this type of
community and infrastructure rot. They are not encouraging developers to help
them, and a not-surprising number of serious issues have shown up in released
products in recent years.

I am one of those people who gave up on BugReporter. Lots has been written on
how tedious it can be but even its “rewritten” version a few years ago is
frustrating (basically you could be reporting a damned spelling error and they
would still want a “sysdiagnose”). Bugs stay open _for years_ and still can
somehow be Duplicate. Meanwhile, very obvious broken features make it into new
versions.

------
jakobegger
This is extremely frustrating to me as a macOS developer.

I put a lot of effort into my app. I spend about 50% of my time on triaging /
fixing / preventing bugs. We check every crash report that is logged, and we
attempt to reproduce every issue reported by customers as quickly as possible.

By now, a significant fraction of bugs are bugs in Apple's frameworks. We try
to report them to Apple, but they are ignored, or simply closed because they
are related to deprecated APIs.

Of course, customers don't complain that Apple frameworks are buggy -- they
complain that our app crashes! So Apple has no incentive to fix it. (If the
bug affects Apple apps, it has a higher chance of being fixed. If a framework
isn't used a lot by Apple, it will be full of bugs)

So we have dozens of workarounds in our code, and have stopped using some
frameworks altogether (if possible).

But I can confirm that Corbin was quick to review bugs -- NSTableView related
bugs were the only ones were I got fast feedback.

~~~
xhruso00
customers don't complain that Apple frameworks are buggy -- they complain that
our app crashes -> if you try to mention this on apple's developer forum you
will get banned. Apple doesn't like damaging their brand. As you have said
customer blames us, developers.

------
matthew-wegner
It's buried in the article:

 _When I worked at Apple I practiced what I am now preaching. I would screen
all my bugs within a day or so. I would verify bugs within a week or so. I
would poke around at code and attempt to make theoretical fixes for bugs I
couldn’t reproduce. I felt like it was part of my job, and I just did these
things as a daily task._

From Corbin's about page
[https://www.corbinstreehouse.com/blog/about/](https://www.corbinstreehouse.com/blog/about/)
:

 _For 13 years I worked at Apple on Cocoa, mostly doing UI implementation in
AppKit, such as NSTableView, NSWindow, NSVisualEffectView. I also did quite a
bit of work in UIKit – mainly on the early iPhone releases, but also helping
with the first few versions of the public SDK after that. I wrote and worked
on the base classes, such as UILabel and UITableView. You may recall seeing me
at WWDC in the labs, or up on stage giving a talk. Before Apple, I worked at
Borland, primarily on Delphi._

------
pkamb
The entire value of WWDC is going to the Labs, giving an Apple engineer your
Radar number, and having them read you or paraphrase the internal-only notes
attached to the ticket. Half the time the question/bug will be clearly
resolved internally or a workaround posted. But no updates are added to the
public ticket, and it will remain open and unchanged for years.

------
qhwudbebd
I gave up reporting bugs because they were ignored and not even acknowledged
let alone fixed - for years. There is no evidence they haven’t gone to
/dev/null. I won’t bother in future.

With Free software, I just do the work myself and include a patch. But beyond
the tongue-in-cheek possibility of scripting things to submit every bug
thousands of times to make it ‘popular’, it sounds like there’s no workaround
to make an Apple bug report worth the time taken to submit it.

~~~
user2342
Yes, I also gave up more or less. I recently had a problem with AppKit,
created a small example to reproduce, and used one of my Developer Incidents.
I got an acknowledgement from them, was asked to open a radar, and provide the
ticket number back. I did that but from that point on on there was silence. It
may be possible that it will be fixed for 10.15, but the experience from my
point of view is lousy.

------
RyJones
Logged a bug in Xcode that was breaking our CI. This was never fixed.

It was frustratingly easy to reproduce. Clear the process is broken internally
(as source article says) as well as externally. I guess when you sit on the
highest volume pipe of money coming out of wallets for consumer tech, there is
no reason to change?

~~~
WrtCdEvrydy
Yeah, I used to report those to Blackberry, but when they owned the entire
mobile field, noone gave a shit that you can't you get 'your code' on their
platform.

~~~
vardump
Oh yeah, I can remember when trying to deal with Blackberry over a decade ago,
when they were still relevant.

Ultimately, we just gave up and didn't release our app on their platform at
all.

~~~
WrtCdEvrydy
When their BB10 platform rolled out, the first thing they did was to make the
code deployment process simpler.

------
ThJ
Isn't this how bugs are handled in most software companies? I have never seen
anyone ever fix a bug that I report as a user. I have been asked for plenty of
irrelevant diagnostics logs, seen bug tickets get relegated to low priority or
won't fix, or closed because the ticket wasn't updated, though. (User
perspective: I shouldn't have to nag. Dev perspective: Maybe it fixed itself
or the user doesn't care anymore.) Plenty of that. And I'm on the other end of
that at work. I'm a developer who must handle issue tickets, and there are
tickets that have been sitting there for months, and might sit there forever
unless my boss nags me to deal with them, because there are always more
pressing issues to deal with, not to mention tasks that are far more rewarding
to work on. I think there are limits to how much you can torture or bore a
developer before they just up and quit.

~~~
username223
Yeah, that's pretty much how bugs seem to be handled at most large software
companies/projects, though they all do it a bit differently. Google is famous
for making it next to impossible to speak to a human, instead sending obtuse
automated replies. Apple does "radar to /dev/null." Linux? I wouldn't even
consider wading into the cesspit of toxicity and petty dictatorship that is
LKML.

On a positive note, I have had some great experiences interacting with the
maintainers of smaller projects, and even some larger ones like Emacs. But
massive companies and projects? Not worth the effort.

------
jmull
> Or, they just won’t want to install the latest OS because it doesn’t have
> anything they want or need (that’s me, I run 10.13).

(Context is Apple asking bug originators to verify whether or not the bug
still occurs in the latest OS.)

This seems contradictory to the entire article. The point of logging a bug at
all is to get it fixed. The fix would be coming in a future release. If you
don’t want to take the latest updates that’s fine, but how are you going to
get bug fixes?

~~~
DoofusOfDeath
I wonder if the author mean that he's stuck on OS X 10.13.n, and would be
willing to install OS X 10.13.(n+1), just not 10.14?

~~~
jedieaston
Apple doesn't do point upgrades for previous releases unless there's a
critical security bug in them though, so that wouldn't happen. If you want the
problem fixed you have to jump to the next major release because that's what
90% of the active install base is on, since it's free and easy to install.

~~~
valleyer
> Apple doesn't do point upgrades for previous releases unless there's a
> critical security bug in them

This is not true. Minor updates to Mac OS X pretty much always contain a
handful of non-security bug fixes.

~~~
saagarjha
I have never seen Apple push an update to a previous version of macOS unless
there was some critical bug or security vulnerability in it.

~~~
valleyer
Many minor macOS updates add features. 10.6.6 added the App Store. 10.7.2
added iCloud. More recently, 10.14.3 [0] added things as small as "a menu item
to News for opening a story in Safari". (As an arguably even more trivial
example, 10.14.1 added new emoji.)

Most minor updates also fix small, non-critical bugs, like 10.13.4 [1], which
"fixes an issue that may prevent web link previews from appearing in
Messages".

Do all of these updates fix security bugs too? Yes, because a company as big
as Apple is probably always fixing security bugs. That doesn't mean they ONLY
do minor updates for security bugs though.

[0] [https://support.apple.com/en-
us/HT209149#macos10143](https://support.apple.com/en-us/HT209149#macos10143)

[1] [https://support.apple.com/en-us/HT208533](https://support.apple.com/en-
us/HT208533)

~~~
saagarjha
None of those were released when the operating system they applied to was a
"previous" version of macOS.

~~~
valleyer
Oh ok, I misunderstood you... sorry

------
rado
My “favourite” macOS bug is 14 years old and I have reported it countless
times, including in email exchanges with Federighi. The OS doesn’t respect an
app’s input language and changes it when switching apps. Default input,
English, gets overwritten on app switch. There is an option, it doesn’t really
work. This is my only serious problem with the macOS I love and it’s sad we
get new bugs galore while old ones are ignored.

~~~
zeckalpha
I've been bitten by the Spotlight key combination and the input switching key
combination being similar, but this is a PEBKAC rather than a bug.

I've since disabled the input switching key combination because it conflicts
with Emacs' set-mark-command.

------
saagarjha
> It’s okay to log bugs that are missing information; developers can ask for
> specifics of what they need. In general, for AppKit related bugs, I would
> find that a “sysdiagnose” rarely would provide any useful information. I
> would hate it when people (usually QA, and not the engineers who worked on
> the code) would ask me for a sysdiagnose, when I know it wouldn’t help them
> figure anything out.

Yeah, except Bug Reporter nags you _multiple times_ for a sysdiagnose if you
don't attach one (they have been slowing adding these, I've noticed), and if
you ignore these screening is going to send it back to you a week later,
without reading the bug, asking for you add one. Sometimes if you _do_ have
the sysdiagnose they'll ask for something equally stupid (system profile,
usually).

~~~
redwoodmonkey
The sysdiagnose does contain useful information that is useful for certain
bugs. If an app crashes, then the crash report is essential. The log file is
also (sometimes) useful. It is easier to ask for a sysdiagnose that gathers
the kitchen sink instead of the developer just providing the piece needed (ie:
the leaky faucet). -corbin

~~~
saagarjha
Yeah, I know that, but the screeners will often ask for a sysdiagnose in cases
where it's clearly not useful (for example, visual bugs that exist on an out-
of-box installation). So it's pretty clear they're either not reading the bug,
or going through some sort of checklist that one of the engineers gave them
(don't bother me with a bug without a sysdiagnose!)

~~~
redwoodmonkey
Yeah, it is stupid. I used to see this all the time.

~~~
rogual
The other day I reported a bug in Safari's regex engine and they asked for a
sysdiagnose.

I just don't have the energy to perform pointless tasks, so their browser can
stay broken.

(In case Apple is reading: /.*?x/u.exec('ふx') should match but doesn't.)

------
stubish
At an Agile presentation several years ago, they suggested the following
approach:

a) Bugs get broken down into manageable tasks of roughly the same size (eg.
about 1 day to fix).

b) You measure the average velocity that your team fixes these triaged bugs.

c) When the number of bug reports exceeds the teams velocity, you know that
number of bugs will never be fixed.

d) Be brutal, and flag that number of the lowest priroty bugs that will never
be fixed as WONTFIX.

Now you have a manageable queue of bugs, say as many as your team can fix in
two months. And the submitters know the score and know waiting isn't going to
help. Lots of productivity and efficiency bonuses all around.

But nobody does this or any of the similar variations, because people rather
leave bugs that everyone knows will never be fixed in the system 'because they
might be' or because they don't want to listen to the butt hurt submitters
complain because they place a higher priority on fixing the bug than the
people they are trying to convince to fix it.

And yes, this approach might seriously piss customers off enough that it is a
PR nightmare, so I'd be interested in anecdotes of this sort of approach being
applied and how it worked out (even just internally).

~~~
tonyedgecombe
Or just have a zero tolerance for bugs.

~~~
gok
That's effectively having a zero tolerance for functionality too.

~~~
tonyedgecombe
No it isn’t, in fact you could argue a better quality code base supports
faster development.

~~~
gok
You have some examples of bug-free software that does useful work?

~~~
tonyedgecombe
That's not what I was suggesting and I think you know that.

~~~
gok
What are you suggesting? I truly don't know what "zero tolerance for bugs"
could mean.

------
valleyer
> Management needs to allow engineers to have more time to do this, which is
> at the expense of working on features or fixing already screened bugs.

IMO, this sentence is the key. For Apple to improve its software quality, it
needs to stop trying to do many other things. We don't need new versions of
Mac OS X and iOS every year anymore, each with incremental feature additions
and new bugs.

------
macdice
Anecdote: Yeah I was a bit surprised that a kernel/filesystem bug I logged
back in November complete with a reproducer didn't even get acknowledged. I
hope they'll eventually get to it, as it causes spurious build farm failures
for PostgreSQL. My hypothesis was that the lack of appropriate category might
have doomed my report to the random/other/who-cares pile.

[http://www.openradar.me/radar?id=4970461708419072](http://www.openradar.me/radar?id=4970461708419072)

~~~
throwaway333333
It’s prioritized and in the queue...

~~~
macdice
Thank you!

------
abacadaba
> This is terrible because many engineers will not look at bugs with a low
> priority.

So taking inspiration from the React scheduling work, would it make sense to
give bugs a timeout based on priority to keep them from sitting around
forever? Say you have a low priority bug, it sits around forever while more
important stuff gets fixed, but if it's been sitting idle for 6 months it gets
an automatic priority bump?

------
banachtarski
They can't even fix external display issues with their laptop. I would say
bugs in their issue tracker that aren't even that high profile are hopeless.

------
bronson
Does this help explain Apple's bizarre inaction after learning about the
FaceTime bug?

[https://news.ycombinator.com/item?id=19029573](https://news.ycombinator.com/item?id=19029573)

------
hmottestad
Filed a bug with Apple 6 months ago. Haven't heard anything and the bug is
still there.

How to reproduce the bug on iPhone/iPad:

\- start playing some music

\- open control center and swipe down on the volume to mute it

\- turn up the volume one click on the physical buttons

\- swipe down on the volume in control center

\- repeat turning up the volume with the buttons and muting in the control
center until the control center volume stops affecting the actual volume

~~~
user2342
Yes, I could reproduce this easily.

------
jteppinette
I haven’t had an empty backlog since 2011.

~~~
ThemalSpan
It seems that Radar queues can only grow.

~~~
n0p3
When I first joined my team, my manager told me to take a screenshot of my
queue, because that was the only time it would ever be empty...

------
mikerg87
“Feedback” on public betas seems to be an even lower priority.

I’ve reported that each iOS beta gnaws away system storage since they started
the program.

My phone is currently giving up 176GB to system storage and seems to have no
end in sight.

So many enthusiast users I know are so frustrated they want to give up.

~~~
saagarjha
Feedback on public betas is probably lower priority than radars, unless a lot
of people are reporting something. I'm sure the quality of the bug reports is
much lower.

------
valleyer
> I would poke around at code and attempt to make theoretical fixes for bugs I
> couldn’t reproduce.

This sounds like a great way to accrete unnecessary code, some of which will
create even more bugs.

~~~
redwoodmonkey
Code to fix problems, even in theory, isn't unnecessary. Statistically,
changes I did were very unlikely to produce bugs. I wrote about this in my
post about why you shouldn't do code reviews:
[https://www.corbinstreehouse.com/blog/2018/01/code-
reviews-p...](https://www.corbinstreehouse.com/blog/2018/01/code-reviews-
part-1-dont-do-them/?cat_id=86&order=DESC)

~~~
valleyer
Interesting article!

I worked for 5 years on a team who had just started using code reviews. What
we found was that code reviews didn't often find bugs in the changes
themselves, but they were very effective at preventing bad decisions that
would cause bugs in the future.

There are lots of ways to make a change that gets the job done, and might even
fix a bug, but can mean twice as much work to deal with in the future. These
range from intentional but shortsighted hacks to simple mistakes by
experienced engineers working in a large project.

You also mention style nits. I admit even I get annoyed by these. But bad
style can make it harder for new hires to learn the codebase, which slows the
team down. In rare cases, they can even result in bugs of their own like the
infamous Apple "goto fail" bug.

Every team is different, but I can't imagine working on one without code
reviews now.

------
cinquemb
And this is the foundation they want to build their little new "services"
empire on. Good to be a fat cat… until it isn't.

------
opportune
Encountered a pretty major macOS bug recently where some new OS update made it
impossible to enable a root user via the regular route. Made it impossible to
set up a local/pseudo-distributed hadoop cluster since the main hadoop "start"
command requires "sudo su" to work. Doesn't surprise me their bug reporting
system sucks considering I couldn't figure out the right place to submit my
bug nor could I find a new way to do this

~~~
saagarjha
What was wrong, exactly?

------
chris_wot
Apple have a real quality problem. I would say it is worse than Microsoft
Connect.

~~~
Ono-Sendai
MS connect was deleted, breaking all old bug links:
[https://docs.microsoft.com/en-us/collaborate/connect-
redirec...](https://docs.microsoft.com/en-us/collaborate/connect-redirect)

~~~
chris_wot
Sorry - worse than it was. And yes, that decision was sheer idiocy.

