Disclaimer, I worked with Corbin for a decade, and have tons of respect for him. I agree with the diagnosis here, and I have one additional treatment suggestion:
The single easiest and most effective thing Apple could do to improve its SWE org is to invest in Radar.
Radar's importance within Apple cannot be overstated. It subsumes what would be multiple tools in other orgs. As an Apple SWE you spend a massive amount of time in it. And yet Apple treats Radar as a cost center, developed by an outsourced offshore team. It's slow to search, supports only plain text, is hard to script, and is missing obvious features, e.g. automatic duplicate finding.
Hire five good SWEs, give them a mandate to serve the needs of the org, and you will massively increase the effectiveness of every other engineer.
(My experience is now rather stale and I hope they've already formed such a team!)
Anecdotally, I didn't find the situation internally to be much better. Many bugs internally go unanswered because there is not enough time allocated to fixing core systems and designing better replacements. The truth is, I know personally of several teams that aren't able to get through the queue of internally filed and scheduled bugs.
To me, it feels like Apple hasn't resourced core pieces of infrastructure and engineering teams in line with upper management's plans for growth. While many teams are relatively sequestered, once you start talking to folks elsewhere in the company it becomes clear that many teams are struggling to stay above water. More still, everyone shrugs about it because it's not clear exactly what is wrong. The best description I've heard is in many cases engineers are willing to offer hacks as a solution to meet management's demands, and management is either willing to accept those hacks or doesn't know better. That is far from a full picture, and its an example drawn from a small slice of an enormous company. But it seems telling to me. I find it completely reasonable to imagine that most teams in a place to deal with customer facing bugs don't have adequate time to do so. Not to their satisfaction, never mind customers.
At the same time, I think it can be hard to appreciate the ways in which Apple is ahead of the curve as far as the categories of software projects it tackles. So I don't mean to imply that anyone is really to blame per se. And it's also a shame because Radar was the greatest bug tracker I have ever used. It is unclear to me if Apple doesn't prioritize menial things like handling external bug reports, or if problems like that are not visible to / perceived by those in the company who could do something about it.
I wonder if it's as simple as this: a company that's making so much money just has no incentive to improve. After all, if the end goal of a company like Apple is to make money, they're already doing it.
probably not, but from my anecdotal experience with upper managers (i don’t work at apple), every little bit counts towards “hitting the targets”...it seems unfortunately almost nowhere is immune to this pathological way of thinking...
Apple's culture is such that everything is in Radar. EVERYTHING. It doesn't get done if it's not in Radar.
Positive: Everything is searchable. It is organizational history going back decades. The company is built around this tool.
Negative: The importance of such a tool goes unrecognized compared to the criticality of the system to Apple as a whole. So when it's slow or breaks, everyone is having a bad day.
> …it becomes clear that many teams are struggling to stay above water. More still, everyone shrugs about it because it's not clear exactly what is wrong…
I was on the team that wrote the requirements for the original version of Radar, in 1988. At that time each department was using it's own flat file database for bugs. Also at that time, there was a very large testing group. All testers at Apple were together: more than 400 testers. We had a strong culture of testing. We had pride in ourselves and our company. This was pretty much thrown away when the group was disbanded in 1989. Professional testing culture was replaced with the attitude that only developers mattered. Hurray for "Total Quality Management" and later Agile, I imagine. I was gone by '91.
We originally designed Radar so that bugs would be verified as closed by the person with most interest in seeing this happen: the tester assigned to that part of that project. Then management swooped in with an edict that bugs must be verified as closed by whomever originally reported them. This is a stupid idea, because it creates the perverse incentive that no one should report a problem if they are outside the team (because then you are committing to verify the fix, which just means more work for you that has nothing to do with any of your main responsibilities).
When I pointed out that the system would now discourage people on different teams from helping each other, the sponsoring director said "that's what pink slips are for." Direct quote. Soon after that I resigned from the design team.
Without reasonably skilled and principled leadership, you just don't get quality software. And "quality is everyone's job" is just an empty and childish slogan. Excellence is not transmitted through slogans and wishful thinking. You have to assign responsibility, provide resources and time (which means lowering velocity of new development), and follow-up.
The fundamental reason why it doesn't happen is the technology market is not efficient. Quality is, in fact, not as important as career testers wish it were. You can get away with doing terrible work and not lose your job. The fact that Apple pays no significant penalties for having buggy products insulates it from our slings and arrows.
I worked at Apple from ~1995-2002. The status quo described here, where the original bug reporter has to verify and close the bug was a beautiful thing. Nobody saw it as a disincentive to file bugs. It's not "extra work" to confirm that the team you begged to improve software had actually done what you suggested. The alternative, that software teams internally determine that the bug has been "fixed" without understanding nuances of the original report, is much worse. I love Apple's standard that the original bug reporter should be responsible for closing a bug, and I only wish that the privilege extended to third-party bug reports made via Bug Reporter.
I could easily see it go either way; it really depends on the bug. Some obvious issue, like "this button should do X but it does Y" can be verified by almost anyone. But some issues need the attention of the original author to really verify the bug. Maybe what needs to be done is someone in QA needs to attempt to verify the bug, or "pre-verify", and then it goes back to the originator for final verification, who can also verify it, or simply close it if they feel like QA did a good job.
When a process is annoying and you do nothing, people eventually do give up and leave. When it reaches that point, they’re not coming back even if you finally wake up and fix what bugged them.
Apple software quality is in serious danger precisely because of this type of community and infrastructure rot. They are not encouraging developers to help them, and a not-surprising number of serious issues have shown up in released products in recent years.
I am one of those people who gave up on BugReporter. Lots has been written on how tedious it can be but even its “rewritten” version a few years ago is frustrating (basically you could be reporting a damned spelling error and they would still want a “sysdiagnose”). Bugs stay open for years and still can somehow be Duplicate. Meanwhile, very obvious broken features make it into new versions.
This is extremely frustrating to me as a macOS developer.
I put a lot of effort into my app. I spend about 50% of my time on triaging / fixing / preventing bugs. We check every crash report that is logged, and we attempt to reproduce every issue reported by customers as quickly as possible.
By now, a significant fraction of bugs are bugs in Apple's frameworks. We try to report them to Apple, but they are ignored, or simply closed because they are related to deprecated APIs.
Of course, customers don't complain that Apple frameworks are buggy -- they complain that our app crashes! So Apple has no incentive to fix it. (If the bug affects Apple apps, it has a higher chance of being fixed. If a framework isn't used a lot by Apple, it will be full of bugs)
So we have dozens of workarounds in our code, and have stopped using some frameworks altogether (if possible).
But I can confirm that Corbin was quick to review bugs -- NSTableView related bugs were the only ones were I got fast feedback.
customers don't complain that Apple frameworks are buggy -- they complain that our app crashes -> if you try to mention this on apple's developer forum you will get banned. Apple doesn't like damaging their brand. As you have said customer blames us, developers.
I received an email yesterday that a bug I filed in 2013 in the Android Google maps SDK has been assigned to someone. Might as well just not fix it now.
Additionally, something will work with the stock SDK, but will break on Samsung devices (e.g. camera SDK).
When I worked at Apple I practiced what I am now preaching. I would screen all my bugs within a day or so. I would verify bugs within a week or so. I would poke around at code and attempt to make theoretical fixes for bugs I couldn’t reproduce. I felt like it was part of my job, and I just did these things as a daily task.
For 13 years I worked at Apple on Cocoa, mostly doing UI implementation in AppKit, such as NSTableView, NSWindow, NSVisualEffectView. I also did quite a bit of work in UIKit – mainly on the early iPhone releases, but also helping with the first few versions of the public SDK after that. I wrote and worked on the base classes, such as UILabel and UITableView. You may recall seeing me at WWDC in the labs, or up on stage giving a talk. Before Apple, I worked at Borland, primarily on Delphi.
The entire value of WWDC is going to the Labs, giving an Apple engineer your Radar number, and having them read you or paraphrase the internal-only notes attached to the ticket. Half the time the question/bug will be clearly resolved internally or a workaround posted. But no updates are added to the public ticket, and it will remain open and unchanged for years.
I gave up reporting bugs because they were ignored and not even acknowledged let alone fixed - for years. There is no evidence they haven’t gone to /dev/null. I won’t bother in future.
With Free software, I just do the work myself and include a patch. But beyond the tongue-in-cheek possibility of scripting things to submit every bug thousands of times to make it ‘popular’, it sounds like there’s no workaround to make an Apple bug report worth the time taken to submit it.
Yes, I also gave up more or less. I recently had a problem with AppKit, created a small example to reproduce, and used one of my Developer Incidents. I got an acknowledgement from them, was asked to open a radar, and provide the ticket number back. I did that but from that point on on there was silence. It may be possible that it will be fixed for 10.15, but the experience from my point of view is lousy.
Logged a bug in Xcode that was breaking our CI. This was never fixed.
It was frustratingly easy to reproduce. Clear the process is broken internally (as source article says) as well as externally. I guess when you sit on the highest volume pipe of money coming out of wallets for consumer tech, there is no reason to change?
> I guess when you sit on the highest volume pipe of money coming out of wallets for consumer tech, there is no reason to change?
This reminds me of the concept of "f* you" money that lets employees quit their jobs. It just dawned on me that some companies have that as well, w.r.t. their customers.
Yeah, I used to report those to Blackberry, but when they owned the entire mobile field, noone gave a shit that you can't you get 'your code' on their platform.
Isn't this how bugs are handled in most software companies? I have never seen anyone ever fix a bug that I report as a user. I have been asked for plenty of irrelevant diagnostics logs, seen bug tickets get relegated to low priority or won't fix, or closed because the ticket wasn't updated, though. (User perspective: I shouldn't have to nag. Dev perspective: Maybe it fixed itself or the user doesn't care anymore.) Plenty of that. And I'm on the other end of that at work. I'm a developer who must handle issue tickets, and there are tickets that have been sitting there for months, and might sit there forever unless my boss nags me to deal with them, because there are always more pressing issues to deal with, not to mention tasks that are far more rewarding to work on. I think there are limits to how much you can torture or bore a developer before they just up and quit.
Yeah, that's pretty much how bugs seem to be handled at most large software companies/projects, though they all do it a bit differently. Google is famous for making it next to impossible to speak to a human, instead sending obtuse automated replies. Apple does "radar to /dev/null." Linux? I wouldn't even consider wading into the cesspit of toxicity and petty dictatorship that is LKML.
On a positive note, I have had some great experiences interacting with the maintainers of smaller projects, and even some larger ones like Emacs. But massive companies and projects? Not worth the effort.
> Or, they just won’t want to install the latest OS because it doesn’t have anything they want or need (that’s me, I run 10.13).
(Context is Apple asking bug originators to verify whether or not the bug still occurs in the latest OS.)
This seems contradictory to the entire article. The point of logging a bug at all is to get it fixed. The fix would be coming in a future release. If you don’t want to take the latest updates that’s fine, but how are you going to get bug fixes?
They have a habit of spamming this request semi-randomly, seemingly without underlying knowledge of an attempted fix - e.g. you get the same request, at the same time, on several bugs. Sometimes it’s obvious it doesn’t come from any knowledge of a fix, e.g. asking to verify if a feature request is still a problem when no such feature was added.
So you say no, it wasn’t (it never is for me) and the bug goes back into limbo.
I am the author. I always do dot upgrades, as they are (usually) security updates. I wouldn't say I am "stuck" on any particular OS; I choose to remain on it for stability.
Apple doesn't do point upgrades for previous releases unless there's a critical security bug in them though, so that wouldn't happen. If you want the problem fixed you have to jump to the next major release because that's what 90% of the active install base is on, since it's free and easy to install.
Many minor macOS updates add features. 10.6.6 added the App Store. 10.7.2 added iCloud. More recently, 10.14.3 [0] added things as small as "a menu item to News for opening a story in Safari". (As an arguably even more trivial example, 10.14.1 added new emoji.)
Most minor updates also fix small, non-critical bugs, like 10.13.4 [1], which "fixes an issue that may prevent web link previews from appearing in Messages".
Do all of these updates fix security bugs too? Yes, because a company as big as Apple is probably always fixing security bugs. That doesn't mean they ONLY do minor updates for security bugs though.
My “favourite” macOS bug is 14 years old and I have reported it countless times, including in email exchanges with Federighi. The OS doesn’t respect an app’s input language and changes it when switching apps. Default input, English, gets overwritten on app switch. There is an option, it doesn’t really work. This is my only serious problem with the macOS I love and it’s sad we get new bugs galore while old ones are ignored.
> It’s okay to log bugs that are missing information; developers can ask for specifics of what they need. In general, for AppKit related bugs, I would find that a “sysdiagnose” rarely would provide any useful information. I would hate it when people (usually QA, and not the engineers who worked on the code) would ask me for a sysdiagnose, when I know it wouldn’t help them figure anything out.
Yeah, except Bug Reporter nags you multiple times for a sysdiagnose if you don't attach one (they have been slowing adding these, I've noticed), and if you ignore these screening is going to send it back to you a week later, without reading the bug, asking for you add one. Sometimes if you do have the sysdiagnose they'll ask for something equally stupid (system profile, usually).
The sysdiagnose does contain useful information that is useful for certain bugs. If an app crashes, then the crash report is essential. The log file is also (sometimes) useful. It is easier to ask for a sysdiagnose that gathers the kitchen sink instead of the developer just providing the piece needed (ie: the leaky faucet). -corbin
Yeah, I know that, but the screeners will often ask for a sysdiagnose in cases where it's clearly not useful (for example, visual bugs that exist on an out-of-box installation). So it's pretty clear they're either not reading the bug, or going through some sort of checklist that one of the engineers gave them (don't bother me with a bug without a sysdiagnose!)
If it can be collected, I guess it's a good process for them. It may not be super user-friendly, but if enough bugs require sysdiagnose for solving, then it's an improvement to not even bother devs until it's provided.
It may not be needed then, but it still saves time.
Even if it weren’t, at least some amount of thinking should be involved in requesting it, and e.g. don’t do it on feature requests or API defect problems. But they do, oh they do.
It should also be noted that it doesn’t save time: it off-loads some time wastage to an external developer who is filling the bug. And who has to pointlessly collect it (it takes several minutes and knocks your system off), then upload 200-400MB file to a notoriously crappy form that it will time out or error out, or sometimes say the upload was successful only to get a bitchy QA reply a week later to upload sysdiagnose.
A few years back, I had a sysdiagnose so huge that Radar just wouldn’t take it. I spent an hour trying to upload the damn thing and eventually uploaded it elsewhere and posted an URL.
To this day, that sysdiagnose file was not downloaded a single time.
FWIW, Bug Reporter has not yet required sysdiagnoses for feature requests. And for large sysdiagnoses, you can ask for an alternative method to upload them; at some point I was given credentials for a FTP server I could upload it to.
At an Agile presentation several years ago, they suggested the following approach:
a) Bugs get broken down into manageable tasks of roughly the same size (eg. about 1 day to fix).
b) You measure the average velocity that your team fixes these triaged bugs.
c) When the number of bug reports exceeds the teams velocity, you know that number of bugs will never be fixed.
d) Be brutal, and flag that number of the lowest priroty bugs that will never be fixed as WONTFIX.
Now you have a manageable queue of bugs, say as many as your team can fix in two months. And the submitters know the score and know waiting isn't going to help. Lots of productivity and efficiency bonuses all around.
But nobody does this or any of the similar variations, because people rather leave bugs that everyone knows will never be fixed in the system 'because they might be' or because they don't want to listen to the butt hurt submitters complain because they place a higher priority on fixing the bug than the people they are trying to convince to fix it.
And yes, this approach might seriously piss customers off enough that it is a PR nightmare, so I'd be interested in anecdotes of this sort of approach being applied and how it worked out (even just internally).
He makes a reasonable case that not keeping the bug queue manageable leads to increased cost. I've definitely been in bug triage mtgs where we spend more time discussing a bug than the code change would take.
I don't know if you mean fixing every bug, or not creating bugs in the first place, but either way it is just the case where the teams bug fixing velocity is greater or equal to the rate of incoming bugs.
> Management needs to allow engineers to have more time to do this, which is at the expense of working on features or fixing already screened bugs.
IMO, this sentence is the key. For Apple to improve its software quality, it needs to stop trying to do many other things. We don't need new versions of Mac OS X and iOS every year anymore, each with incremental feature additions and new bugs.
Anecdote: Yeah I was a bit surprised that a kernel/filesystem bug I logged back in November complete with a reproducer didn't even get acknowledged. I hope they'll eventually get to it, as it causes spurious build farm failures for PostgreSQL. My hypothesis was that the lack of appropriate category might have doomed my report to the random/other/who-cares pile.
> This is terrible because many engineers will not look at bugs with a low priority.
So taking inspiration from the React scheduling work, would it make sense to give bugs a timeout based on priority to keep them from sitting around forever? Say you have a low priority bug, it sits around forever while more important stuff gets fixed, but if it's been sitting idle for 6 months it gets an automatic priority bump?
They can't even fix external display issues with their laptop. I would say bugs in their issue tracker that aren't even that high profile are hopeless.
Feedback on public betas is probably lower priority than radars, unless a lot of people are reporting something. I'm sure the quality of the bug reports is much lower.
Code to fix problems, even in theory, isn't unnecessary. Statistically, changes I did were very unlikely to produce bugs. I wrote about this in my post about why you shouldn't do code reviews: https://www.corbinstreehouse.com/blog/2018/01/code-reviews-p...
I worked for 5 years on a team who had just started using code reviews. What we found was that code reviews didn't often find bugs in the changes themselves, but they were very effective at preventing bad decisions that would cause bugs in the future.
There are lots of ways to make a change that gets the job done, and might even fix a bug, but can mean twice as much work to deal with in the future. These range from intentional but shortsighted hacks to simple mistakes by experienced engineers working in a large project.
You also mention style nits. I admit even I get annoyed by these. But bad style can make it harder for new hires to learn the codebase, which slows the team down. In rare cases, they can even result in bugs of their own like the infamous Apple "goto fail" bug.
Every team is different, but I can't imagine working on one without code reviews now.
Encountered a pretty major macOS bug recently where some new OS update made it impossible to enable a root user via the regular route. Made it impossible to set up a local/pseudo-distributed hadoop cluster since the main hadoop "start" command requires "sudo su" to work. Doesn't surprise me their bug reporting system sucks considering I couldn't figure out the right place to submit my bug nor could I find a new way to do this
Each team at MS is independently ran. I worked on teams where all bugs, and I mean all bugs, with more than ~5 repros in the wild, got fixed.
Then I worked on other teams that just gave up. Sometimes for good reason, fixing long standing bugs could mean a major re-architecture of existing code that would likely induce more bugs than it would fix.
But I also saw teams give up due to sheer overwork and lack of time.
The single easiest and most effective thing Apple could do to improve its SWE org is to invest in Radar.
Radar's importance within Apple cannot be overstated. It subsumes what would be multiple tools in other orgs. As an Apple SWE you spend a massive amount of time in it. And yet Apple treats Radar as a cost center, developed by an outsourced offshore team. It's slow to search, supports only plain text, is hard to script, and is missing obvious features, e.g. automatic duplicate finding.
Hire five good SWEs, give them a mandate to serve the needs of the org, and you will massively increase the effectiveness of every other engineer.
(My experience is now rather stale and I hope they've already formed such a team!)