
“A Cold War Every Day” inside Apple's internal tools group - ttepasse
https://www.buzzfeednews.com/article/alexkantrowitz/always-day-one-exclusive-excerpt-apple
======
motohagiography
Sounds like a bank, a government agency, or any business over a certain size.

I tell people you only call it politics when you are losing. More accurately,
it's a layer of literal stupidity above the competent to shield the money side
of the company from the leverage that operations people would have if they had
any information about how the money side worked.

Instead of a hierarchy, rethink a company as a hub and spoke model with
concentric rings. The main differences are the implication in a hierarchy that
there is "gravity," keeping people down and that they need energy and leverage
to climb "up," which further implies there is a place to "fall," and that
there is only one way "up," instead of many possible paths to the centre from
all directions. There is no gravity, only gates and barriers, and even these
are just information. Politics is how a middle manager runs interference and
creates distractions to make sure you can't see over, around, or through them,
and that the people behind them closer to the money can't see you. Tech is
usually outside the main perimeter, mediated by contracting companies or
middle managers whose job is to compartmentalize the value people create, and
be sure it is replaceable.

Viewed this way, of course this demented political farce is how Apple works,
because it's how everything seems to work when you have internalized the
precise and specific mental model someone uses to take advantage of you.

Sorry if you can't unsee it now, but hopefully it will be funny and we can get
good, competent people who value tangible skills into positions of power.

~~~
seph-reed
This is just your own personal model? It's beautiful!

Is there a community where we could discuss something like this? "The spoke."
I want to get obsessed with this.

~~~
sabalaba
[https://www.ribbonfarm.com/2009/10/07/the-gervais-
principle-...](https://www.ribbonfarm.com/2009/10/07/the-gervais-principle-or-
the-office-according-to-the-office/)

~~~
pm90
Its hard to not be cynical about corporations after reading that excellent
post.

------
IST-Throwaway
Fascinating to see an article about IS&T, definitely not something I ever
expected to hear about on HN!

I was part of a team inside Apple that developed internal-facing tools for the
retail teams. Our team was formed because of the extremely high cost in
dollars and time that IS&T wanted to charge to develop some fairly
straightforward tools. My team was taken from the departments that used the
tools, and we developed things that were very custom-fit to what those
departments needed.

Eventually politics overcame us, and IS&T finally managed to take over our
team. We all became contractors in order to support the migration to their
infrastructure, and continue development of the tools. A bit later we all
became fired as our project was outsourced to contractors in India, managed by
IS&T PMs who had no idea what our tool was used for and had never been inside
a retail store. The tools all died shortly after that, some killed by IS&T,
some petered out due to lack of use now that they no longer worked correctly
or were a good fit for the users.

As far as we could tell, IS&T was run as a unique company inside Apple, which
did it's fair share of price gouging in order to make itself money to keep
going. Its purpose never appeared to be helping Apple customers or Apple
employees, it was simply to get bigger, absorbing more money and power
wherever possible, with no apparent reigning in from the parent org.

Although my experience is several years old, everything in this article rings
true. The contracting companies they had us working for were taking a huge
cut, the quality of the code they produced was dismal, (as soon as we were no
longer allowed to re-write their code major things began breaking almost
immediately) and people getting transferred around constantly and having no
time to understand any one project was common. (rkho's comment about their
hiring process seeming like it was simply a beard for a nepotistic contractor
conversion was something we definitely saw a number of times.)

All in all it was an extremely eye-opening experience. Considering how "do it
the Apple way" every other department we interacted with was, being in the
IS&T buildings was like landing on an alien planet.

~~~
_bxg1
When will companies learn that cheap development labor always ends up being
more expensive than starting with quality? This same story has been playing
out over and over for _decades_.

~~~
hermitdev
Around 12-13 years ago, I was working (as a team of one on a highly technical
upgrade, 32-64 bit upgrade, new compilers, OS changes, etc. Basically
everything was changing). Naturally, I was falling behind debugging obscure
issues largely due to 32->64 bit (this was C++ cross platform between windows
and Linux).

My director offered to toss some offshore resources my way to help. Bluntly
told him: I think I have 3 months work ahead of me. Give me offshore
resources, it'll more likely be a 9-12 months, because by the time ive
described the problem in sufficient detail, I could have just done the work.
Instead, id have to review a broken "solution" that doesnt work, probably
makes things worse.

Offshore is fine when you need grunt work, pounding out CRUD interfaces or
already having a detailed spec. But, offshore doesnt work when you're flying
by the seat of your pants.

Most of the big ossues I jad durong that release werent even my own. A few I
remember: an x86/x64 inline ASM bug, use of an uninitialized variable in a DB
lib that led to random failures dealing with NULLs, an overflow in a 3rd party
datetime lib and ABI mismatch because of wrong datatypes used in an OSS ODBC
wrapper that worked in 32-bit, but failed horribly on 64-bit.

None of the offshore talent I worked with at the time could have identified,
let alone fix those issues. Several of those took reading the actual generated
assembler to see what is going on. I dont write much x86/64, but I can read
enough to know when shits going to hell.

The 2 big agencies I delt with were Salient and Accenture. I could work with
Salient, most of their asserts were Indian and spoke English. Our Accenture
resources were all Chinese. It was a pain in the ass because none of our
engineering resources spoke or read/wrote English. Everything had to go both
ways through a translator. Something was always lost in the translation. We
ended up eating the contract and backing out with Accenture.

~~~
_bxg1
> Offshore is fine when you need grunt work, pounding out CRUD interfaces or
> already having a detailed spec.

My partner works at a startup with a very simple CRUD project. They outsourced
one of the early sections of that UI. Even in such a simple case, integrating
that component has been a huge time-sink and maintaining it is basically out
of the question. They're working on a plan to rebuild it, as so often happens
with these things.

Fire-and-forget software simply doesn't work.

~~~
hhas01
Any monkey can write code. Writing the _right_ code critically depends on
communication.

------
rkho
A while ago I interviewed for a senior fullstack role with a very homogenous
team at Apple -- every single person on the team had been converted from the
same consulting company (I believe it was Wipro).

It was a bizarre interview with three of the engineers on the team on the same
call asking me very specific programming language trivia and nothing about my
experience nor ability to navigate complex political structures -- you know,
the things that you would expect a senior fullstack engineer to have before
advancing them to the next round.

Unsurprisingly, I wasn't advanced to the next round. In light of this article,
I can't help but wonder if it was all a ruse to simply demand another
nepotistic FTE conversion from their same consultancy.

~~~
timwaagh
Perhaps, but perhaps the role just wasn't fit for you. A lot of 'senior dev'
roles are much more about programming language trivia and much less about
leadership.

~~~
mabbo
I can't tell if you're being sarcastic, or if you live in some kind of
Kafkaesque dystopia I never want to be part of.

At my employer, the best senior developers are the ones who know they can
google the answer to trivia so they don't need to remember it.

~~~
hermitdev
> the best senior developers are the ones who know they can google the answer
> to trivia so they don't need to remember it.

I've +20 years developing professionally. I remember the stuff I use day to
day, and know how to navigate the docs of the technologies I use frequently
for the less used tidbits. Being a good Senior isnt about knowing the answers
to everything, but knowing how to find the answers quickly. Also, how to ask
the right questions.

I don't know is always an acceptable answer, but should be followed up with
"I'll get back to you in an hour or two.", or something to that effect
(obviously for some problems, may need to experiment/prototype).

------
coleca
I had a similar experience w/Apple. A startup I worked for was brought into
Cupertino for a meeting w/their internal business teams. They wanted us to
build them an app which seemed relatively simple involving their internal Cafe
Mac cafeterias, data centers, and possibly retail locations. It was something
that a company like Apple could build in their sleep. But the business folks
we met with said that's how it is at Apple, all the engineering talent goes
towards the product side and almost nothing is left for internal IT. They told
us how they struggled to get anything done and there were almost no resources
available, so the business teams had to go and hire their own IT if they
needed things done.

~~~
ethbro
_> But the business folks we met with said that's how it is at Apple_

That's how it is at most companies.

Split orgs (IT & business) result in only work of sufficient size, scope, and
impact being able to cross the barrier.

Multi-year logistics management rewrite? You'll get two IT teams.

Frank in accounting needs to schedule a daily job to copy from one datastore
to another? He doesn't have permissions to, and has been doing it manually for
the last 5 years.

IMHO, every org along those lines would benefit from an independent IT tiger-
team whose sole job it is to find business problems and apply technology to
solve them.

~~~
dtech
And god forbid IT allows tech-savy or power-users to do things, that's not
secure.

~~~
jfb
At least at Apple, in the groups I was in, this was never an issue.

~~~
ethbro
That's the difference between "the business" being tech, vs everything else.

Kind of hard to lock down everything to personal environments, when the
majority of the company are developers!

------
Waterluvian
I could use some blog posts about quite functional teams.

Like someone on a team for a decade talking about why it works so well and
discusses cultural and political issues and how the team overcame them.

~~~
ChrisMarshallNY
I managed a high-functioning C++ image processing group for 25 years. I kept
senior-level engineers on staff for decades.

We tended to write "engine code." Like pipelines and whatnot.

The company had a very ( _very_ ) long tradition of engineering (Like, 100
years).

There were many downsides to the way they worked (over-structured, mostly),
but they always gave my team and me a great deal of respect.

They didn't use contractors very often.

To this day, design quality, code quality, product quality, documentation, and
focus on deliverable are the major cornerstones of my software work, but I am
shocked at how little that matters to modern development shops. It's been a
really disheartening experience.

I guess I was in a silo of quality-focus, all those years. They would treat
very minor bugs like extinction-level emergencies (not fun).

~~~
pm90
> but I am shocked at how little that matters to modern development shops.
> It's been a really disheartening experience.

Modern development shops operate in a hyper competitive environment with razor
thin margins. Unfortunately, great documentation is often the last thing
that's prioritized for such shops.

Personally, I think this is a HUGE mistake; talented technical writers are in
good supply and should be hired by any development shop to improve adoption of
their product. HOWEVER, as a developer, the shop will prefer that you work on
features rather than documentation.

~~~
ChrisMarshallNY
There are ways to write code in a manner that leaves a documentation trail.

I write about that here: [https://littlegreenviper.com/miscellany/leaving-a-
legacy/](https://littlegreenviper.com/miscellany/leaving-a-legacy/)

------
epberry
> For Apple, fixing its broken IS&T division would not only be the right thing
> to do from a moral standpoint — it would help the company’s business as
> well. If Apple is going to become inventive again, it will need to give its
> employees more time to develop new ideas.

I actually laughed. What kinds of applications do people think IS&T is
building? Supply chain management, HR, support software (Apple forums
anyone?)... I don't see how Apple sticking its core engineering resources or
at all innovating within IS&T helps their business.

Steve was the one that coined the 10X rule, or popularized it. That idea must
still hold within Apple's executive ranks. I guarantee Apple's top engineers
never interface with IS&T and their time won't be saved by improving it. In
fact I would bet that managing the deliverable of one of IS&T's projects is a
sign that you're not one of the golden ones.

That said, I obviously think these contractors should be treated much better
and overworking people for low pay is never okay. Maybe Apple could use some
smaller dev shops that treat their people better.

------
PeterStuer
"They’re just fighting for the roles,” Sabapathy told me. “That’s all they
care about, not the work, not the deliverables, the effort they put in, or
even talent. They’re not looking for any of those aspects.”

This is spot on for every large organization I ever consulted for. Moreover.
Even in much smaller businesses, typically starting at around 70-100 people
(sooner if management has prior large business experience) you can already see
this pattern starting to seep in.

This is also why the whole private vs public sector efficiency debate is
farcical. They are both identical in this aspect.

------
riskneutral
Sounds like the politics of every large corporate IT department, but with some
twists given that it's Apple. Maybe this helps explain why Apple is not
Amazon.

~~~
dehrmann
If there's a company CEOs should be afraid of, it's Amazon. They killed
bookstores, mall stores, how many other retailers? They were the first major
cloud provider. They tried their hand at phones, streaming video, and
streaming music, and the bought a grocery store. There's nothing they won't
try, and they have the war chest to do it.

------
seemslegit
Is there any big consumer product company out there that doesn't deprioritize
management and engineering talent for internal tools and systems ?

~~~
dcolkitt
It's an interesting question. And if the answer is no, I guess the logical
conclusion is that internal tools make very little difference to the success
of consumer product companies.

That doesn't seem like an intuitive conclusion, but it's a hard circle to
square. As someone who mostly builds internal tools (though in a very
different context), it certainly seems like good systems are an Archimedean
lever that acts as a force multiplier across the entire org. Yet, if good
internal tooling was important, you think at least one company would have been
able to achieve success partially on this basis.

The consumer product industry is extremely competitive, so the counter-
hypothesis of entrenched sclerotic incumbents refusing to pick up $20 bills
off the sidewalk doesn't really seem likely.

~~~
chillacy
To the contrary, some of the most successful software companies have strong
cultures of internal tools (think FAANG companies). How much of this is cause
and effect I'm not sure though. To some extent it's easier to make investments
in internal tooling if you're profitable.

~~~
astrange
Facebook in particular seems to wildly overinvest in internal tools, and if
you ask them why they don't use external stuff they tell you it's because "it
doesn't scale". We get things like Phabricator and HHVM out of it, but they
could've just not written those, and I'm pretty sure their business would be
fine.

They do it because it gets them promoted, and because FB and Google rely on
hiring every engineer ever and giving them make-work so they won't leave and
start a competitor.

~~~
epriest
Interesting! This isn't how I remember things at all. Do you know where you
got this sense of things from, specifically?

In particular, do you remember what gave you the impression that Facebook
"overinvested" a large amount of effort into Phabricator, that I developed and
open sourced Phabricator primarily to get promoted, that I built Phabricator
because of scaling concerns, or that the primary value I provided to Facebook
during my employment there was just in not starting a competitor?

------
jldugger
IS&T is run by the CFO and it shows.

------
samdung
The contract companies (Wipro, Infosys) hire graduates by the 100's from
mediocre engineering colleges, slap on a little bit of training and advertise
them as world class. The starting salaries are something like INR 25,000 (USD
350) per month. An onshore posting for these low skilled graduates is a quick
way to make money. So they suck up to their managers who suck up to bigger
managers. Their main criteria is to get hold of contract work by undercutting,
knowing very well there is vendor lock in.

------
5cott0
Body shop wars. What gets really interesting is working for a body shop with
consultants from 2 other shops on the same team.

------
captainredbeard
IS&T shouldn’t be part of Apple. It’s culture and engineering talent aren’t
representative of core engineering.

~~~
chillacy
In the old adage that "you get what you pay for", the internal tools at Apple
aren't quite the best as a result. The Radar tool I recall was pretty horrible
in particular.

~~~
astrange
Radar is the world's best bug tracker. The sole reason for this is that the
originator gets to close bugs, not the fixer.

~~~
_rs
Tell that to all the bugs I've filed that have been closed by Apple for some
bullshit reason or another :)

~~~
astrange
The new feedback assistant is actually a lot better about this - you get more
status updates and you can see the original bug state even if it was a
duplicate.

------
ummonk
"Don't work in IS&T" has been the common wisdom in Blind for years now.
Interesting how long it takes for the media to pick up stories like this.

------
larrik
Is this the team handling iTunesConnect and other such "services"? I'd
definitely believe it.

~~~
danpalmer
From what I understand, probably not.

iTunesConnect has had an overhaul in recent years to give it a much shinier
and more reliable frontend. I believe the original was closer to the original
WebObjects code, and now they've extracted out the user-facing bit into
something a bit more modern. There's a lot of legacy WebObjects stuff around
the iTunes backend, and my guess is that it's good engineers held back by old
tech, rather than just bad engineers.

~~~
saagarjha
It's App Store Connect now!

------
chmaynard
The days when the primary mission of IS&T was to develop custom software to
run Apple are probably long past. The big exception is Radar, an ideosyncratic
issue-tracking system which was developed in-house by IS&T around 1990. As far
as I know, Radar is still used in R&D and elsewhere at Apple.

Tim Cook probably couldn't care less about the internal affairs of IS&T. As
long as they do their job, play by the rules, and don't impose too much
overhead on the company, it's all good.

------
xp84
The part I don't get is, why does a company, especially one like Apple who
would count software quality as a core competency, choose to pay $120-150 an
hour to get contractors who aren't even good enough to command a salary (or
hourly rate) of half that. I get why like, United Airlines does that. Or the
US Government. They probably rightly assume that they can't properly manage a
huge software development or IT org. But Apple? Really? What the hell are they
gaining? If they paid a developer full time $120k a year plus benefits that
still beats $120+ an hour. WTF.

------
apple2019
I would be grateful if any Apple insiders here can discuss the risk of layoffs
if one were to join the company now.

I have an offer to join an engineering team at Apple, but the chances of
getting hit by layoffs is giving me pause. More details on my situation are
here:
[https://news.ycombinator.com/item?id=22809769](https://news.ycombinator.com/item?id=22809769)

~~~
sjg007
Apple has a ton of cash. I think layoffs are unlikely. Many times new hires
are protected as well for a bit.

~~~
apple2019
Do you work for Apple? Might you have any insight into how the company decides
how / who to layoff, when the time comes?

I would have thought that the newest would be the most vulnerable, since they
haven't had the time to absorb institutional knowledge and make internal
connections.

------
danzig13
Sounds exactly like government contracting.

------
rclayton
Lol. Just like Govt contracting.

------
ksec
Does Radar belongs to IS&T?

And I wonder if this IS&T power grabbing and politics happen and grow within
the past 10 years. i.e Happen under Tim Cook's watch.

It surely doesn't seems very Steve Jobs approach.

~~~
nemo
Radar's owned by IS&T.

IS&T was completely broken under Jobs and hadn't changed much under Cook while
I was there.

------
ping_pong
One of the world's most successful and wealthy companies in history shouldn't
be bottom fishing for the best "deal" on employees. I wonder how much better
they could be if they decided to hire at the top of the market like Netflix
instead of the bottom.

That is probably why Apple software is among the worst in the industry, and
it's a shame. I know IS&T isn't strictly software like iTunes but it fits the
pattern I've seen at Apple. But I guess if it doesn't affect their sales maybe
I'm wrong and it doesn't matter and they are right.

~~~
saagarjha
The quality of Apple's software engineering organization is head and shoulders
above IS&T's; iTunes (…and Apple Music, News, etc.) is run under a different
(services) group.

