
Microsoft has been making quality assurance mistakes lately - tanglesome
http://www.computerworld.com/article/2859902/at-microsoft-quality-seems-to-be-job-none.html#linkedin
======
cbd1984
One of the main points:

> how in the world can a patch to the Web browser lock up the operating system
> when you have an office suite installed?

In a perverse way, I enjoy stuff like this. Why? Because, someday, Raymond
Chen will write about this and I will get to read about it on The Old New
Thing, and while Chen will do a wonderful job of explaining the immediate
causes, they will point to other technical oddities so bizarre that I will be
reduced to a kind of mute incomprehension usually reserved for people who deal
with the really _interesting_ anal insertion cases in our fine nation's ERs.

I enjoy that feeling.

[http://blogs.msdn.com/b/oldnewthing/](http://blogs.msdn.com/b/oldnewthing/)

~~~
mark-r
Too bad there's only one Raymond Chen. I'm sure there is enough material
within Microsoft to keep 10 of him busy.

------
amputect
This is exactly what I thought would happen when they gutted their QA staff
and reassigned the surviving testers as developers. Your employees act in
accordance with their incentives, and the incentives aren't there for QA.
Microsoft made it clear that they only value feature development and shipping,
and their employees now associate "doing QA" with "being fired en masse".

To corrupt a phrase, they've crapped their bed and now they have to lie in it.

Edited to add: I'm not some hardcore "M$FT SUXXXX" guy. I like and use their
products, but I'm deeply concerned about the direction the company seems to be
heading.

~~~
AceJohnny2
This is something that deeply bothers me in the software industry, the
segregation between Dev and QA, and the low value placed in QA. "Oh whatever,
QA are low-paid techs that just run scripts!" No. QA is vital to the
_functionality_ of the product, and a proper QA person should be just as
capable a programmer as a developer in order to 1) write and maintain a test
suite and 2) understand the tested program and how best to test it.

My previous company in Europe understood this, and that seemed the norm in the
surrounding industry. My current in America doesn't, and that seems the norm
in the Silicon Valley.

~~~
geoelectric
I'm a Quality Engineer, and have worked for a number of large and small
companies in the Valley after moving into quality from software engineering 13
years ago. I've seen a lot of teams.

Problem is it's a feedback loop. A lot of companies maintain (or did maintain)
armies of unskilled people who basically were script executors, with maybe a
bit of barely-trained skill in finding equivalence classes and boundary
testing so that basic test plans could be written.

I wouldn't even call this QA, even if it was misidentified as such--it's
Quality Control to catch problems after the fact, with no front end assurance
whatsoever. It's honestly really only good for pissing off people when you
stop the shipment at the last minute. And that lack of value has been broadly
recognized. When you combine the high cost associated with the brute-force
documentation associated with verbose test scripts, result templates, and
other prehistoric process artifacts aimed at this level of tester (and which
are usually simply unnecessary and actually detrimental to agility and pace)
it's no wonder they're being cut.

Problem is as classic brute force QA teams get gutted, the market has been
flooded with these testers, and most of them kind of suck. Hiring a decent
quality professional has gotten damned difficult. And when everyone you meet
with "QA" in their resume is a non-technical, uninfluential yo-yo, you get a
pretty bad impression of the discipline.

In the meantime, everyone as skilled, interested, or immersed as you describe
has been moved into software engineering, because QA was treated as an "entry-
level" position and nobody wants to be a non-technical, uninfluential yo-yo.
The rise of the Software Engineer in Test is as much a collective gambit to
keep skilled people in quality as it is identifying a new skill-set. And for
people in my line of work, identifying yourself as a SET has about a 30% pay
differential over identifying yourself as a QA Engineer, so there's plenty of
incentive to move in that direction.

But even then, the rise of the SET reflects an over-reliance on automation,
and an under-reliance on process-based quality improvement and the type of
fuzzy testing and intelligent guessing only a human is going to reasonably
provide. The net result is that edge and corner cases escape, because
automation is pretty shitty at catching those unless a fuzzer or an automated
monkey (which is essentially a UI fuzzer) can be used. Generally, you only
find problems where you've programmed it to look, which will be necessarily be
pretty narrow, and the automation cannot use discretion to vary off the script
to where new problems actually might lie. People have conflated the value of
QA with the value of bad QA, and have chucked it out the window without
understanding the cost. I'd guess that many have never even seen good QA to
understand what it can do.

Nevertheless, this particular story is egregious. However much you cut back on
quality control or assurance, the _ONE FUCKING THING_ you always guarantee is
that the user can get out of any problems that arise with minimum damage to
money, data, time, and customer goodwill, roughly in that order of priority.
Bugs in updates that kill your code signing stack and block further updates
are just plain unforgivable.

------
sjm-lbm
Unless I'm missing something, this article really seems unreasonably harsh. He
cites two bad updates within the past months (one for Windows 7, one for
Exchange) that have had to be pulled due to issues, but it seems like well
over half the article is devoted to complaints about patches for Windows 10 -
an (officially) unreleased product.

~~~
exelius
Yeah, this article is FUD.

The Windows 10 piece is stupid - it's a pre-release OS, these things happen.
It's why they haven't released the OS yet, so I'd say their QA process worked.
Pre-release OSes don't go through the same level of QA - if you don't like it,
you should use a supported OS.

Second, with Exchange updates, there's kind of an unspoken understanding that
there is no way MS can test every scenario their users will throw at Exchange.
So updates are often released that can cause edge case issues, with the
understanding that their large customers will test the updates in a test
environment before deploying them. If issues come up, Microsoft can pull the
update. These kinds of retractions are not uncommon with server software.

~~~
Ded7xSEoPKYNsDd
How long can you responsibly run Exchange on the internet after a security
patch was released? The recent Drupal SQL injection [1] showed that attackers
have databases of fingerprinted services, just waiting for an update that
shows them how to exploit these systems. Reverse engineering the binary
patches for Exchange and building a reliable exploit out of them probably
takes longer than it did in the case of that SQL injection in Drupal. But how
long a testing period is too long?

1: [https://www.drupal.org/PSA-2014-003](https://www.drupal.org/PSA-2014-003)

~~~
exelius
In the case of something like a corporate Exchange server, you can run an
IDS/IPS system in front of them to block any specific exploits. Even then,
enterprise configurations of Exchange are usually quite unique to the company,
so there's a low likelihood of a security problem being exposed to the
outside.

------
Corvus
Ordinarily I would brush off complaints about prerelease alpha software
quality. But the same problems are affecting mature flagship products.

Last month a Microsoft Visual Studio update intercoursed all our Azure
development machines. We never got any reply from Microsoft, and had to
reimage all the machines. I see from
[https://connect.microsoft.com/VisualStudio/feedback/details/...](https://connect.microsoft.com/VisualStudio/feedback/details/1029189/after-
update-4-has-been-installed-such-packages-are-failed-to-load) this has nearly
a dozen repos, and still no official reply or workaround.

~~~
teh_klev
Forget Connect. It's a barren dustbowl full of tumbleweed. If it's urgent
shell out for a PSS support issue. Can't remember what we pay per "urgent/high
priority" incident but it's small potatoes compared to waiting around for a
response on Connect.

I've emailed Scott Guthrie a couple of times in the past about the sorry state
of Connect. Maybe they'll just shoot it in the head and stop pretending it's
useful to anyone.

------
josefresco
I stopped reading when he suggested MS buy Stardock and "call it a day". This
is what passes as news/journalistic content at Computer World? And how in the
world did it bubble up onto HN?

I hold out hope that Microsoft has bigger plans for it's operating system than
simply pandering to the tech-media's demands for, of all things... the start
menu to return.

~~~
nailer
The Start8 Windows 8 dock (in windows 8 mode) is more in place with current UI
of Windows than Microsoft's Windows 10 one.

------
jaxbot
The title seems to apply to Apple, too (see iOS and Yosemite WiFi bugs), and
could probably be applied to Google, too (Hangouts bugs, Lollipop problems,
etc). But mistakes happen and many bugs are less obvious to reproduce than
others, so I wouldn't point fingers too hard, personally.

~~~
TheOtherHobbes
If a car blew up and left you stranded whenever you filled it with gas, or
unlocked its doors whenever you left it out in the rain, you'd demand a refund
and perhaps compensation.

It's a testament to the usefulness of lawyers that the big players in the
software industry don't even pretend they can work at the level of
professional and managerial competence offered by almost all other consumer
and B2B industries.

~~~
breischl
Seriously! You never hear about a car would accidentally accelerate when you
try to brake, or that the tires would spontaneously blow out, or that they
would just plain roll over. I mean, the phrase "automobile recall" is
practically an oxymoron!

[1] -
[https://en.wikipedia.org/wiki/Toyota_Prius_(XW30)#Recall](https://en.wikipedia.org/wiki/Toyota_Prius_\(XW30\)#Recall)
[2] -
[http://www.forbes.com/2000/08/18/mu3.html](http://www.forbes.com/2000/08/18/mu3.html)
[3] - [http://www.cbsnews.com/news/toyota-will-recall-lexus-suv-
for...](http://www.cbsnews.com/news/toyota-will-recall-lexus-suv-for-rollover-
issue/)

~~~
dangrossman
The fact that "automobile recall" is something that exists and is expected
when defects occur, but "software recall" largely does not exist, was his
point.

~~~
wvenable
It's almost as if software and automobiles were completely different things!
These comparisons are pointless.

Reliability of hardware or software is a trade off. If your software or
hardware is responsible for keeping people alive then more time and money is
spent on ensuring its stability and that is then factored into the cost.

------
brudgers
The _Windows 10 Technical Preview_ is available to author as part of
Microsoft's quality assurance program. Technical Previews are alpha software
[1] by another name and Release Candidates are beta software in Microsoft
speak.

[1] see:
[http://www.hanselman.com/blog/IntroducingASPNETVNext.aspx](http://www.hanselman.com/blog/IntroducingASPNETVNext.aspx)
and
[http://blogs.msdn.com/b/cesardelatorre/archive/2014/05/12/th...](http://blogs.msdn.com/b/cesardelatorre/archive/2014/05/12/the-
future-of-net-in-the-server-asp-net-vnext-optimized-for-cloud-and-server-
workloads.aspx) for examples

~~~
AceJohnny2
Are you making the point that "it's beta/alpha, deal with it"? Because the
author adresses exactly that, and the headline incident is only a one
(egregious) example of a series of recent problems out of Microsoft.

~~~
brudgers
My point is that the software is equivalent to an alpha release.

People can build there own points based on the author's calling it beta
software.

------
mark-r
My own WTF moment came recently when we dropped XP support for our product.
Suddenly the window frames started painting improperly. One of the system
calls we use in NCPaint returns different values depending on whether the
executable is marked as XP compatible or not, even with no other change to the
program.

How are you supposed to create reliable software when you can't even count on
the raw API that's been unchanged since forever?

~~~
Retra
They go through a lot of effort to make sure their software is backward
compatible. (You're not supposed to make reliable software, you are supposed
to make broken software and convince your customers to pay Microsoft to make
it work on their systems.)

------
snarfy
I've been using Visual Studio since it was called Visual C++. VS 2013 is one
of, if not the buggiest versions I have used. The only responses I seem to see
are "It'll be fixed with Roslyn, just wait!"

I'm curious as to what has shifted inside Microsoft. Part of me believes I'm
seeing the work of new devs as the old devs have left. Another part believes
this is the result of agile taking over. I liked my visual studio with
waterfall releases. At least it worked without constant hangs, locks, and
crashes.

~~~
internetisthesh
Interesting, I use VS2013 with C++ and C# all days and rarely have any issues.
In my team with 10 devs the only guy who run into problems often is the one
who installs all extensions he can find (his VS looks like my mothers IE with
all the toolbars). Compared to how often VS6 crashed for me in the late 90s
and how buggy VS2003 was, VS2013 feels pretty solid to me. Guess we are not
using it in the same way or for the same things.

~~~
teh_klev
I agree, 2013 is pretty solid and feels a lot snappier compared to 2012 and
2010 (I have all three installed).

------
frik
I found it a bit irritating that there were no news about the various
Microsoft update issues on HN last week. The article at least mentioned the
various problems, that's good.

Some articles that capture the whole timeframe:

[http://goo.gl/eOaCX3](http://goo.gl/eOaCX3)

[http://goo.gl/c8PJ3W](http://goo.gl/c8PJ3W)

[http://goo.gl/PzxJPX](http://goo.gl/PzxJPX)

[http://goo.gl/ozN3zT](http://goo.gl/ozN3zT)

~~~
dang
> I found it a bit irritating that there were no news about the various
> Microsoft update issues on HN last week.

Why didn't you post one?

(Edit: I hope that didn't sound accusatory; I mean it as a cordial invitation
to post good stories that you think belong on HN.)

~~~
frik
I read about it only in german IT news portals like
[http://www.heise.de](http://www.heise.de) .

I really like HN, it replaces /. for me.

Therefor I try to not submit controversial posts (is it in US?). I assume it's
not optimal to submit Google Translate links of german articles (like my links
above).

~~~
dang
Yes, that's not optimal; the only exception is when there isn't yet a
substantive English version of a major story, or something particularly
interesting.

I don't think this particular story would have been unduly controversial, but
more importantly, thanks for your concern for the quality of stories on HN.

------
nutmeg
My favorite change lately is the one where they increased the version number
on System.Web.Mvc in the GAC, thereby breaking almost all old ASP.NET MVC
builds.

[http://blogs.msdn.com/b/webdev/archive/2014/10/16/microsoft-...](http://blogs.msdn.com/b/webdev/archive/2014/10/16/microsoft-
asp-net-mvc-security-update-broke-my-build.aspx)

The comments on this article have some great vitriol.

------
pauldbau
I think this about sums up the quality and obvious bent of this article:
[http://cdn.meme.am/instances/500x/57216937.jpg](http://cdn.meme.am/instances/500x/57216937.jpg)

------
jimbobimbo
Come on, the Preview is not even close to beta! That's why it's not pushed to
everybody, but offered to enthusiasts who are fine with things being broken
for a while. I mean, you can be upset all you want, but blaming an early
preview for being unpolished is reaching.

------
kitcar
As an aside but related - I bought a Surface Pro 3 and was shocked to discover
major issues with its WiFi when you have Hyper-V enabled. Basically WiFi
"breaks" whenever you wake the device, and can only be re-enabled through a
reboot. ([http://winsupersite.com/surface/will-microsoft-ever-fix-
surf...](http://winsupersite.com/surface/will-microsoft-ever-fix-surface-
pro-3-wi-fi-issues))

------
polskibus
That's the new Microsoft - less thinking about backwards compatibility and not
breaking things, but faster development, reduced management overhead. Faster
time to market HAS to come from somewhere.

------
dang
Since the HN guidelines ask for linkbait titles to be changed, we changed the
somewhat baity title to a representative sentence from the article.

~~~
debacle
dang, have you considered aggregating these kinds of notifications onto a
separate page, or maybe using a specific account for administrator action? It
would be helpful for those of us who value transparency (and thank you for
your continued transparency - it has been great).

~~~
dang
Yes, we consider that and similar ideas from time to time, but I worry about
unintended consequences.

~~~
towelguy
Just a plain log would do.

~~~
dang
It's the unintended consequences of a plain log that I worry about.

------
finid
> Microsoft has been making quality assurance mistakes lately

Well, that's news to me!

I've always thought that everything that came out of MS is a quality assurance
mistake. Heck, even MS itself is a quality assurance mistake.

------
maxpain
lately? since 1980s

