
We called it RAID because it kills bugs dead - soheilpro
https://devblogs.microsoft.com/oldnewthing/20200317-00/?p=103566
======
chadd
I left MSFT and went to Viair (with folks from the MSN Mobile and Outlook
Express team), and we didn't have a bug tracking system... so we had someone
send us a few screenshots of RAID and built a VisualJ clone one weekend. Fun
side project, didn't really want to be in the business of bug tracking, was
slightly before the web app era... We called our bug tracking system
"BlackFlag" because it was a competing brand of bug spray. (And obviously we
used the logo from the punk band...)

~~~
OakNinja
Fun fact, one of Sweden’s most influential music- and culture journalists was
being interviewed on national television about a weird documentary wearing a
pink “Pink Flag” t-shirt with a modified Black Flag logo. Beyond meta.

~~~
brigandish
Is that a clever way of referring to Black Flag at the same time as Wire's
Pink Flag?

~~~
OakNinja
That’s how I interpret it. I actually asked him on Twitter an hour ago - I’ll
keep you posted :)

------
zestyping
The bug puns were everywhere!

When I started working at Industrial Light and Magic in 1998, there was no bug
tracker in regular use. I ended up building a bug tracker in Python called
"Roundup" (inspired by a program by Alan Trombla, who came up with the name).
To create the thinnest veneer of plausible deniability in case the bug spray
lawyers came after us, the icon was a little spinning GIF animation of a
cowboy on a horse, lasso curled in the air.

I only stayed at ILM for two years, but Roundup remained still in use as the
ILM software department's main issue tracking system long after I left, at
least 6 years later. It was one of the early issue tracking systems that made
it easy to subscribe to any issue, so you'd get e-mail notifications whenever
the issue was updated.

The issue identifiers were numbers prefixed with a code to indicate the
affected software project, like "REN003" for a bug in the rendering system.
Three digits, of course, because I could hardly imagine that a single project
could ever have more than a thousand bugs!

I remember that along with statuses like "in progress" and "done", we had a
status I've never seen anywhere else: "cbb". As in "done, but could be
better." What optimistic software engineers we were back then!

~~~
rconti
CBB reminds me of the pieces of hardware I've labeled as "maybe bad".

Only to run across 2 years later, and find myself wondering "who was this
'future me' that I envisioned might be interested in an ethernet card that
exhibits flaky behavior under high load when bonded?"

~~~
Anthony-G
Been there and learned that lesson! Let's waste _future me_ 's time by having
them figure out in what way the hardware might be bad.

------
jkcorrea
Fun fact: Apple's internal bug tracking tool, Radar, traces its origins to a
pen-and-paper tracking system from the very early days.

In a similar fashion to RAID from the article, it's use expanded way beyond
the original use case and is now used by nearly every team at Apple, from
hardware, to test, to marketing, retail, and everything in between.

It's often the butt of a lot of jokes as most people hate its dated and
confusing UI :).

edit: Though to be fair, the team maintaining it has made massive strides
recently to completely revamp the UX which is no small feat given such a wide
range of business critical use cases.

~~~
saagarjha
As far as I can tell, most teams think Radar is the best bug tracking tool
ever. Externally, it's extremely frustrating to interact with, even more so
through Feedback Assistant.

~~~
jkcorrea
Actually you bring up a good point - the reactions are perhaps more bimodal
than anything. If you're on one of the teams it was designed for, it is indeed
near perfect. If not, it's far from it.

~~~
lilyball
The UI was old and crufty and it was a pain in many ways to work with. But I
really miss it because it’s still better than any other issue tracking system
I’ve ever used.

~~~
PeterStuer
Can you elaborate on what made/makes it better?

------
lifeisstillgood
>>> they might have been too scared to write it. When looking back on the
origin of RAID, one of the original developers confessed, “It really wasn’t
made to last that long. Sorry!”

This speaks to me - too many times we think we need clever architecture and
well thought out methodologies and ... the most useful tools are a mix of
inspiration and timing and luck.

Had the developers stopped to plan they would have built something with over
16 bits of primary key - and also made a hundred other "improvements" that
would have destroyed all the value MS got out of it.

There is a point in this - it I am not sure I know what it is.

~~~
hermitdev
> Had the developers stopped to plan they would have built something with over
> 16 bits of primary key

While true, this was in the 16-bit, Windows 1.x era. Also the ~640kb of RAM
era (disks also werent that big either, DOS (v3.x) around that time only
supported 32 MB partitions). Not sure which version of DOS Windows 1.0 ran on,
I didnt use windows until v3.11 for Workgroups. A 32-bit primary key would
likely have been seen as wasteful (not that I'm excusing it) and impractical,
because you could likely never practically store more than 64k records on a PC
of the day (and even if you could, you wouldnt have an disk left to do
anything else).

~~~
gdwatson
He didn’t say which platform it ran on. Was Microsoft still using Xenix
internally at the time?

~~~
exikyut
Yes, for a time Xenix was used for some proportion of internal development,
IIRC for DOS.

I think I read that on here somewhere. I unfortunately don't have a citation.
Need to get a time machine to take me to whatever point in the future I sort
out my bookmarks ._.

------
m0zg
Google's counterpart is called Buganizer. I've seen bugs in it dating back a
decade or more. I don't know what, if anything, was in place before Buganizer.
They might be exposing some version of it externally as well now:
[https://issuetracker.google.com/](https://issuetracker.google.com/), although
the UI bears little resemblance with the internal version I used years ago, so
it's probably a separate product.

What Raymond's article does not mention (or at least I don't think he does, I
have only skimmed the article) is that at MS each team had _its own_ bug
database. As a result, even filing a bug in another product was an ordeal, let
alone fixing bugs across products. What this friction means in practice is
nobody would bother unless they really couldn't live without a fix. I think
you can easily see the results of that in Microsoft products today.

At Google, _everyone_ uses the same bug DB. If a BigTable bug affects
something in Ads, you can create a bug for BigTable and reference it in a bug
under Ads, which is, IMO, the way it should be. Moreover, you don't need to
get bug database permissions for BigTable, because you already have them.

~~~
RandallBrown
I remember when I first started at Microsoft, I found some bug in some product
and thought "I should file a bug with them"

Turns out there was no easy way to do that. I ended up emailing some person
from the team I found in the GAL (I think that's what they called it?) and
they weren't helpful or appreciative.

Never did that again.

~~~
m0zg
I don't know how it is now, but you've just described every single well-
meaning starry eyed fresh Microsoft employee in the 00s and likely before. The
fact that management did not see how fucked up this was is a condemnation of
the Microsoft company culture of that period. Again, don't know how it is now.
I turned in my blued badge well over a decade ago.

------
benschulz
I'll take any opportunity to mention John Browne's "The Bug Count Also
Rises"[1], which contains the line "The RAID is huge with bugs", referring to
Microsoft's bug tracker. It was previously submitted to HN[2].

[1]:
[http://www.workpump.com/bugcount/bugcount.html](http://www.workpump.com/bugcount/bugcount.html)
[2]:
[https://news.ycombinator.com/item?id=2740452](https://news.ycombinator.com/item?id=2740452)

~~~
dmix
What a poetic (and a little depressing-in-a-good-way) comment. I remember
various times of my career being a glorified bug tracker. It's a bad mode to
get trapped in with no forward progress.

[https://news.ycombinator.com/item?id=2740626](https://news.ycombinator.com/item?id=2740626)

------
shepwalker
Oh lord, product studio. Back in 2013 the preferred method of filing workitems
for a given "wave" was to load this spreadsheet, do weiiird formatting, and
indentation, type up all the workitems you foresee for the next sprint and
then press a magic "send to product studio" button." God help you if you
messed up some of the nesting or wanted to reorder anything - honestly I would
just wipe it clean and start from scratch.

Ironically my team just finished a product called RAID that was designed to
kill bugs in product studio and move them to DevOps. There are still pockets
of teams that use Product Studio within Microsoft - the sufficiency gap is
real - but they're the overwhelming minority now.

~~~
asveikau
I had absolutely zero problems with product studio when I used it from
2008-2011. It was a very 1990s style MDI application, somewhat representing
the peak of win32 ux design, short and to the point, and as long as the
servers were happy it "just worked".

------
dmix
> Mind you, that doesn’t mean that things have been stable, because the name
> of the service changed from Visual Studio Online to Visual Studio Team
> Services, and then again to Azure DevOps Services.

That's the most Microsofty sentence ever. It summarizes their allegiance to
ever changing marketing goals over simplicity and consistency. Even for
something like a bug tracker.

~~~
Avalaxy
Before it was called Visual Studio Online it was called Team Foundation
Server.

To make things even more confusing, there is now another product called Visual
Studio Online, which is a completely different thing (it's a cloud IDE)
[https://visualstudio.microsoft.com/services/visual-studio-
on...](https://visualstudio.microsoft.com/services/visual-studio-online/).

------
smacktoward
The one thing all long-lived systems have in common: nobody ever intended them
to be long-lived systems.

~~~
jedberg
The quick and dirty Perl deployment system I built at reddit lasted nine
years. Five years after I left.

I definitely never intend it to last that long.

~~~
smacktoward
My father wrote a dBase application _forty years ago_ that, to his amazement,
is still in use in some places today.

Inertia is a hell of a force.

~~~
twhitmore
It probably has more digits for the primary key than the MS bug tracker
originally mentioned. Sigh.

------
ikeellis
As an independent dev team working with Microsoft Games in 2002-2003, we used
RAID to ship Rise of Nations. It was great! Rockin' fast compared with any web
tool now and batch operations were near instant. Sadly it was utterly insecure
for remote use and it went away after that game.

~~~
Infernal
I loved Rise of Nations! Mind sharing which bits you worked on?

~~~
ikeellis
Single player campaigns

------
philiplu
I started at Microsoft in 1991, and remember RAID fondly. My favorite part
back then (probably running on Windows for Workgroups): when you minimized the
program back to its desktop icon, the icon animated for a few seconds, showing
a can of bug spray firing on a small flying bug, which dropped to the ground.

~~~
lejohnny
For green-field projects (new projects/apps), you always had bug #1: "[Insert
project codename] doesn't work. Severity: 1. Priority: 1."

Always interesting to read some of the bug reports; some of which could
meander off-topic in interesting ways.

------
bentcorner
Sometimes I think bug tracking doesn't get enough attention. It's gotten a lot
better these days but compared to source control, bug tracking seems to remain
the red-headed step-child.

Bugs can be just as important as source code - if a line of code is changed
and all the comment says is "bug 1234", then hopefully bug 1234 has context
about the scenario that was broken, a history of a conversation between QA and
development, and signoff from QA with whatever was verified.

Sure, this might fall into "write a good commit message", but obviously that
doesn't always happen, and having the historic knowledge of the bug tracking
database at your fingertips is important.

~~~
hyperrail
Fossil ([https://www.fossil-scm.org](https://www.fossil-scm.org) ) has an
interesting approach to this problem - integrate bug and work item tracking,
project wikis, and source control, but make them _all_ distributed, so that
there is not a single master copy of the bug database or the docs wiki.

It attempts to solve both the cultural challenge (bugbase written by a
different company/team from the source control, imposing 2 different styles on
you) and the technical one (bugbase and source control don't talk to each
other, and so bugs and checkins are not linked).

~~~
int_19h
If only Fossil weren't so opinionated... I love the idea of bundling
everything in the repo, and not averse to SQLite database as the storage
format. But there's no way to edit history, even for commits that don't exist
anywhere other than your own local repo - because the author considers rebase
an anti-pattern.

------
vector_spaces
The late Joe Armstrong described the similarly minimal Erlang issue tracking
system in this excellent post about MVPs

[https://joearms.github.io/published/2014-06-25-minimal-
viabl...](https://joearms.github.io/published/2014-06-25-minimal-viable-
program.html)

------
barrkel
Borland's bug tracking tool for Delphi was also called RAID.

It used the disconnected data set and resync technology that was part of
Delphi itself, which was handy for remote working before always-on internet.

~~~
lt
I wonder if Borland's RAID was inspired on Microsoft's, or just the same pun.

------
badsectoracula
> Office cloned the source and built their own custom version

I found this bit amusing considering that (IIRC) the Office team also had
their own C compiler :-P

------
tracker1
It's often the simple things, not meant to last long, but work well that live
the longest.

------
bearer_token
Who wants to celebrate the 18th anniversary of the oldest open Jira ticket
with me?
[https://jira.atlassian.com/browse/JRASERVER-161](https://jira.atlassian.com/browse/JRASERVER-161)

------
m4rtink
And if you have disks in RAID a bug in it kills your data instead. ;-)

~~~
glangdale
I was startled that they used the name RAID given the Berkeley RAID paper...
but on checking the dates, they would have needed a time machine to realize
that this term was about to become heavily overloaded.

------
DannyB2
Isn't RAID a way of storing all of the zero bits on one drive, and all the one
bits on the other drive?

That makes replacement of a bad drive much easier.

------
khy
It's a weird, nice feeling watching your software in use well beyond it's
planned - or more frequently, assumed - obsolescence.

~~~
hermitdev
What ammuses me is looking at some of the hard-coded "end of time" dates at
previous employers. Instead of using null as an unbounded upper bound (because
nulls are hard /s), there are hard coded "end dates". One was 2032-12-31,
another 2037-12-31 and usually 9999-12-31. Obviously, these all have problems,
especially when used in finance. 2037 makes sense, because 2037 is the year
that 32bit linux overflows. 9999 also makes sense. At least none of us will be
around, so it's some future generation(s) problem. The 2032 one made no sense
to me, and I was never able to get a justification.

~~~
kaibee
Now that it's 2020, I've been looking for a new year to think of as "the
future". 2040 seems too obvious and maybe too far away to really imagine, what
just add 20 years? Boring. I think I'll take 2032.

~~~
Paul-ish
2049, for the year the blade runner sequel is set in.

------
martijn_himself
I'm sure RAID was much more usable than some of the ITIL-inspired
monstrosities some of us have to deal with.

------
abiogenesis
> Unfortunately, the archive project renumbers the work items. Fortunately,
> the original work item is remembered in the title, so you can do a search
> for originalid:3141 to find the old work item known as number 3141.

I wonder what happened if a bug was carried over for 2 or more databases.

~~~
lilyball
Presumably the archive projects are read-only and therefore once a bug goes
into the archive project it never moves again.

------
macjohnmcc
I remember using RAID when I worked at Microsoft from 1993-1996

------
jdsully
Product Studio, RAID's replacement was great. I really didn't like the
transition to TFS.

------
SlowRobotAhead
> The database was written back in the days of 16-bit computing, so naturally
> it had a limit of 32,767 bugs

I know it’s a nitpick; but no, that is not natural to me at all. You can’t
have a negative bug or would store it in list with negative indices. So using
signed makes no sense to me.

Sometimes I get really confused why signed shows up in places it doesn’t
belong.

~~~
dragonwriter
> Sometimes I get really confused why signed shows up in places it doesn’t
> belong.

Because in the absence of convenient sum types, you end up using a simple type
with sentinel/special values so often that anything where the main value is
logically non-negative (or strictly positive) but you don't have a clear
reason to never need special values is done with a signed type where negative
(and possibly zero) values are reserved or assigned to special uses.

------
sitkack
I loved RAID. Lightning fast, so fast that speed was its own quality.

~~~
sitkack
Of all my comments, this is oddest thing to downvote! Is RAID bad now?

~~~
marvy
seek not to comprehend the downvoters, for it is written: it is simpler to
predict each gust of wind in a storm, than to explain even a single downvote

or something

~~~
sitkack
Thanks marvy, this made me smile.

~~~
marvy
:)

------
naringas
it killed them dead by means of tracking and keeping them in an inventory...

------
grillvogel
im a CMVC man myself

------
twhitmore
Is it just me, or does this seem terrible? Lack of a stable business key &
limiting key size to 4.5 digits just seems terribly lacking in any competent
database practice.

I was building ISAM database applications for small business in the early
nineties, and we used 6 digit keys.. and an index to access records by.

I'm sure Raymond Chen is a nice guy, but this just makes me think of a retard
stabbing his cheek when trying to eat with a fork. (Dirty Rotten Scoundrels
movie.)

~~~
dang
Yikes, please don't post like that last bit here. I get that you probably
meant it lightheartedly, but it does not come across well.

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

------
KasianFranks
No you did not.

