
Farming hard drives: 2 years and $1M later - uptown
http://blog.backblaze.com/2013/11/26/farming-hard-drives-2-years-and-1m-later/
======
tedivm
Why do they need hard drives when they don't even back up data?

Seriously, you can find reports of this all over the internet. It's on reddit,
twitter and elsewhere- these guys do not back files up properly, and their
systems are so shoddy that they'll report they have files even when they
don't.

My girlfriend brought her computer in for repair and they changed the hard
drive. She specifically checked before she left that the backups were set, and
the dashboard said they were. When we went to restore, however, none of the
files could be.

We contacted support and they kept insisting we did something wrong. Finally
they admitted that they had erased the files (yes, actually erased them). Due
to a "bug" their dashboard reported we still had the files, even though they
were gone.

They did everything they could to hide their mistake. They lied to people on
twitter about what I was saying (since the only way to get real support was to
hound their twitter account). They lied about all sorts of things. I had to
take screen shots of everything and post it online before they'd even admit to
screwing up. Even at that point their dashboard was still saying it had all of
these files, when it in fact had nothing.

Backblaze is a completely untrustworthy company. If they spent half as much
effort on backing up files as they do on marketing that may be a different
case. Anyone who uses them for backups might as well not have backups at all.

~~~
atYevP
Hey Tedivm! Yev from Backblaze here -> It's true that the data was removed
from your account. After your trial lapsed and was not paid for, we removed
the data after 7 days as is our standard practice. Unfortunately in your case
the overview page did not update to show that you had no data associated with
your computer, so when you purchased the license, there was no data on the
machine that it attached too, so when you erased data from the machine while
performing maintenance, you assumed the data was on Backblaze, when it was in
fact purged from our system a few weeks prior.

We've since fixed that visual glitch (if you have a trial that expires and is
not paid for, once the data has been deleted, that will be reflected on the
overview page) AND we're currently working on some other notification systems
to help people avoid losing data. As always, I apologize that you had this
problem, but I'm happy to inform you that we've since fixed that visual bug on
our end.

~~~
tedivm
So everyone else is clear, we did pay for the account after the trial ended.
So we were paying customers, this wasn't an incident where a trial expired and
we tried to get our data back.

Also, to be clear, the customer service experience was horrible. The support
team repeatedly ignored what we were saying and send us form responses back
which were just not relevant. The twitter team claimed we misread the
retention policy. The whole thing was kind of ridiculous.

Even more to the point, and the main reason why I tell this story, is that
people should know how this company operates. People should know that they
don't have backups of backups, that there is no margin for error should your
account lapse, and that their systems in general are not robustly programmed
in the sense that there are ways that your data could be lost and you wouldn't
know it.

From my perspective Backblaze only has one thing going for it over competing
companies, and that's it's marketing department.

~~~
TwoBit
If I understand it right, you purchased a license after your trial expired and
the backup was purged. And you didn't backup anything new upon getting the new
license. So the GUI status bug was a real and unfortunate bug on Backblaze's
side, but it was only a GUI bug and in fact they didn't lose your data in any
unexpected way. This situation seems to be about 2/3rds BackBlaze's fault and
1/3rd your fault.

You also have bonus fault on your side for not testing the BackBlaze backup.
Never ever trust a backup, even if it's right in front of you on your own
disk, unless you can successfully restore it.

~~~
tedivm
Yup, there were serious flaws in our backup plans here. In my defense, we
hadn't been dating long enough at the time for me to really get things set and
we were on our way to burning man ;-)

I am not claiming to be without fault, but I do think that Backblaze is on the
lower end of the reliability spectrum when it comes companies that do this,
which is why I bring up these issues. I've switched over to Crashplan and have
external harddrives doing local backups.

I should also point out that this isn't the only issue that's come up with
this company. Apparently their client side GUI is also not completely truthful
about when it's backed things up, and will claim to have backed up files when
it's really only backed up the indexes. It then spreads the real backups out
in order to increase performance. Although this is a cool feature, it's one
people should be aware of and not hidden. I have less knowledge of this
though, as I only saw it in a few blog posts (google around instead of taking
my word for it).

~~~
ghshephard
"Lower end of spectrum" \- of precisely two companies that I know of,
CrashPlan and BackBlaze, that do "Whole Disk Backups at a fixed rate".

I'm a big fan of using a combination of Arq (Love It), Dropbox, BackBlaze, and
SuperDuper, but - crashplan/backblaze are both pretty similar, and both have
their pros/cons, and, likewise, hundreds of anecdotes from the user community
of each.

For what it's worth, I'm in my fifth year of Backblaze, and I've always been
able to recover all my files - but I've also kept my subscription paid up.

And, I don't believe I've ever seen your "real backups happening spread over"
claim - I am frequently in the field, so I am _very_ aware of exactly when
backblaze is sending data (frequently over my iPhone using Roaming Data) - and
when backblaze says it's done backing up - that's pretty much exactly when
data is no longer sent out.

~~~
tedivm
Yev, the same dev who is commenting here, has admitted to the claims about
back up times. You should read through this reddit thread, it shows that
Backblaze does not do things as you'd think-

[http://www.reddit.com/r/YouShouldKnow/comments/1q14ca/ysk_th...](http://www.reddit.com/r/YouShouldKnow/comments/1q14ca/ysk_that_backblaze_may_not_be_backing_up_your/)

> If you boot up real quickly and want to do a scan one thing you can do is
> open up the control panel and hit "Alt" \+ "Restore Options", we'll do a
> small-file rescan immediately and schedule a large-file scan.

>Small is files under 30MB or so, it's a quick index, the larger files take a
bit longer and we try to spread those out over the course of a few hours so as
not to be too heavy on your system.

~~~
atYevP
Yes, that's accurate. The program was designed to be light on systems so we
are not constantly scanning and indexing your hard drives. In our best
practices we tell people to leave their machines on for a few hours. We cannot
run when the machine is off, so if the machine is constantly being turned on
and off it'll disrupt our scans. The "Alt + Restore Options" method forces a
scan, but you'll still need the machine to be turned on/connected to the
internet in order for us to transmit the data. Just pressing "backup now" will
start pushing the data that has already been queued for upload during a
previous scan (if there was anything left over), but will not necessarily
initiate a scan (unless there was nothing left over from the previous
uploads).

~~~
tedivm
So, to be perfectly clear, it is more than possible for you to sync the photos
from your computer, hit the "backup now" button, have Backblaze respond back
that a backup is complete, and then find out that all of those photos you
thought you had backed up were not there.

All of this to make it look like your application is performing better than it
actually is.

On top of that you keep doing that infuriating thing where you ignore what
people are saying and respond as if they're idiots or by trying to deflect the
issue. No one expects you to run when the computer is off, and pretending like
that's what's being said is downright insulting. What they do expect is that
hitting the "backup now" button, and having your application respond that
their files have been backed up, should actually mean that the files are
backed up.

The more you talk the more you try to twist this around into something it's
not- a visual glitch, misreading your policies, or people turning their
computers off when they shouldn't- rather than trying to understand why your
customers are upset and actually deal with it. That's why this issue has
grown, and why we keep having these conversations- you guys have zero concept
of responsibility and would rather insult your customers than actually make a
viable product.

~~~
atYevP
No, we should display the last time you were fully backed up in the console.
If you turn on your computer, move photos to it, and turn it off, even if you
press "backup now" likely they won't be included in the scan. In our best
practices and introductory email we say that if you want at least a "daily"
backup, you have to let us run for at least 3 hours so that we can fully index
your new/changed files and back them up. If you added a lot of files it'll
take longer, if you only add a few, it can be pretty quick.

If your complaint is that the Backup Now button does not do a full-system
scan, that is totally valid and after the reddit thread our engineering folks
are looking in to changing the "backup now" button's behavior. One of the
reasons it doesn't do that now is because a full-scan will hang your system,
where uploading the remaining files is very light and unnoticeable. Like with
any functionality decision it's tough to say what is the best answer, hanging
someone's system each time they press a button, or kicking off a remaining
files upload and gradually scanning the drive over the next hour.

I sincerely apologize if I am not communicating well though, I am not trying
to talk down or assume you or anyone else is an idiot in any way. I've tried
to address everything that you bring up on here and on reddit. As far as
expecting Backblaze to run when the computer is turned off, you'd be surprised
at how many of our support tickets ask, "If my computer is off, are you still
working?" so we do see that quite a bit. I don't bring it up to try and dig at
anybody. I also think we've taken responsibility for the bug that mislead you
in to believing you had data on our system when it had already been removed.
Once we realized what had happened in your case we offered a refund and have
since fixed it so that it does not happen to anyone else.

We're in the business of backing up data. When customers lose data, whether
it's something they did or something that occurred on our end, we feel badly
about it and try to make it right. We do have a viable product. We've restored
over 4 billion files for the customers that have accounts with us. We take it
very seriously.

------
carlmcqueen
I've been trying to reduce my data hoarding. I had a larger back-up hard drive
go down and take a lot of data with it. I felt sick knowing I potentially lost
years of funny images, videos, music, writings, etc.

What I found is I had back ups of back ups of the things I actually cared
about and the things I lost I hadn't truly looked at in a long time. A lot of
the music I hadn't listened to in years.

I have found that I can fit on small SSD drives and as a result of the things
I do wish I hadn't lost, I'm just keeping up with the practice of keeping
multiple back ups.

TL:DR: I have made an effort to keep a smaller data footprint instead of
hoarding.

~~~
alexkus
If the only instance of a thing is on your "back-up" drive then it isn't a
back-up.

~~~
mikeash
Data that isn't backed up (as in a second copy) doesn't really exist.

Likewise: data that isn't backed up off site isn't really backed up.

~~~
brazzy
> Data that isn't backed up (as in a second copy) doesn't really exist.

An IMO more realistic statement: Data that is not backed up is implicitly
classified as "nice to have" \- in many cases a perfectly rational choice, if
done consciously.

> Likewise: data that isn't backed up off site isn't really backed up.

You may as well argue that no data is ever "really" backed up because there is
always the possibility of simultaneous failure. It's always a choice of how
much risk you're willing to accept vs. how much money and effort it would cost
to reduce it further.

~~~
mikeash
Yep. 1 copy of data is not enough, because it can be lost. Now, assuming that
N copies of data is not enough, it's easy to see that N+1 copies is also not
enough, because that extra copy can be lost, thus reducing to N copies, which
is not enough.

This proves by induction that no matter how many backups you have, it's not
enough.

------
ISL
The original "Drive Farming" link is at least as interesting as the new
article:

[http://blog.backblaze.com/2012/10/09/backblaze_drive_farming...](http://blog.backblaze.com/2012/10/09/backblaze_drive_farming/)

~~~
SloopJon
And here's the discussion of that story:

[https://news.ycombinator.com/item?id=4631027](https://news.ycombinator.com/item?id=4631027)

I browsed the comments with two things in mind: was anyone uncomfortable with
shucking and farming on this scale, and were the drives of equivalent quality?

Regarding the latter, user rsync of rsync.net claimed that "the drives inside
these enclosures are the worst spec'd, highest failure rate drives." User
budmang of Backblaze, however, said, "The drives were literally the same
inside the external enclosures." User atYevP of Backblaze posted a similar
comment.

~~~
jotm
High failure rates of external drives can also be attributed to the USB
controllers - they're better than their IDE predecessors (which failed a lot,
sometimes taking the drives with them - only low level utilities could revive
them). Otherwise, I don't see how or why the hard drive manufacturers would
bother to send the worst batches (which aren't _that_ much worse) to external
hard drive assemblers.

~~~
ZoFreX
I repair computers for friends and families and the majority of the time I get
a broken external to look at, it's the enclosure that's at fault and the drive
is absolutely fine.

I encourage people to buy their enclosures and drives separately - it's a
little more expensive, but this way you can RMA just the enclosure if it
fails. With an all-in-one, you either void your warranty to get the data back,
or ship the whole unit back for a replacement.

~~~
jotm
That's an interesting point, since I have a Samsung hard drive I took OUT of
an enclosure and out of curiosity I checked the warranty on it - it's a 3 year
warranty that is still active and the web page said I could initiate a return.

So, technically, I could RMA the drive myself if it fails... or would they
notice it and not process the return?

------
EGreg
Sadly I read this as a precursor to shortages in water, fish and
fruits/vegetables that are coming. The populations of monarch butterflies,
bees and other insects are dropping like a stone year over year. The fish near
asia's costs are disappearing. The culprits include pesticides, overfarming,
overfishing, and pollution. I am not even as worried about the garbage anymore
-- scandinavian countries have figured out ways to burn the garbage and
"contain" the waste gases. However, the main problem of capitalism is the
numerous externalities that are exploited until it's too late to fix. It's not
just global warming. It's raping the ecosystems for profit. No futures are
going to save us from that. We need sustainable practices but I really don't
see a way that'll happen at this point.

~~~
TrainedMonkey
Too fatalistic. In 19th century they predicted that Earth can't sustain
population over 1 billion. In comes oil and agricultural revolution. I agree
what we are doing with ecosystem in general is a crime, but it won't kill us.
At least not anytime soon.

~~~
EGreg
Just because advances were able to stave off collapse and allow billions of
people to consume resources at an ever greater rate does not mean that the
same can be done with the effects of overfarming, overfishing, and pollution.
The fallacy is thinking "since we did it once we can always continue doing
it." The probability exponentially rises that you can't, since this is
basically the same as the probability of always succeeding in time to avert a
disaster.

Actually garbage / messing up ecosystems is a different type of problem than
obtaining resources. It is somewhat related to the second law of
thermodynamics. Once you introduce so much entropy into a system the genie is
out of the bottle.of course this isn't exactly thermodynamics but I can say it
this way -- putting humpty dumpty together again is different than figuring
out how to make humpty give you more food for decades.

~~~
TrainedMonkey
I really agree with you on the point that we are doing all those things. I am
a fan of the [http://physics.ucsd.edu/do-the-math/post-
index/](http://physics.ucsd.edu/do-the-math/post-index/) blog.

What am I arguing for - that while we do consume more resources, progress is
exponential as well. For example if we seriously begin to run out of fossil
fuels, amount of effort that is put into other energy sources would increase
by order of magnitude and solution will be found. It might not be right away,
it might not be silver bullet, but when push comes to shove and there are no
alternatives, human ingenuity is frightening.

In the end progress stacks multiplicatively - you add great battery technology
to great energy generation technology and you got more than sum of the parts.
So more people there are doing research - faster it goes _.

_ This point is only true from certain point of view. Consider genius starting
from nill, how far could he take technology within his lifetime? Electricity,
probably. Microprocessors and genetics, probably not. Now consider million of
people working, each discovery builds on an earlier one, each tool makes other
people researching more efficient or enables them to make new discoveries. Now
think about multiple generations of people building discoveries on top of each
other while sharing information.

------
post_break
Why didn't they touch on why the Hitachi drives weren't the same quality as
the other brands with this statement:

"Switching hard drive manufacturers from Hitachi to Seagate and Western
Digital. The assumption here is that all drives are the same and as we’ve
learned that’s not really the case."

Where is the footnote. Why aren't they the same?

~~~
ewams
In the parent article they said they switched because of cost: "When Hitachi
drives got too expensive we switched to Seagate and Western Digital. When
internal drives got too expensive we turned to external drives. When hard
drive availability became an issue we turned to Drive Farming."

From their article on v2 they said: "The Western Digital and Seagate
equivalents we tested saw much higher rates of popping out of RAID arrays and
drive failure. Even the Western Digital Enterprise Hard Drives had the same
high failure rates. The Hitachi drives, on the other hand, perform
wonderfully." [http://blog.backblaze.com/2011/07/20/petabytes-on-a-
budget-v...](http://blog.backblaze.com/2011/07/20/petabytes-on-a-
budget-v2-0revealing-more-secrets/)

There are other articles where they mention that they do testing and some
brands perform better than others.

~~~
sard420
We used to make PODs like this at work for netbackup storage.. started with WD
drives ~128 WD drives, at about two years we were at ~8 WD drives still
working. At least once a week we had people in there swapping drives. I have
yet to see a WD not fail.

~~~
dublinben
On the other hand, I've got a WD Caviar Black that's been running strong for
more than five years. None of my WD drives have ever failed (knock on wood!).
I don't run 100+ drives though, so I'm probably just hiding in the statistical
margin.

~~~
dspillett
Also your thermal environment is going to be quite different, which could be a
significant factor. How often do you power-cycle that drive BTW? I've had more
drives go bad on power-up (just failing to spin up and move the heads
correctly) than through surface errors and such.

~~~
dublinben
I try not to restart that machine more than once a month.

~~~
dspillett
That si better than running the drives 24/7 - if one is going to go then that
way it is more likely to go alone. If every in an array (anything I even half
way care about get s the RAID1 or RAID5 treatment) has been spinning for years
then a power cycle can be quite scary. Restarting once month or so increases
the chance that if they fail they'll not all fail at the same time.

Having said that, my main little server at home hasn't had a power cycle for
somewhat longer than a month as there have been no kernel updates or that like
that have required it...

------
michaelt
The extrapolations in the article were pretty optimistic to begin with.

I mean, when you look at the dotted trendline in this graph:
[http://blog.backblaze.com/wp-content/uploads/2013/11/blog-
co...](http://blog.backblaze.com/wp-content/uploads/2013/11/blog-cost-
trend-1.jpg) you'd think by 2016 drives would cost $0 per gigabyte and by
2020, drive companies would be paying you to take drives off them.

~~~
pedantry
Well, at that point, if the current graph charts cost per gigabyte and
approaches zero, then you would change the scale of the graph to show costs in
terms of terabytes and eventually petabytes.

Ultimately, after traversing enough benchmarks, the scale would go
logarithmic, and measurements would have to take orders of magnitude into
account.

~~~
redacted
Indeed, if you click through to the article the historical data is drawn from,
you will see it plotted on a semilog plot. The trend is well described by an
exponential function.

------
recuter
Aren't they big enough at this point to cut deals directly with the vendors?
10,000 drives at a discount, that sort of thing.

~~~
SloopJon
In the previous story, user brianwski of Backblaze said, "We have been told
that until we buy blocks of 10,000 drives, we cannot deal directly with the
drive manufacturers. We're not there yet."

[https://news.ycombinator.com/item?id=4633213](https://news.ycombinator.com/item?id=4633213)

~~~
grecy
Cool.

I know a few Backblaze people are reading/posting in these comments, so I'll
ask here:

Do you guys know the approximate price of when you _can_ purchase them direct
in 10,000 blocks?

~~~
ChuckMcM
I'm not at Backblaze, but when I was at Google I did know what Google was
paying for drives direct from the manufacturer and it was not as much lower
than the distributor price as you might expect. Margins are pretty thin in the
disk drive business.

------
ghshephard
I have a colleague who works in the industry and purchases a few petabytes of
spinning disk a quarter. I love reading the backblaze blogs, and constantly
forward them to him. He thinks the backblaze guys are pretty crazy with their
"shucking strategy", finds it hard to believe seagate/WD guys will honor
warranties on those "shucked" drives, but if they will, all the more power to
backblaze.

He does note that his costs per drive didn't go up dramatically after the
Thailand floods, and they were pretty close to what backblaze was playing for
their "shucked drives" \- though, they didn't go _down_ signficantly either.

I'm guessing Backblaze is still at the volume levels where they can't
negotiate directly with Drive Vendors - I'm not sure where that level is - 10
Petabbytes/Quarter? 100 Petabytes/quarter?

Regardless, I continue to be very impressed by how backblaze manages to
circumvent what others might consider insurmountable hurdles. Truly hacking
the system in a good way.

~~~
atYevP
Yev w/ Backblaze here -> Yea unfortunately when we shuck the drives the
warranties are voided, but we add that factor in to the costs that we are
willing to pay for the hard drives. Truth was, it was cheaper for us to buy
the externals and shuck them, even with the voided warranty, than pay the
larger fees for internal drives.

Part of the is price sensitivity. External drives are geared towards
consumers, who simply will not buy the drives if they go up too much in cost.
The internal drives we were buying are targeted at business' who have a choice
to make, either buy at the higher price, go out of business, or get creative.
We try to go the go creative route because we really don't want to go out of
business, and paying the higher prices for drives, especially in this case,
would have gotten us there.

As for the negotiation, BrianWski mentioned it in another reply on here, but
we keep getting weird answers from vendors and manufacturers. Typically they
want a minimum order of about 10,000 hard drives in order to work directly
with them and we're simply not willing to buy that many at a time, especially
since normally the price goes down monthly, meaning carrying inventory is
leaving money on the table.

Love that you enjoy your blog posts, stay tuned ;-)

------
pcunite
I personally (for home use) held off buying drives for two years, found out I
did not really need so much space and discovered SSDs would do the trick.

~~~
dublinben
Same here. I bought my last 2 TB for $80 shortly before the crisis, and was
able to scrape by at 90% capacity until 4 TB drives became cheap enough.

------
pwg
> When the Drive Crisis started, industry pundits estimated that the hard
> drive market would take anywhere from 3 months to 1 year to recover. No one
> guessed two years. Was the delay simply an issue in rebuilding and/or
> relocating the manufacturing and assembly facilities?

This is a typical response in all markets that are commodities (or approaching
commodity status). Once makers have had an excuse to move prices up, they want
to hold onto the extra profit for as long as they can, so prices are fast to
rise, but slow to fall.

The most direct effect can be witnessed in gasoline prices. Notice how a news
report that a camel farted in the desert of Saudi Arabia results in the price
of gasoline at your local station rising by 25 cents a gallon (or more) nearly
instantly. Even though the gas in the underground tanks was the same gas (at
the same cost to the station) that it was 15 minutes ago.

But then, it takes weeks before that 25 cent increase goes away when no more
camel farts occur over those same weeks. Same effect here, just with hard
drives rather than gasoline.

~~~
brianwski
Six months after the Thailand drive crisis, drive prices "peaked" at about
double their earlier prices and began falling. Just about then, an article
came out featuring a bunch of panic inducing quotes from a Seagate VP loudly
claiming it might be a year before they could get enough drives made,
shortages would be terrible. It felt so disgusting to me, none of it was true,
the VP was just trying to encourage pre-buying and drive hoarding to
artificially pump sales.

------
shmulkey18
I wonder why external drives are cheaper than internal drives. One would think
that they would be more expensive, since the cost to the manufacturer of an
external drive includes not only the drive itself but the adapter contained in
the enclosure, the enclosure itself, cables, possibly a power supply, more
elaborate packaging and the additional labor required to assemble the external
drive. I assume that this consumer cost differential arises from differences
in the markets for internal and external drives, but naively it seems like the
internal drive market would be more price-sensitive and thus lower margin.

Does anyone know why external drives are typically cheaper than internal
drives?

~~~
mangotree
Maybe, with the advent of laptops becoming more popular than desktops, there
is just a bigger market for them. Cost is usually due to demand constraints.

~~~
sliverstorm
I doubt it. Server farms consume an unreal quantity of drives. Desktops
consume 1 drive per person.

~~~
Retric
Server farms store data for actual people. So sit back and think about how
much unique data you have on servers somewhere. Add it up and I dont't think
the total data storage at all data centers everywhere in the US add up to more
that 20GB per person in the US as we are just not that interesting even at 5x
redundancy your still at 1/30th a drive per person. Don't forget at 300
million people that would still be 10 Million drives which is a lot of
servers.

PS: The NSA could be recording every phone call ever made and it's still not
all that much actual data.

~~~
sliverstorm
The dropbox model is not the only place for server farms.

How about render farms & their associated data stores. How about Google Maps &
the associated scads of satellite imagery.

You could be right, of course, but I'm not sure we can say with certainty.

~~~
Retric
Google Maps stores a 'tiny' amount of data per person. The surface area of the
earth is a little less than 200million square miles or a little less than 2/3
of a square mile a person. at 1 foot accuracy that would only be two 10
megapixel pictures per person not that we actually have maps anywhere near
that good of most small towns let alone the oceans or arctic. The only thing
that really spit's out data is raw scientific data, but even Nasa just dumps
well over 99% of what they collect so it's more of a short term problem.

------
sliverstorm
I don't know a whole lot about backblaze; as they are a backup service, is
there a good reason why they couldn't ramp up usage of tape drives? Perhaps
keeping recently uploaded backups on-disk, and shuttling older ones to tape?

~~~
smackfu
Their backups are progressive, so a single disk image might be spread across
100 tapes. A restore would be a nightmare. Tapes really only work if you are
doing a full image each time.

~~~
JoeAltmaier
or a program of partial backups interleaved with full backups, say 3-to-1. You
balance a reasonable restore time with a reasonable storage requirement.

------
alayne
This seems like something that should be part of their business instead of
waiting for catastrophes.

~~~
dublinben
Since the price of storage is consistently declining, buying any more than
necessary is wasteful over the long term.

~~~
clarky07
yeah, but it seems like you'd give yourself more than a few weeks of runway.
Don't have to buy 5 years worth, but perhaps 3-6 months would be reasonable.
The peace of mind has got to be worth the ever so slight cost increase.

------
digitalzombie
Still... no linux.

