
“Let's talk about a hypothetical public-facing service” - kgm
https://www.reddit.com/r/sysadmin/comments/3do9k0/sourceforge_is_down_due_to_storage_problems_no_eta/ct77o49
======
smacktoward
_> You have a moment where you envision the future of virtualized storage and
think about how great it will be when storage is nearly free and outsourceable
and you can stop buying disks from Amazon every few months_

The future is now! Instead of you sending money to Amazon and them sending you
disks, they keep the disks and you send them the money anyway.

Progress! :-D

~~~
ThePhysicist
Sadly, depending on what you want to accomplish with your storage the future
is still not here. For example, if you want to share a virtual disk with
multiple servers through NFS you will still need to manually create a NFS
high-availability cluster using e.g. DRBD on most cloud providers (e.g. EC2 or
Azure). To be fair, MS Azure offers fully managed SMB shares as a service, but
the transfer speed is quite low (60 MB/s in the best case) so they are not
very useful for most tasks.

~~~
mrud
AWS has EFS [1] which should allow you to share one FS to multiple serves. It
is still in preview but could help address these needs.

[1] [http://aws.amazon.com/efs/pricing/](http://aws.amazon.com/efs/pricing/)

~~~
ThePhysicist
That looks interesting, thanks for the link!

------
rosser
_...Extremely Massive Corporation..._

I knew EMC storage was utter shit when, upon attempting to create a new RAID
group, I realized that the configuration tool's default was to stripe across
drives _within a shelf_ , not to create stripes that _span shelves_.

Worse, to create the more fault-tolerant, shelf-spanning RAID volumes, one
must manually add drives, one by one to the array, in a process that involves
about 44 (slight hyperbole) clicks per disk.

And then there was the fact that the configuration tool was Windows-only.

Yeah, screw those guys.

~~~
angry_octet
Actually it sounds like the EMC software was trying to save you from something
which is a bad practice. If you want raid 1+0 it is better to have two disks
on the same controller/chassis in raid 0, then raid 1 (or raid 5/6) across
chassis. Otherwise a chassis failure will take fail both leaves of your raid
mirror.

Having said that, EMC is stupidly overpriced bloat.

~~~
rosser
Did I really need to spell out the geometry of the RAID I was creating to
avoid this kind of nitpicky follow-up?

It was a 20-something disk RAID 10 [1], arranged so that every mirrored pair
of disks spanned different enclosures, in order to mitigate the failure of any
one shelf — that is, interleaving mirrors across controllers and shelves,
exactly as you suggest I should have done — and further, such that any one
shelf failing only affects the mirrors that had disks on that shelf.

EMC's software wanted to allocate the drives from two shelves, with an unequal
number of drives per shelf. It was just grabbing the next however many disks,
linearly.

So, no, they weren't trying to balance the mirror across enclosures or
controllers. They just weren't thinking.

[1] By "RAID 10", I mean "striped mirrors" — that is, create a bunch of
mirrors that span shelves and then stripe across them — not "mirrored stripes"
which is what you appear to be suggesting, with "it is better to have two
disks on the same controller/chassis in raid 0, then raid 1".

A striped mirror is recommended in everything I've ever read on the subject,
because it puts the redundancy at the lowest level of the array's geometry.

Using a mirrored stripe, on the other hand, means that when one disk fails,
any other disks striped with it, still presumably perfectly functional, can't
be used; the controller must instead read from and write to the mirror. If a
disk in that mirror subsequently fails, you've lost data — and remember that
when striping, the chance of failure is _multiplied_ by the number of disks in
the stripe.

EDIT: Footnote.

~~~
angry_octet
Don't get all upset. You said _stripe_ disks in your first comment, not
_mirror_ volumes.

In a big system I am usually more concerned with avoiding any SPOF (that will
cause downtime), then adding redundancy for the most likely failures.

Individual mirrored pairs on split chassis will mitigate against long rebuild
times for a single disk failure but it requires an all software architecture
as the individual HBA can't do hot spare rebuild. It also reduces the benefit
of the HBA cache. So intra chassis, striped mirrors; inter chassis, mirrored
stripes. Of course, mirroring across striped mirrors would be better, but the
cost to peak write and capacity might rule that out.

Also, yes I believe you, EMC sucks. But they do have some good engineers.

------
intrasight
Got a good chuckle. Such is life in IT. Gotta admit I missed the "Extremely
Massive Corporation" hint.

~~~
glass-
It reminds me of Douglas Crockford's talk where he mentioned that a company
asked him for an exemption to the "do no evil" clause in the JSON License, and
he said he didn't want to name the company as that would embarrass them so he
would instead give their initials: IBM.

~~~
MichaelGG
It's not embarrassing; their lawyers are just being cautious, as befits a
company with huge customers like theirs. The "do no evil" clause is stupid and
utterly ambiguous. If Crockford becomes a fundie, then is any gay-related org
in violation of the license?

It's his right to make up ridiculous licenses. Like the sisterware license
(you can use the software if you send me a pic of your sister if you have
one), people shouldn't take it seriously and avoid code licensed like that.
That Crockford doesn't get this is either him trolling or being clueless.

------
zatkin
I was hoping they would stay offline. It's really disappointing to see a good
service turn into a money wringing desperation.

~~~
morganvachon
The way I discovered SF was down was when I was trying to install some
Slackbuilds and kept getting MD5 mismatches on the source downloads. I noticed
I was getting the same MD5 for several different packages, so I tried to
manually grab a package and that's when I hit SF's error page (which _sbopkg_
was happily downloading and trying to pass off as somesourcepkg.tar.gz).

I found myself in the odd position of hoping SF would come back up so I could
finish what I was doing, when I'd normally welcome the news that they had shut
down.

~~~
pascalmemories
Sadly I too found it was screwed a few days ago when trying to pull some
source code hosted there (ironically so I would have my own copy in case
things went wrong there...). I just got their 'Disaster Recovery Mode' notice
and a subset of binary packages available.

Lots of google searching for what's wrong with Sourceforge revealed nothing
(other than complaints about their packaged installers/crapware). Now I have
the answer to the question I was really asking.

I just need to wait and see if the developers of the packages I need will
somehow migrate to github so I can get the source...

------
galoppini
SourceForge has posted info about current infrastructure and service
restoration activity at [http://sourceforge.net/blog/sourceforge-
infrastructure-and-s...](http://sourceforge.net/blog/sourceforge-
infrastructure-and-service-restoration/)

[Disclosure: I work for SourceForge]

~~~
M2Ys4U
>[Disclosure: I work for SourceForge]

My condolences.

------
eCa
> made by an Extremely Massive Corporation who until now you've generally
> respected.

As usual, one's respect for BigCompany is inversely correlated to one's use of
their products.

------
nitrogen
One can hope that if this hypothetical public-facing service never returns,
they will ship the backup tapes to the Internet Archive instead of Honest
Bob's Social Data Mining and Market Manipulation.

------
amelius
> It writes a full 32 bits of numeric user ID to its filesystem, but to save a
> few bytes it only stores 16 bits of group IDs. Some engineer probably
> thought that'd be enough for anybody.

I'm having the same issue with the number of hardlinks, which, for linux ext4
systems, is limited to 65000.

------
bootload
_' all the knowledge about how to handle this moody piece of hardware is lost,
like tears in the rain'_

Offworld, Roy Batty reference.

------
chris_wot
Yeah, they dealt with EMC. That's never going to go well.

(continues reading)

------
simonebrunozzi
"Let's talk about a hypothetical public-facing service that offers tools for
collaboration, revision control, and software publishing." \- it was hard not
to notice that he was referring to SourceForge. :)

~~~
Spivak
You know the title of the Reddit post was "Sourceforge is down due to storage
problems, no ETA", right?

