
A Dropbox account gave me stomach ulcers - slyall
https://www.reddit.com/r/sysadmin/comments/eaphr8/a_dropbox_account_gave_me_stomach_ulcers/
======
someonehere
I am not a fan of Dropbox. Stories like this are why I don’t use it.

A couple of years ago I inherited managing Dropbox for a company I was at. I’d
never really used Dropbox in a more daily use company.

I discovered Dropbox wasn’t very IT admin friendly. If you share out a
personal folder as your team folder, IT can’t find your folder to share it to
new members of your team. The person who owns the folder has to share it out.
So when new employees ask why IT can’t access and share the folder, we had to
explain that’s how Dropbox works.

The other crappy thing I discovered about Dropbox from that time. I have no
way to audit or tell who has what folder. If someone was sharing something
illegal from their folder, I had to turn on the auditing capability and
download logs as they were generated. It was confusing because if someone were
sharing something and I was the admin of that account, I should have access to
all files and accounts. Nope. So we dumped Dropbox before I left to avoid
scrutiny down the road for any acquisitions of the company.

Dropbox is still a consumer product trying to be a workplace tool IMO. Maybe
others have a different opinion.

This is a good story to tell on Monday.

~~~
jborichevskiy
> The other crappy thing I discovered about Dropbox from that time. I have no
> way to audit or tell who has what folder.

Have felt similar pain with G Suite Drive. Perhaps I simply didn’t find or
didn’t have the proper permissions (I wasn’t the top level admin on the org)
but at some point I wrote a Python script to output a list of users who had
access to each file in our organization. I saw no way of generating this
automatically.

~~~
vonseel
Drive can be a nightmare.

------
doctor_eval
What I find interesting is that the “insane asylum” apparently knew that it
was wrong.

e.g. when asked why they need so much space on their laptops, they were
evasive.

To my mind this makes it much worse. Not only were they doing something dumb,
but they _knew_ it was dumb, and they _hid_ it instead of fixing it.

It certainly invites speculation about the company culture in the acquired
business.

~~~
nikanj
Based on my experience: IT was well aware it was insane. Alas, IT got
absolutely no budget for fixing it from top brass, as kicking the can down the
road always made more sense short-term

~~~
Thorrez
IT of the insane company, or the acquiring company? Because the post says IT
of the acquiring company didn't know until the problem blew up.

~~~
swarnie_
DD failed to find it during the M&A, issue didn't become apparent until the
purchased company lost multiple employees who kept the plates spinning.

It's a failure all round but i would say acquiring companies IT team are least
to blame here.

~~~
vezycash
From the article, it doesn't seem to be the fault of the IT guys. It's more
like some rich guy who goes out buys a shiny used plane and hands over to the
pilot to use.

------
Tistron
I worked in a software company once where we acquired a product to integrate
into ours. It was developed by a self taught programmer in his 50ies that did
versioning by making copies of the whole directory when he had something that
worked, the way I also worked before university. Since he was the only
developer and he was pretty good this mostly worked. His first name was/is
Leif so from then on me and my friends speak of this style of "version
control" as "using Leif Points". I guess because it is phonetically somewhat
close to Save Points like in some computer games.

\- Are we using git or Leif Points for this project?

\- eh, how about git?

This story sounds like they also used Leif Points, but for their database :O

~~~
narag
I won't name the offender, but a well-known company I worked for did something
funnier: zipping entire directory and checking _the zip files_ in Visual
Source Safe.

~~~
masklinn
Given how broken VSS is, that might have been safer than using VSS as if it
worked properly.

~~~
narag
LOL, I used it in 1996-97 and it worked OK for a modest setup with only six
users and blocking at file level. Later heard some horror stories from people
working at big shops, but no worse than svn stories, anyway I haven't seen any
of those.

~~~
mikestew
About your time of using VSS, I was working at Microsoft in developer support
(working on COM data access, and _not_ VSS, thank $DEITY). It was fine for
small internal stuff we’d do, the stories I heard from industrial-strength use
did not instill confidence for anything more than a half-dozen devs. OTOH, we
shipped small components for a larger product using VSS, and I don’t recal any
issues. Hated the workflow and interface, but it worked

------
Iv
> 5\. Audit has some explaining todo.

Yes, that's a total fuck up on their side as well. That an incompetent company
dies while doing stupid stunts like that is understandable and how the market
filters out stupidity.

That another company looks at it and thinks "Oh that looks fine", that's a
total failure at their work.

~~~
kyberias
What "audit" are we talking about here, though?

~~~
DoingIsLearning
Usually when you acquire a company you sign of some intent lettet that shows
you are serious about the purchase provided they can pass your smell test.

Then after NDA's and a few other docs if a company accepts being purchased
they have to give access to their internal docs. Finance books, IT, etc.

If during this "due diligence" period the buyers find a major red flag they
might pull out, revise the price down, etc.

~~~
distant_hat
Having seen a couple of acquisitions what I see is that things go forward
because the top execs are close and went to the same school and so on. The
technical details shared are light and often don't matter until things are
settled and the plebs have to merge everything.

------
doctor_eval
I actually have had the reverse experience.

My company was acquired 18 months ago. We were the smaller company, about 1/4
the size, and as the founder/CTO, I expected the acquiring company to be all
over us. I mean, we had a pretty tight operation but there was only so much 16
people could achieve. There were plenty of things we could improve, and I was
looking forward to the resources and experience of this bigger company to help
us.

Was I in for a shock!

While their software dev practices were OK (certainly no better than ours),
what hit me hardest was their secops. It was a nightmare. And these guys
service some brand name clients.

As an example, not only was the corporate wifi password memorable - not
actually “P455w0rd” but similarly bad - but it was written on a whiteboard in
view of the whole office! Service passwords are still stored in clear text in
the corporate wiki, and few people understand why this is terrible. It’s like
a 90s IT shop, where people used to ask me “why would anyone care about our
password?”

Kicking the can down the road is fine for a while, but without strong
technical leadership, kicking the can becomes how things get done, and that’s
been my experience here.

I’m not long for this place.

~~~
n00b123
> Service passwords are still stored in clear text in the corporate wiki.

I know this painfully well :(. Can you recommend some good solution? Ideally
open source and self hosted, instead of cloud password manager.

~~~
pixiemaster
Disagree about the Cloud requirement.

Because: in this scenario you need to drive adoption first,if it is any
complicated the humans will not use it. so you need the easiest tool possible.

when everybody is using it, THEN go for the real security move.

(you can try to find a non-cloud one that is super easy, but...)

~~~
bartread
> Because: in this scenario you need to drive adoption first ...

Be careful: that sort of behaviour can get you fired and potentially land your
company in hot water from a legal and/or compliance standpoint, as well as
lose them customers.

Compliance is often as much, or more, about having and following documented
processes, as it is about having good processes. This can include an approved
list of software and services. If you step outside this you are likely
breaching your compliance regime.

In the specific case of passwords you may have policies and procedures around
secret management that would be violated by storing this information in the
cloud, even with a service specifically designed for this. As stupid as it is,
the quality (or lack thereof) of your current solution is not the key point
here.

If you decide to get all subversive and start using a cloud service then, at
the very least, if discovered you're going to get into trouble. At worst, if
customers become aware they might leave, and as I've already said, the company
might find themselves in legal difficulties.

------
_bxg1
I remember as a freshman in college using Dropbox as version control for a
4-person group project. Even then it was a hurdle. Even with everyone sitting
in the same room literally saying "okay I'm working on X nobody touch that
file for a couple minutes". Jesus.

~~~
sombremesa
I was imagining (wishful thinking, I'm sure) that they were pushing to a Git
repo hosted on Dropbox, which actually works remarkably well.

~~~
tigershark
No, he said that they had someone to manually download the last version of
each file. If they were really using git hosted over Dropbox they could have
simply cloned the repo.

------
amarshall
It should be noted that the evidence that stress is a primary cause of gastric
ulcers has been fairly weak since the discovery of H. pylori, though there is
some evidence stress may be a contributing factor.

~~~
semi-extrinsic
FWIW stress is a contributing factor in pretty much all diseases we know.
Particularly those where the immune system is active.

~~~
vezycash
I've got ulcer. My last ulcer attack, 5 years had me rolling on the floor.
Since then, I avoided stress. Ate at the slightest hint of hunger, even as I
grew a pot belly.

Also, I avoided situps, cos the muscle pains felt like ulcer attacks.

There's this specific food that I love. It's eaten RAW but often prepared in a
not too hygienic manner. Since I banned myself from eating it, I've slowly
become able to stay a few hours without food and can handle more stress.

~~~
koolba
What’s the food?

~~~
vezycash
It's called garri - popular in most (if not all) west african countries.
Here's a video of Nigeria's ex-president eating it.

~~~
dorfsmay
Have you seen this entry in Wikipedia:

[https://en.wikipedia.org/wiki/Garri#Health_implications](https://en.wikipedia.org/wiki/Garri#Health_implications)

> Garri is made from cassava which contains hydrocyanic acid .../... can lead
> to .../... worsening of ulcers.

~~~
vezycash
Thanks for the link.

The ground cassava is left to ferment for some days which removes MOST of the
cyanide. There's little to no health control for food in these parts. I've
read that over 50% of food stuff exported from West Africa to Europe and US
get culled at the border.

See:
[http://europepmc.org/article/PMC/3074370](http://europepmc.org/article/PMC/3074370)

See the abstract at:
[https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6526674/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6526674/)

When it's prepared with hot water, it forms a solid called Eba. And it's
consumed with soup. The heating makes it safe from my own observation.

------
andrewfong
Uh, as much as I enjoy reading this, posting this on Reddit seems particularly
unwise for job security purposes. This seems to be an ongoing incident and
there's more than enough info shared to de-anonymize.

~~~
Operyl
I’ve come to the assumption that either this happened already months ago, or
the much more likely scenario that it’s probably largely fake (if based on a
true story)

~~~
Thorrez
It mentions December, so either that's a lie, it's recent, or it was December
in some previous year.

~~~
masklinn
Updates are written in present tense as actions being undertaken so this seems
to be an ongoing trash fire.

~~~
Operyl
But that could also be the day by day as it went in the past, it still fits
the explanations I wrote out above.

------
danielovichdk
When Danske Bank aquired Sampo Bank in Estland, Danske Bank spend one million
hours on moving and lifting Sampo Bank's IT into Danske Bank's systems.

That didn't work out.

And we all know what happened then.

When the board and ceo want's to aquired a business, IT is just something they
believe can follow along easily.

Most developers are straight up lazy and uprofessional to do a job, and mixing
in security is just scary as hell.

~~~
hsivonen
Danske had some _really bad_ technical ideas: [https://hsivonen.fi/sampo-epic-
multifail/](https://hsivonen.fi/sampo-epic-multifail/)

------
useful
haha this is the kind of shit that happens when the engineering team cant get
anything from the either the sysadmins or leadership.

Can't run an application? Can't afford a license? Use the network share. No
network share? Use dropbox.

------
eschneider
And this, kids, is why companies do a technical due diligence during mergers.
:0

~~~
fortran77
It doesn't matter. The board and CEO will _never_ listen to the CIO and
technical team. If they want to do an acquisition, they'll do it. I've seen
this at least a dozen times in my 36 year career.

~~~
Gibbon1
I keep coming back to from the board and CEO's perspective they have a choice,
spend a lot of their time and effort to gain market share by investing in
their own company. Or acquiring it through an acquisition.

Seems like an acquisition is a lot less hassle and risk. And can be done
quickly. Most 'problems' can be made to go away by throwing money and fixers
at them.

~~~
guitarbill
This doesn't always work, see HPE vs Autonomy. That case is incredibly
interesting for an insight into how idiotic and lazy execs can be. Usually
though, it works out for them, probably by luck. Would be, err, "interesting"
if there were more Mike Lynch around to provide similar insight (Autonomy's
CEO). Then again, the case is interesting as HPE was willing to litigate, and
a certain British tech news outlet picked it up.

------
ermir
What a complete nightmare! I bet that most of that data is useless, and the
Dropbox account is just used as a dumping ground for everything and there's no
tagging, cataloging, or any sort of data structures.

~~~
Iv
My assumption, given the storage requirements for laptops, is that the actual
database is 2-3TB is size and that the rest is archived snapshots of the DB.

~~~
ceezuns
Assuming that this database has been building up over the past decade, I'm in
awe over how much money they must have spent on Laptop's in the early part of
this decade.

~~~
bsaul
i had trouble understanding the laptop part of the story. You can’t sync 450
terabytes to a 4terabytes laptop anyway. So why the big storage ?

~~~
jdbernard
It's probably as poster above says: each snapshot (sounds like it's DB + app
instance or something, maybe a VM snapshot?) is probably > 1TB in size. If you
have a 1.5TB image it would make sense to want 4TB storage so that you could
have two versions locally at a time. Also explains how long it takes them to
do things. Every time they switch versions they have to swap out TBs of data
to pull latest. Because it also sounds like they don't have a way to grab just
the diffs. They grab whole images at a time.

------
farrelmahaztra
The best worst thing I’ve ever read. No idea how they pulled that off for so
long.

~~~
allthecybers
Reminds me of a firm I worked for. They decided to adopt Google Drive for the
team of about 20 employees.

Every couple weeks an employee would mistake the synced folder for their local
disk and rearrange files or delete files and then the files would disappear
for the rest of the company.

Then we’d have to go into the backups and restore the files and send a message
out to everyone to hold the files they were working on before saving them to
the drive again.

~~~
blfr
How come? Google Drive has fairly granular access control access control.

------
kstenerud
That was a cool nightmare scenario story, but holy mother of mercy was it ever
tedious to read!

I can tolerate some bad grammar, but somehow the combination of incorrect
articles, words auto-spellchecked to something completely different, run-on
sentences, and lack of punctuation, made it a real chore to read through
without getting lost...

~~~
ghego1
The author is probably a non native English speaker

~~~
beojan
It actually reads to me like the author is definitely a native English
speaker.

~~~
AndrewDucker
Agreed.

The patterns sound like native English speech.

I suspect dyslexia.

------
lazylizard
Is this a problem with tech? I imagine the users who decided to put sensitive
data on dropbox are personally liable? Legal might have work to do? But i fail
to see the tech challenge?

~~~
robjan
It seems that Dropbox was hosting all of the production data and being used as
some kind of document store / db system in this story.

~~~
lazylizard
Can tech fix it by 1\. Disabling the dropbox account? 2\. Stopping dropbox
urls from resolving in their lan if need be? Or blocking dropbox ips? 3\.
Buying a md1280 or similar storage quickly n put a freenas or similar on it?
4\. Put syncthing or similar on all the user computers n this new nas?

~~~
lazylizard
Actually why is dropbox involved if its a git directory???

~~~
lazylizard
Git annex? Sparkleshare?

Can tech just stop the dropbox bit n just set up sparkleshare for everyone if
git is holding up? If git is less than efficient..then syncthing, seafile
would all work for their file sharing needs n r trivial to setup?

~~~
djrogers
I think you missed the part where it’s not Dropbox as a file sharing solution
that’s the only problem - their entire web app runs via Dropbox API calls from
Heroku!

------
ryandrake
I hate to be That Guy but how on earth is this not discovered during due
diligence before “insane asylum” Company was acquired?

~~~
bakul
Reminds me of HP's 11.1 Billion dollar blunder of buying Autonomy....

[https://www.nytimes.com/2012/12/01/business/hps-autonomy-
blu...](https://www.nytimes.com/2012/12/01/business/hps-autonomy-blunder-
might-be-one-for-the-record-books.html)

~~~
HeWhoLurksLate
That article was worth clearing cookies for. Thank you.

------
planetjones
Dropbox would really let you get to half a Petabyte ? I know it says as much
space as your team needs, but wow. Do Dropbox start to charge per data stored
when you take out an enterprise account ?

------
eismcc
While it all feels insane from one perspective, it’s not hard to imagine a
different narrative where the “founders of ‘insane asylum’ hacked the startup
system, were product first, got acquired, and used their new home to pay down
tech debt.”

~~~
malux85
and made everyone else suffer?

Also - they are NOT paying down the tech debt. The programmers and sysadmin
staff doing overtime mentioned are.

------
im3w1l
What are the advantages / disadvantages of going panic mode like this?

Would just increasing the size of the dropbox to keep things muddling along
and then trying to figure out a migration strategy come Monday morning be
impossible for legal reasons?

~~~
geuis
Sounds like the big issues are at the compliance level. They’re in a time of
the year where lots of companies have much more limited availability of
engineering, management, and legal staff. Sure they could easily eat the cost
of temporarily upgrading the Dropbox account. But as OP describes, there are
huge implications in terms of security.

They didn’t find out until today that a number of senior staff were laid off
during the acquisition. Lots of room there for bad feelings. That opens them
up to all kinds of vulnerabilities from disgruntled ex employees.

Imagine the repercussions come post-holidays if something bad happens during
the next few weeks before everyone is back on board and there’s a security
breach over the meanwhile.

------
z3t4
This story has many angles, the developer who built the software, the founder
who took all the money, then sold the software and fired the developer. Then
Dropbox who we all laughed at, is now used for distributed databases.

------
donio
I don't use Dropbox so I am wondering, is this fundamentally different from
storing everything in S3? Is it like a single S3 bucket? Or multiple buckets
on the same account?

~~~
FascistDonut
Dropbox is essentially just a cloud synced folder. Anyone who had the shared
login credentials had access to the entire production server and fragments of
it could be left on any computers that were synced to it. Anyone could also
delete or modify anything and it would propagate the changes to everyone
else's copies with very little logging or version control. You can restore a
previous version of the file, but it isn't like git or anything... just more
like recently saved versions.

~~~
donio
Is it shared credentials even for business accounts? Surely they must have
some kind of team based shared access option for that?

~~~
FascistDonut
The post said they had an enterprise account, which I assume could limit
access via separate logins, but many of the users were sharing the same single
set of credentials (e.g. everyone logging in with the same account), so there
was no real access control or knowledge of who all had access using that
account.

------
manceraio
This looks like how no-code projects will end up in the future.

