
GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down - craigcannon
https://blog.ycombinator.com/gitlab-distributed-startup/
======
bryanh
160 employees remote is impressive and commendable. Zapier is fully remote as
well (but half the size in employee count). I'd say "write everything down" is
a great shortcut to the sorts of practices you need to cultivate.

We've also noticed that over-communicating is critical but hard - it is
surprising the things that are "yeah yeah, we know" to some but are "oh we're
doing that?" to others. This is only natural - organizations become complex as
they grow, and individuals are busy doing their thing. You often have to bring
the important data to them.

On another note, working remote is awesome. I recommend everyone give it a
spin once in their careers - but try to find a team that embraces it. I've
heard mixed experiences from those who were the single remote person on a
team.

~~~
Perihelion
Honestly, I haven't seen a company that does the hybrid remote/on-site thing
well. I'm sure they exist, but every time I've worked at one the people who
were remote were out of the loop on almost everything.

Hybrid remote/on-site requires some great discipline. If you have a
watercooler chat with someone about a feature, it's easy to forget that a
remote colleague wasn't there for the discussion/not document what was said.

+1 for working remotely on a team that makes an effort to make it work.

~~~
scruple
Yep, this is absolutely correct in my experiences. I'll chime in, since I'm a
remote employee at a company that is hybrid remote/on-site but with the vast
majority of employees being on-site.

Long story short: I actually started on-site but negotiated part-time remote.
Then after about a year of that my spouse accepted a dream job offer on the
other side of the state and so it was no question that we were moving. They
wanted to keep me on and so I transitioned to full-time remote.

It has it's ups and downs. From my experience, it depends on the team and
ultimately the lead and/or the manager. If they get it, it's good. If they
don't, don't bother.

My last team, it was beautiful. Everything worked great. It became obvious to
me after transitioning away from that team just how much extra work my manager
was doing to keep his remote employees in the loop. With the team I'm on now,
it's not working at all. Why? Because, as you so eloquently put it:

> the people who were remote were out of the loop on almost everything... If
> you have a watercooler chat with someone about a feature, it's easy to
> forget that a remote colleague wasn't there for the discussion/not document
> what was said.

This is exactly the problem I am facing down. About a month ago, I even tried
talking to my manager about it. Even going so far as using this exact
phrasing. His response was that I should come to office for 1 or 2 day trips!
Completely missed the point. I'm not surprised, I guess I just expected some
effort.

I'm trying to remain positive but it's tiring. I can do good work and make
progress but I'm with a lead who is insulating me from meetings and it feels
deliberate. The few times I am invited in to a meeting, it's easy to see that
he accepts and assumes credit for _the teams_ efforts. He was hired recently
as a Senior, our manager wanted him to transition to a Lead, and he's actually
landed somewhere around trying to be an architect. Ranting aside, when I bring
my concerns up to him it's obvious that there's no attempt to better
understand the issues/challenges of the hybrid approach. It doesn't help that
he's never worked with remote employees before, either.

~~~
ghaff
>His response was that I should come to office for 1 or 2 day trips!

I understand he missed the point in this case. However, if a company is
willing to pick up the travel costs associated with a partly distributed team
getting together on a semi-regular basis, that's often a pretty good approach.

Yes, processes should be such that ongoing communication is good. However one
of the costs of having a more distributed workforce (which has various
benefits to the company, including financial) should be a bigger T&E budget.

~~~
nojvek
It seems that having great speech to text would make remote teams workable.
Meeting notes are automatically taken and organized into a searchable journal

~~~
passiveincomelg
If nothing else, that would at least be an improvement over "let's take a
photo of the whiteboard after this one hour ad-hoc meeting and attach it to
the jira ticket".

------
deepaksurti
>> 2:41 – GitLab values boring solutions: our product should be exceptional

Exceptional products have exceptional UX. Gitlab IMHO has the worst UX of all
git based products out there, I much rather take BitBucket over Gitlab. I
tried using Gitlab, but no, I would much rather pay the 7$ to GH for my
private repos.

I sincerely hope they make an exceptional product. And 'should' better be
'must'!

~~~
zenlikethat
I'm curious what you dislike about the UX. Personally I like the "feel" of the
website generally (though it's not as polished as GitHub) and just find it to
be slow, which I think they're working on improving.

~~~
bpicolo
Personally, not a fan of how a readme-less repository just has an entirely
useless main page. I much prefer having the file view a'la github on that
page. That's the main UX complaint for me. Still use gitlab bunches.

Main complaint overall is the incredibly sluggish git push speed.

~~~
eknkc
You can change that on your personal prefs to show the file list by default,
with Readme on bottom:
[http://c.ekin.io/3z1V2v3P041b](http://c.ekin.io/3z1V2v3P041b)

git push speed is fine on our own ce installation but gitlab.com is beyond
shitty.

~~~
sytse
We're considering changing the default so people don't have to find the
setting [https://gitlab.com/gitlab-org/gitlab-
ce/issues/26834](https://gitlab.com/gitlab-org/gitlab-ce/issues/26834)

~~~
zenlikethat
I'd +1 this as default behavior. I want to see the source code!

------
dandersh
The most frustrating thing about working remote was the complete lack of
documentation around anything -- meetings, requirements, internal
frameworks/libraries, etc. Good for them for emphasizing documentation.

I lost track of the amount of times I was told to "look at the source code"
when asking about documentation for the internal framework.

Also fun was being assigned a feature and having it's functionality explained
over the phone, with any subsequent follow-up being met with "Didn't we
already talk about this"?

I never would have thought that a remote based team would have the worst
documentation out of anywhere I've worked, but that's exactly what happened.

~~~
bgaluszka
> I lost track of the amount of times I was told to "look at the source code"
> when asking about documentation for the internal framework.

Can you elaborate on this one? This is my preferred way.

~~~
overcast
You prefer just being handed a pile of code, rather than documentation on how
to use it? please.

~~~
bgaluszka
In case where I work for a given company and it's internal tool, always.

~~~
crpatino
Imagine this scenario.

You work on product A. Your VP of very important things declares that every
product should use library L. Library L has been designed to incestously
interact with product D and makes a bunch of assumptions about what the
"working environment" should be.

Teams B and C have already made the transition, so L made some kludgy APIs to
make B and C interaction possible, though it is maintained with low priority
because everybody knows D is the money maker. Both B and C have a bunch of
assumptions in common that A does not share, and they both had to compromise
and emulate D environment to some exent.

You are given L binaries, a few toy examples of how to use the most basic
functionality in L, and access to B codebase. L codebase does not use the same
source control system, so you don't even know where to begin looking for their
code.

Please keep in mind that your manager does not expect you to learn all the
inner workings of L. Your priority task is to use L to support some new
feature in A. You still have a number of other tasks to accomplish in A that
are not related to L, and some of those might raise in priority with little or
no notice.

Do you still think it is a good idea to just browse L code, without knowing
how large (or well writen) it is?

~~~
bgaluszka
Note that I said it's my preferred way but by any means that doesn't mean that
if there is some documentation I won't look at it.

~~~
crpatino
The fact that it is your preferred way tells me that the only internal
frameworks you have encountered in your career were tidy, tiny, and self
contained.

Having struggled with very large tools that were anything-but, I beg to
differ... but I guess it is a matter of personal experiences.

------
manojlds
After the mess up, I don't really like seeing these posts about Gitlab. Maybe
this is their problem after all.

~~~
cookiecaper
I think the problem has more to do with their recruiting. To be a successful
distributed company, you need a disciplined and highly experienced workforce
(note that this does not mean a highly _educated_ workforce, which may
actually be a contraindication).

GitLab offers a middling base salary for a role and applies modifiers for
experience and city-based cost-of-living (both of which may modify the base
downward; my CoL adjustment is -40%, and I live in the United States!). They
explicitly state on their hiring page that they prefer people from cheaper
areas. [0] [1]

On top of that, just from their headcount, it sounds like they have way too
many people. 160 for a technical company like this is hard to manage when the
employees are local; it's even _harder_ to manage when employees are remote.
Again, for remote work, you need mature, highly-skilled staff with a strong
history of successful self-direction. You're not going to get that when you're
chopping their salaries in half based on their location. Bad local wages is a
major reason to work remotely in the first place.

[0]
[https://about.gitlab.com/jobs/developer/](https://about.gitlab.com/jobs/developer/)
; use the calculator at the bottom of the page

[1] [https://about.gitlab.com/handbook/people-
operations/global-c...](https://about.gitlab.com/handbook/people-
operations/global-compensation-calculator) ; gems include "you are required to
notify us when you move and we may adjust your compensation up or down" and
"[a]ll things being equal we will hire people in lower cost markets vs. higher
cost markets."

~~~
pc86
Just ran my numbers and got about 55% of my current salary assuming a lead
position (what I have now, which I would probably not get with them) and "a
lot of experience" which I'm on the fence about and they would probably
disagree with.

Getting rid of the COL deduction by selecting NYC as my location actually
makes it reasonable so something tells me not pinning the US salary floor to
1.0 is saving them money and simultaneously getting them lower quality
developers.

~~~
sotojuan
It's probably set up to attract cheap Eastern European devs.

~~~
mdekkers
I applied for a position at Gitlab a week ago, and didn't even get a response.
Probably because I am too expensive, too old, and live in an area with strong
labour laws and staff protection. It's a shame, because it would have been
decent to at least get a "sorry, not interested" reply, but no response at all
is just not cool. They write on their hiring page: _Always leave feedback,
this will help everyone to understand what happened and how you came to your
decision._

Nice to "write everything down"

~~~
NadiaVat
@mdekkers glad we saw your post. I tried searching for you in our ATS, but
could not find any application under that name. Can you email me directly on
jobs@gitlab.com? I'd really like to see what happened to your application and
why we didn't respond.

~~~
mdekkers
Hey Nadia, thanks for reaching out. Sure, I will send it along.

------
Kiro
Given the complete backlash I'm seeing on HN it seems like transparency
actually hurt you. Very sad.

~~~
sytse
We're still committed to transparency even if it means taking a few lumps now
and then, it's better in the long run and it's baked into everything we do.

~~~
photonwins
Kudos to the entire team, as another commenter said, please don't be
discouraged by the petty comments. It takes balls to admit mistakes in public.
You have earned many people's respect.

------
JustSomeNobody
Wow. You all in the comments must be a bunch of perfect people, huh?

Good grief.

Also, nice cameo by the office cat.

~~~
sytse
Thanks, but the critics are right. This is not a good time to be promoting
ourselves. We should be working on our postmortem and figure out how to
improve our service. We're actually doing that, this video was recorded before
the outage. But impressions count.

------
Cofike
Judging by the number of joke and low hanging responses on this thread I think
they should have held this article for a little longer.

~~~
sytse
I think we earned a reality check. Have such a promotional video go out after
the outage was out of touch on our part. We were aware it would be posted and
didn't ask to hold it.

------
geoelectric
While I was at Mozilla, John O'Duinn gave a pretty great presentation about
this sort of stuff.

The core message (mixed with my own takeaway) was that you have to consider
all of your offices, even HQ, more like just another field office or coworking
site--no location is "central" compared to others. As long as you consider
accommodating remote people to be a separate task, it's a task that can be
deprioritized. Ideally, anyone in your office should be able to work remotely
at moment's notice with little-to-no change in procedure.

Much of his presentation was outlining concrete techniques towards making this
actually doable. He still has the slide deck online.

[http://oduinn.com/blog/2014/11/09/we-are-all-remoties-
nov201...](http://oduinn.com/blog/2014/11/09/we-are-all-remoties-nov2014/)

In practice, of course, real life was imperfect and there's no question that
you take a productivity hit over people in the same room. But if you do want a
heavily location-agnostic organization as a core value, his take is a nice
start.

------
nedsma
Glad that Gitlab is pushing the remote working boundaries even further. Yes,
document everything, write easy to understand code, offer your remote
coworkers to call you whenever they need help, leave feedback, praise work as
if they're local. It all matters and even more when remotely.

------
smarx007
It would be so much better if YC would made it a podcast instead (or in
addition). Would be a perfect thing to listen on the go.

------
peterwwillis
I think the worst team I've ever worked on was the one that had a team lead
that literally never wrote anything down. Everything was verbal, and what
wasn't verbal was private messaged. We would have the same discussion three
times because nobody wrote it down the first or second time. Nobody even knew
what the changes to the formal design were, because they never had a
documented change review.

I always tell people on my team that if it isn't written down, it doesn't
exist. When you get into the habit (and learn how to manage all that
information) it really saves your bacon.

The other thing that would have saved them from the backup drama was learning
how to document and date your to-do list. Suddenly someone notices that you've
had a test process for backups on the to-do for over a year and they bring it
up at the next meeting.

And another thing: in team meetings, people can write their own meeting notes
so everyone becomes responsible for documenting their responsibilities and
what affects them. It's easier to do this remote than at a stand-up, because
you're already at your keyboard.

------
siliconc0w
People seem to have some subtle insecurities around broadcasting their
activities which contributes a lot to the problem of distributed teams.
Ideally all communication goes to everyone and everyone can apply their own
client-side filters for what they want to care about. The problem is people
want to present a certain face up the org and a different one to their peers
so they PM each other and create isolated channels of communication
(distribution lists, slack channels, etc). Now have some cognitive overhead of
whom to loop in when which creates some other complications because it means
if you're receiving a message it's explicit rather than implicit. TLDR -
broadcast everything to everyone by default and create a culture of
transparency.

~~~
wallstquant
That's impressive. Do they use open channels to communicate everything?
compensation? Financials?

~~~
mahyarm
Well they do have a compensation calculator on their job listings along with
the reasoning

------
newsat13
Last I checked a year ago, GitLab had just 80 employees. So, it's now doubled.
That is quite some growth. More often than not growing companies just collect
lots of employees and then a startup comes and beats them...

------
gravypod
I can't wait for a remote-work company that has a policy like this to get into
some kind of legal troubles that warrents subpoenas. This is a prosecutors
gold mine. I feel bad for the defense lawers already.

~~~
yuhong
I am taking issues like admission of wrongdoing in settlements seriously now.

------
whoiskevin
I have to watch a video that then tells me to write everything down? Sorry I
couldn't resist. I believe and practice that as much as I can because it makes
things easier regardless.

------
ceejay
I think hindsight will reveal that the things that make a "distributed
company" successful are really the same things that make a "localized company"
successful. I think it's just that having all employees on-site probably makes
it easier for companies to "fake it" and stumble into success through sheer
grit and determination. Probably a lot of times even when they have less-than-
adequate (but highly motivated) human resources.

~~~
ddebernardy
When everyone is on site, there's enough implicit communication going on that
you can get away with not documenting things in writing for a lot longer than
in a remote team.

------
Paul-ish
This is interesting, as I interned at Mozilla, and they have a large number of
remote employees. There was only one person (other than me) on my team in my
location. Mozilla also records a lot of things, but the intent is more to
publish that stuff publicly, in the spirit of openness towards the community.
For example, all my team meetings had public notes. I wouldn't be surprised if
this tendency to record everything for the sake of the community also has
positive benefits inside Mozilla.

------
MrFurious
Obviously, Gitlab don't have the secret for best uptime.

------
winteriscoming
More than knowing how they manage remote employees, at this point, with more
than a week since the data loss incident, I would be genuinely curious to know
if their backups are now functional and what plan they have put in place or
planning to, to verify backups work.

Furthermore, I won't be surprised if they have seen more concentrated spam
attack, after the news of this data loss surfaced.

~~~
YorickPeterse
There's a work in progress merge request for the post mortem, which can be
found at [https://gitlab.com/gitlab-com/www-gitlab-
com/merge_requests/...](https://gitlab.com/gitlab-com/www-gitlab-
com/merge_requests/4779).

This blog post mentions a whole bunch of issues that are being worked on, this
list can also be found at [https://gitlab.com/gitlab-com/www-gitlab-
com/issues/1108](https://gitlab.com/gitlab-com/www-gitlab-com/issues/1108).

> Furthermore, I won't be surprised if they have seen more concentrated spam
> attack, after the news of this data loss surfaced.

Not yet. Since Akismet has been enabled for snippets it seems the amount of
spam is gone, or at least not noticeable.

------
jgalt212
160 employees? That seems like a lot support a product that is a wrapper
around another product. I know the lede is skeptical, but does anyone have any
idea how those numbers break down? i.e. is there a large number of
sales/support due to complex installs?

------
chmars
What kind of tool would you recommend to write everything down, especially a
self-hosted tool?

------
uladzislau
Obviously what they say and what they do are two different things based on the
recent events.

------
WalterSear
And don't store it on gitlab.

------
the-dude
I have the impression this is being moved off the frontpage in an accelerated
manner ...

~~~
the-dude
And it is back up, from place 22 to 12 again.

~~~
dang
Sometimes software and moderators do the same thing, which doubles the effect.
When we notice that, we turn off the moderation and just rely on the software.
That's why this one fell and then went back up.

------
isaac_is_goat
And yet their database gets blown away, and now their redis is acting up. Go
Gitlab.

------
tra3
How do you organize internal knowledge base?

------
krisdevelops
write everything down where? and in what format?

~~~
cmatija
Hi Kris,

We usually tend to add every single bit of information that we attain to our
handbook[0].

Other than that, we try to over-communicate over issues, slack and email.

[0][https://about.gitlab.com/handbook/](https://about.gitlab.com/handbook/)

------
hartator
or "Shut Everything down". Ok, that was lame.

------
Philipp__
Everybody is prone to making a mistake! As long as you learn something from
it, so that it never happens again, it's ok. And I really don't get why people
give them bad rep for this post. I mean it's obvious why it's posted and we
all know what happened week ago, but consider their size and scale, it's not
easy to manage so many people remotely.

~~~
snarf21
"...but consider their size and scale, it's not easy to manage so many people
remotely." This is very true but the title of the post is "GitLab’s Secret to
Managing Employees in 160 Locations: Write Everything Down".

That is the backlash, don't say you've figured out the secret then make a near
catastrophic mistake.

~~~
e40
I don't see how their remote structure has anything to do with their mistake
with backups, etc talked about here lately.

------
johansch
I mean, really: after figuring out after the fact that five out of five
backups were malfunctioning - go here to read our secret to managing 160
employees remotely?

Nah... no thank you. I think I will get my management advice from a company
that is not totally broken. I don't care how transparent you are, or how
remote-worker friendly you are, if you can't do the basics right. You are in
the data storage business! Shut down the company and refund the money to the
investors.

------
pzh
So after losing a ton of user data and repositories about a week ago, because
a remote engineer couldn't bother to check to which remote machine he was
issuing 'rm -rf' commands as superuser, now GitLab is teaching us its 'secret'
to success?

------
yunolisten
> Write Everything Down

Then rm -rf the paper....

[https://www.theregister.co.uk/2017/02/01/gitlab_data_loss/](https://www.theregister.co.uk/2017/02/01/gitlab_data_loss/)

~~~
Spooky23
Not really germane to the topic. This type of op fuck-up happens everywhere.
It's hard to build solid process, particularly in growth phases.

Unless there are 2x a year restore tests, I personally assume a 60% backup
fail rate.

~~~
mbiondi
The only reason they are still in existence is due to a chance backup they
took for a tangential reason. From the sounds of it, their solution is held
together with bubble gum, some tape and lots of hand waving. Being in 160
different locations probably doesn't help much either.

~~~
chousuke
I'm pretty sure most solutions on the internet consist of bubble gum, some
tape and lots of handwaving. Gitlab's screwup is hardly unique, even if it was
very public.

It's difficult and expensive to build and _maintain_ a solid system, and even
if you want to, time and financial pressures often just don't let you, on top
of the issue of just communicating the need for solid engineering, as it
usually only becomes apparent when the problems start occurring.

