
Gitlab was down - vickyonit
https://status.gitlab.com/
======
darekkay
It is up again.

> [Monitoring] GitLab.com is now recovering. We found 2 last DB nodes which
> had not reverted their change. Apologies for the disruption. [1]

[1] [https://status.gitlab.com/](https://status.gitlab.com/)

------
1337shadow
Back in the days GitLab made it clear that it aspires to be a software company
rather than an infrastructure company.

The reason I favor GitLab over GitHub is that I can install it on a dedicated
server and have blazing fast performance, which helps development performance
on an every day basis.

I've been maintaining a bunch of deployments of all sorts (k8s, docker-
compose, baremetal) for the last years for several customers of all sorts of
size, the upgrade process has always been pretty smooth, sometimes I ran in
edgecases yes but always found a solution.

My deployments are always up and kicking, unless I'm messing with the
configuration and doing some mistakes.

I highly recommend hosting your own GitLab instance, even on a single server.

~~~
oefrha
> have blazing fast performance

Sorry, but GitLab has never been blazing fast. The gitlab.com instance is
notoriously slow — supposedly improved a lot over the past few years, but
still feels pretty sluggish. My self-hosted instance isn’t much better.

In fact, a couple of open source maintainer friends looked into migrating to
GitLab when GitHub was acquired by MS; they did not precisely because GitLab
was too slow.

~~~
1337shadow
Let me clarify: git clone/fetch/push on gitlab.com or github.com is sluggish,
but on my private instances it's blazing fast, because I get great and un-
metered bandwidth from my dedicated servers which might not be the case for
everybody / every country.

Another thing: CI starts the very second I push, and since I configure
dedicated runners extremely well GitLab makes it easy keep every pipeline
stage under 5 minutes, which is critical to me, as such I recon I have
acquired countless tricks for that matter.

Also, the ChatOps integration (GitLab to Mattermost or Slack) seems instant
for me.

Those factors matter for me because I'm an extremely fast iterator and often
iterate on several repositories at the time.

The cherry on the top is that the mirroring between my gitlab instance and
github is also blazing fast.

As for the web interface, well they all feel slow to me so I don't spend my
time waiting for them so it's ruled out of the equation for me anyway, I'm
just far more comfortable with command line anyway.

But, pushing to gitlab prints out the URL to create a pull request from that
branch, or to view the existing one, also a time saver that's not available on
github, not sure why though.

~~~
chriscool
> pushing to gitlab prints out the URL to create a pull request from that
> branch

You can use push options to automatically create a pull request and set it up
when you push:

[https://docs.gitlab.com/ee/user/project/push_options.html](https://docs.gitlab.com/ee/user/project/push_options.html)

~~~
AdamJacobMuller
Had no idea that existed, super cool, thanks!

------
Uninen
We use GitLab at work and I use it for personal projects as well, it has been
very slow for several days now and they had downtime yesterday as well.

I don't mind the general sluggishness of the system that much (as I love the
platform in general) but when you can't get your work (nor hobbies) done
because of tools breaking, it gets really annoying really fast.

~~~
tadzik_
> when you can't get your work (...) done because of tools breaking, it gets
> really annoying really fast

I always assumed that the publicly hosted version of gitlab is basically a
giant demo version of the enterprise edition you buy to host it yourself.
Hell, you can even host the community edition for free.

If your work relies on it, why rely on a free online product?

EDIT: TIL gitlab.com also has a paid options. I stand corrected.

We have a hosted Gitlab at 2 of my clients. Both are up.

~~~
idiocratic
The online version has non-free plans as well, so I don't see your point of
not relying on it.

------
yRetsyM
There has been a theme of instability with Gitlab.com over the last week or
two. I'm not sure if it's growth related (they've seen a steady increase of
users/traffic) and they've reached a scaling peak. OR if it's technically
related - they've been doing a number of different infrastructure changes over
the last few weeks which make a material difference to the main layers of the
service.

For me the real test here is how they respond to this. As a paying customer I
want to understand the issue, the efforts to prevent this in future and how
they communicate this.

~~~
appkate
Suffering the same instability issue and switched back to Github as that's
what a paying customer would do.

------
dijit
Notably they don’t hire any real people with ops experience. Erring instead to
go for developers hoping that they can do everything needed.

I like gitlab as a product but they don’t have a service mindset, and I think
not hiring operations-centric people is a symptom of that which causes these
kinds of issues.

~~~
softwarelimits
How do you know that they do not hire "any people with ops experence"? Do you
have insight? Is there any public evidence for this or anything you could
publish yourself to add some substance to your words?

It would be great if you understand that just saying something is not enough
on the internet.

~~~
dijit
What you say is completely fair.

I’ve been looking at job postings and watching the way they work too for a
little time since it’s all open.

DBAs are “ruby devs who have used Postgres”

[https://about.gitlab.com/jobs/apply/backend-engineer-
databas...](https://about.gitlab.com/jobs/apply/backend-engineer-
database-4473989002/)

SREs are “ruby devs who have used docker/kubernetes”

(No job listing currently)

The only open job labelled “ops” is telling.

[https://about.gitlab.com/jobs/apply/frontend-engineer---
conf...](https://about.gitlab.com/jobs/apply/frontend-engineer---configure-
team-4529842002/)

~~~
toupeira
GitLab engineer here.

The "Ops" section actually consists of product development teams for features
in GitLab itself (the Configure/Monitor "stages"), while the SREs are in the
"Infrastructure" department. See
[https://about.gitlab.com/handbook/engineering/#engineering-d...](https://about.gitlab.com/handbook/engineering/#engineering-
departments-sections--teams). But yes, we don't seem to be hiring SREs at the
moment, but I assume we'll add more openings in the beginning of next year.

Regarding DBAs, that job you posted is more of a normal Backend Engineer role
with a database specialty. We also have dedicated Database Engineer roles:

\- [https://about.gitlab.com/job-
families/engineering/database-e...](https://about.gitlab.com/job-
families/engineering/database-engineer/)

\- [https://about.gitlab.com/job-
families/engineering/database-r...](https://about.gitlab.com/job-
families/engineering/database-reliability-engineer/)

The job description for SREs is here:

\- [https://about.gitlab.com/job-families/engineering/site-
relia...](https://about.gitlab.com/job-families/engineering/site-reliability-
engineer/)

~~~
dijit
Hi Toupeira,

I didn't find some of those jobs, so thanks for linking them.

As a side note I just went through those job descriptions and I -really- like
the layout.

On topic: Unfortunately they really do prove my point. There is a very strong
focus on "strong" programming skills which is rather undefined. It's literally
mentioned in every single role description.

The overwhelming majority of staff that knows how to run software reliably
ironically are not software engineers, although there certainly are some
software engineers who also possess this skill.

The people I'm speaking about typically understand concepts and solutions
(like PAXOS, filesystems or public cloud) more than they understand software
development methodology or software product structure.

I guess you have a global reach and can be quite picky about who you hire,
maybe you /do/ exclusively hire architecture and systems focused programmers,
or maybe "strong" programming skills are a different definition to mine.

~~~
toupeira
That's certainly a valid concern! From my perspective, the programming skills
in these job descriptions are one requirement among many others, and I'm not
sure how much weight it really has in the hiring process for these roles,
especially if your other skills are a good enough match.

We do have this note on all job pages, which maybe should be more prominent:

> Avoid the confidence gap; you do not have to match all the listed
> requirements exactly to apply.

Some amount of general programming experience is definitely required though,
since SREs and DBEs frequently have to dig into our codebase and things like
Ansible runbooks. And especially regarding SQL, a lot of it is heavily
abstracted not only through the Rails ORM but also our own code.

Found some other jobs that focus less on programming, though we don't have
current openings for most of these either:

\- [https://about.gitlab.com/job-families/engineering/cloud-
nati...](https://about.gitlab.com/job-families/engineering/cloud-native-
engineer/)

\- [https://about.gitlab.com/job-
families/engineering/infrastruc...](https://about.gitlab.com/job-
families/engineering/infrastructure/analyst/)

\- [https://about.gitlab.com/job-
families/engineering/monitoring...](https://about.gitlab.com/job-
families/engineering/monitoring-engineer/)

\- [https://about.gitlab.com/job-
families/engineering/security-e...](https://about.gitlab.com/job-
families/engineering/security-engineer/)

\- [https://about.gitlab.com/job-
families/engineering/vulnerabil...](https://about.gitlab.com/job-
families/engineering/vulnerability-research-engineer/)

We don't seem to have a good overview of all roles, I found these through
[https://gitlab.com/gitlab-com/www-gitlab-
com/tree/master/sou...](https://gitlab.com/gitlab-com/www-gitlab-
com/tree/master/source/job-families/engineering) :)

------
kburman
> Website, API, Git (ssh and https), Pages, Registry, CI/CD, Background
> Processing, Support Services, packages.gitlab.com, customers.gitlab.com,
> version.gitlab.com, forum.gitlab.com

How come all of them are down all at once.

~~~
beaconstudios
presumably a single point of failure - my guess would be something at the
network level.

~~~
manojlds
It's always DNS!

~~~
yyx
This time it might be Consul

------
jlengrand
I used to be a total Gitlab fanboy. I was going to Sid's meetups back in
Utrecht 6 years ago, and he's a model of mine to this time.

I would have done anything to stay on the platform, for at home and the office
and setup gitlab instances in two different offices.

Since somewhere last year, I moved back to Github primarily, and I'm sad to
hear that my company is likely to make the same choice soon.The only reason is
stability, and that's a little sad to me. I really want to love the product,
but I need something that just works; not bells and whistles

~~~
kungtotte
Whenever GitHub is down you see comments saying much the same thing only with
the names reversed.

The thing that strikes me each time is how fragile everyone's setup is if
GitHub/lab is a single point of failure for them...

At least gitlab let's you self host, which would let you run backups on
offsite hosting meaning zero downtime.

~~~
jlengrand
Agreed, nobody's perfect.

My experience with github is much better though on average. Haven't had a
single bad experience personally the past year for my personal stuff. Not
once. Github does only one thing, the CI is usually somewhere else so there
have less reasons to fail as well... It might be comparing apples and oranges.

We use Gitlab at the office and it is constantly slow. Might be our setup I
agree, but in the end it also tarnishes the brand to me.

All in all, my general experience is that speed and stability has degraded
over the past years, in favor of the 'whole in one devops pipeline'. Github
stayed focus on its core market.

~~~
lima
>Github does only one thing, the CI is usually somewhere else so there have
less reasons to fail as well... It might be comparing apples and oranges.

This is changing with Github Actions.

~~~
jlengrand
> This is changing with Github Actions.

Well gitlab does artifactory/docker management, security, value stream
management, . . . So still

~~~
manigandham
GitHub now has package registries and security scans too, and adding more
features to complete the lifecycle.

~~~
jlengrand
Wasn't aware. Will look into it. Thanks!

------
bvm
I love gitlab, I really do, but their uptime is atrocious for a multi-billion
dollar company.

------
PhilKunz
It seems like GitLab should indeed focus on stability now. The feature set is
great. But it is all not worth it if it keeps disrupting work on a constant
basis by being unstable.

~~~
toopok4k3
I recommend hosting it yourself, works great and is fast on local network. You
can run updates when it's convenient to you, not to others.

GitLab is a rare software that enables you to be in control.

~~~
PhilKunz
I've always been a little cautious of running it myself. When I look at the
components involved there are a lot of moving parts. And I don't want to be
ever in a position where I can't get things started again. If I'm running on
premise I'll switch to gitea with drone ci instead.

~~~
niceworkbuddy
GitLab has virtual machine image, so it's really unobtrusive for your system.

~~~
jeltz
They also have a Debian package with sets up the whole environment. It has
worked well for me so far. The only thing I had to add was a backup script.

~~~
apple4ever
The omnibus package makes it real easy to install, especially with Ansible.

FYI if you are uploading to AWS or other cloud providers, you don’t even need
a backup script. You can configure it in the gitlab.rb file:

[https://docs.gitlab.com/ee/raketasks/backup_restore.html#upl...](https://docs.gitlab.com/ee/raketasks/backup_restore.html#uploading-
backups-to-a-remote-cloud-storage)

------
tyingq
Updated just now...bad firewall change.

 _" [Identified] We have identified firewall misconfiguration was applied that
is preventing applications from connecting to the database."_

------
unilynx
It was already a bit flaky since last friday (although the status was still
green then) - perhaps having to roll out two security updates yesterday pushed
it over the edge...

------
the-dude
One starts to wonder if they will ever get stability under control.

------
BlackLotus89
It was an iptables problem

[https://gitlab.com/gitlab-com/gl-
infra/production/issues/142...](https://gitlab.com/gitlab-com/gl-
infra/production/issues/1421)

The rollout of new iptables rules blocked database server connections

------
bArray
> [Identified] We have identified firewall misconfiguration

> was applied that is preventing applications from connecting

> to the database. We've rolling back that change and expect

> to be operational again shortly.

Heh, happens to the best of us! Seems to be coming back online now.

------
aabbcc1241
Any service on clearnet can go down. p2p alternative like git center (git over
zeronet) is more resilient to service provider mul-configuration and
censorship

git center:
[http://127.0.0.1:43110/1GitLiXB6t5r8vuU2zC6a8GYj9ME6HMQ4t/](http://127.0.0.1:43110/1GitLiXB6t5r8vuU2zC6a8GYj9ME6HMQ4t/)

client: [https://zeronet.io/](https://zeronet.io/)

proxy: [https://zero.acelewis.com/](https://zero.acelewis.com/)

------
intellix
Not sure if I have anything else to add on top of what people are already
saying but the performance of GL is extremely frustrating for the past couple
of months. Viewing the diff of a PR can sometimes take around 20 seconds just
to load the tab.

I think it's about time they dedicated some resources or addressed the public
about the efforts they're making to address these issues. It's becoming
excruciating.

I've used Github every day for years and years and I feel like every page load
has been pretty much instant forever.

------
greens231
Originally joined gitlab for their free private repos but with the recent
downtime/sluggishness, i have jumped over to github (now that they offer free
private repos too)

~~~
Dayshine
Gitlab is still the only viable choice for non-commercial groups who want
private repos though, the 3 private members and no ability to have mixtures of
public/private repos in organisations on Github is very limiting.

~~~
scaryclam
Sorry, but this isn't true at all. Bitbucket works fine, and GitHub offers
plans for non-profit groups for free:
[https://github.com/nonprofit](https://github.com/nonprofit)

Self-hosting has multiple different options as well.

~~~
Dayshine
Registered non-profit <> non-commercial.

\- a private website for my local sailing club. \- a mod for a game \- an open
source project that requires a private repository for a few things \- any
project relating to a private community

None of these are registered non-profits.

Bitbucket is capped at 5 users as far as I can see, and self-hosting is just a
recipe for lost data. I don't know of many amateur groups that can safely host
a server.

------
the-dude
It is worth mentioning their status page is not down.

~~~
spzb
It looks like it's externally hosted at status.io. It makes sense to have your
status page on separate hosting otherwise it can go down at the same time as
your main infrastructure and that defeats the point of having a status page.

------
softwarelimits
Hmmm, just today I wanted to install a new virtual machine with the latest
Gitlab release to check if it makes sense to run that inhouse... does anybod
know if there is a VM, a vagrant machine, an ISO or a repository online that
still can be used? Thanks!

Or does exist a mirror on github.com? ;)

~~~
andrelaszlo
If you just want to try it out, the easiest is often Gitlab Development Kit.

    
    
        gem install gitlab-development-kit
        gdk init
        cd gitlab-development-kit
        gdk run
    

I doubt it will work today though; it's probably pulling everything from
Gitlab.

For Docker, check out [https://hub.docker.com/r/gitlab/gitlab-
ce](https://hub.docker.com/r/gitlab/gitlab-ce) I haven't tried it out for
myself but it seems popular :)

~~~
leipert
The GDK installation has changed a bit (just set it up today, you need to run
gdk install and gdk start (instead of run) for example). Please also note that
it involves installing dependencies on your local machine, like e.g. Ruby /
Postgres, etc. Here is the link to the GDK: [https://gitlab.com/gitlab-
org/gitlab-development-kit](https://gitlab.com/gitlab-org/gitlab-development-
kit)

Otherwise I would try the docker container OR just install it with omnibus in
a VM: [https://about.gitlab.com/install/](https://about.gitlab.com/install/)

~~~
andrelaszlo
Oh, thanks. I got that from some old notes since my bookmark didn't work for
some reason ;)

For me, running Gitlab locally is extremely heavy (webpack especially). How
beefy is your computer, as a Gitlab developer?

~~~
leipert
I have got a 16 GB MacBook from 2018.

Funny that you mention Webpack, I am part of the webpack working group and we
are trying to improve developer experience:

[https://about.gitlab.com/company/team/structure/working-
grou...](https://about.gitlab.com/company/team/structure/working-
groups/webpack/)

We were able to reduce memory consumption quite a bit, but we are working on
different other improvements.

~~~
andrelaszlo
Great! I see you were actually involved in one of the problems that bit me (8
GB...). [https://gitlab.com/gitlab-org/gitlab-development-
kit/issues/...](https://gitlab.com/gitlab-org/gitlab-development-
kit/issues/458)

Small world :)

------
castis
oof, literally all of it. dont see that too often. on thanksgiving day no
less.

i root for gitlab when i can, but theres a reason i mirror my repos to github.

------
m712
Ironically, status.gitlab.com takes upwards of a minute to load on my end. I
thought it was going to time out.

------
hieudang9
I hope this incident is not #HugOps again, waiting for new interesting things
from postmortem

------
joeblau
We’re doing this now? For my entire tenure on HN, this site has been a status
page for GitHub. Looks like it’s Gitlabs turn.

------
ifthenelseend
You guys should consider switching to GitHub. Much faster, no 500-errors, less
downtime.

------
sleepnow
Microsoft must have migrated it over to run on Windows Server.

~~~
looperhacks
Maybe you mixed something up. This is about Gitlab, not Github.

------
pearjuice
I don't want to hijack this thread but what was the end result of the decision
of Gitlab not hiring _in_ certain countries anymore? A lot of media coverage
when the incident happened but no idea what happened afterwards and can't
easily find it.

~~~
the-dude
IIRC, they are not hiring 'in' certain counties. This is different than 'from'
certain countries.

AFAIK, nothing material happened after the announcement. There are lots of
companies which do not hire in certain countries.

------
sojmq
This strikes me as odd, because if there's something gitlab engineers are
known for, is their unmatched competence,

~~~
jensvdh
Like when they dropped their database and took a few days to recover?

~~~
YorickPeterse
It took about 24 hours to recover, not "a few days".

