
Degraded performance on GitLab.com - sexy_devil
https://gitlab.com/gitlab-com/gl-infra/production/issues/928
======
samcday
I recently started a new position at a company that is using Gitlab. In the
last month I've seen a _lot_ of degraded performance and service outages
(especially in Gitlab CI).

And then I see stuff like this that makes me scratch my head:
[https://about.gitlab.com/product/service-
desk/](https://about.gitlab.com/product/service-desk/)

If anyone at Gitlab is reading this ... please, _please_ slow down on chasing
new markets + features and just make the stuff you already have work properly,
and fill in the missing pieces. Example: [https://gitlab.com/gitlab-
org/gitlab-ce/issues/63880](https://gitlab.com/gitlab-org/gitlab-
ce/issues/63880)

~~~
kennyGitLab
For a bit more about WHY we choose breadth over depth - We believe that the
company plowing ahead of other contributors is more valuable in the long run.
It encourages others to contribute to the polish while we validate a future
direction. You can read more on our company strategy page -
[https://about.gitlab.com/company/strategy/#breadth-over-
dept...](https://about.gitlab.com/company/strategy/#breadth-over-depth)

As open-source software we want everyone to contribute to the ongoing
improvement of GitLab.

~~~
rightisleft
"We're Open Source!" isn't a valid defense when you have paying customers.
That pitch sounds great for your VCs, but for someone who spends a portion of
their budget on your cloud services - i'm appalled.

Gitlab is a SaaS company who also provides an open source set of software. If
you don't want to invest in supporting up time - then don't sell paid SaaS
services.

~~~
godzillabrennus
Your logic and reason have no place in VC land.

They need more users!

------
nimbius
I know this is probably a controversial post but seeing how big and bloated
Gitlab became after years of devops dragon-chasing, i switched to gittea.
[https://gitea.io/en-us/](https://gitea.io/en-us/)

I keep separate CI and a separate wiki, so i can upgrade them independently
and not have the entire software development process grind to a halt whenever
I need to patch.

~~~
tapoxi
I don't think that's controversial, I think it's different use cases. If I
want a simple git repo I'll go gogs/gitea, but GitLab is an all-in-one tool
for a software shop.

------
grafelic
Indeed. Incredibly slow.

Status page: [https://status.gitlab.com/](https://status.gitlab.com/)

Issue: [https://gitlab.com/gitlab-com/gl-
infra/production/issues/928](https://gitlab.com/gitlab-com/gl-
infra/production/issues/928)

------
Kovah
Even though my builds won't start as runners seem to be blocked, it's kinda
fascinating to see those in-depth details about the service and how the Gitlab
dudes try to find the root cause.

~~~
penagwin
Right? This is EXACTLY what I want to see when there's a service disruption.

A live, in-depth view of who is doing what, any new leads on the issue,
multiple teams chiming in with various diagnostic stats, honestly it's really
awesome.

I know this can't be expected from most businesses, especially non-open
sourced ones, but it's so refreshing to see this instead of the typical "We're
working on a potential service disruption" that we normally get.

~~~
brbrodude
I think it's cool but I'd be totally self-aware and fearful or making the
idiot post in the thread for HN to judge :p

~~~
penagwin
It's certainly something I'd be scared of too. It's honestly really brave of
them to approach it this way, but I do enjoy watching them work :D

------
umvi
GitLab is best (imo) as a private instance; we haven't had any issues with it
after using it for 3 years now

~~~
swozey
There have been a number of updates in the last 3 years that would have
completely broken or caused 500 errors on your Gitlab instance if you
immediately updated it prior to a hotfix being released. If you didn't
encounter these I would be shocked or you're keeping off bleeding edge
releases (smart). I've managed private gitlabs for 4-5 years. I've submitted
~5-10 tickets regarding various things breaking (gitlab-runner, pages 500ing
after an upgrade, etc).

If you give each update a few weeks/month and pay attention to the comments on
each release blog you'll save yourself quite a bit of headaches.

~~~
maverwa
I found that the good old “skip the .0 release” goes a long way.

~~~
lenovouser
Exactly what I am doing too, always wait for the next patch release after a
x.0.0.

Before doing that I had two situations where the major release broke
something. Never looked back.

------
jhgg
Running Redis on GCP? I’ve seen CPU and latency spike on our production Redis
nodes before - solely due to noisy neighbors causing memory bandwidth
contention.

------
kabwj
Funny. Gitlab has always had terrible performance for me, both the website and
the git server.

------
awasum_yannick
Thank you Gitlab for the amazing product and transparency. Just watching the
conversation as everyone over there is trying to get to the root of the
problem is inspiring. What a strategy.

------
wolf550e
Is the conclusion that redis doesn't handle caching items that reach 99MB size
each? I guess I would cache them in blob storage, not in RAM. And make sure to
zstd them.

------
yardie
Not sure if this is related but I've received a bunch of emails from Gitlab
stating my account is locked for too many failed password attempts. Possible
DDoS?

~~~
emilycook
We haven't had any other reports of this, please submit a ticket here if you
think there's an issue so we can track it: [https://support.gitlab.com/hc/en-
us/requests/new?ticket_form...](https://support.gitlab.com/hc/en-
us/requests/new?ticket_form_id=360000515493)

------
baq
you have a convoy.

------
bachmeier
I was trying out Gitlab pages for the first time, but gave up since it was so
slow, moving my stuff to Github. Guess it's not always that slow...

~~~
Kovah
I can say for sure, that Gitlab is not running _that_ slow all the time.

However, the services run noticeably slower when the US awakes (I'm located in
Germany), which is in the afternoon and evening around here. I have no stats,
but it feels like builds and the site run smoother in the morning.

~~~
emilycook
In case you're actually curious, we do publish our grafana stats here:
[https://dashboards.gitlab.com](https://dashboards.gitlab.com)

