On: Moving persistent data out of Redis

[coldcode]

I wonder if this will allow better scaling of GitHub enterprise. We are pegging our usage; if we could we would migrate everything to Gitlab Enterprise (which we also have) which seems to have better scalability.

[sytse → coldcode]

How can we help you move to GitLab EE? (As you indicated it scales to 100k users so that shouldn't we the problem)

[john832 → sytse]

Does GitLab really handle 100k users? Where is this indicated?

https://gitlab.com/gitlab-org/gitlab-ce/issues/26405#note_20...

[sytse → john832]

Thanks for asking. The issue you referred to talks about users on a single machine. Unlike GitHub you can run a cluster of application servers with GitLab https://about.gitlab.com/high-availability/

Some of our users have 25k+ users on their cluster. We know GitLab can scale to 100k users because we run GitLab Enterprise Edition without modifications on GitLab.com

GitLab.com currently has much more than 100k users and the performance leaves much to be desired https://gitlab.com/gitlab-com/infrastructure/issues/947

But we're comfortable that you can run 100k users on a cluster of machines without much tuning.

[john832 → sytse]

> Unlike GitHub

GitHub seems to have a clustering product that came out a year ago, and it looks like IBM has reported they're running over 13,000 users on it (back in August, i imagine it's closer to 20,000 by now): https://www.ibm.com/blogs/bluemix/2016/08/ibm-internal-githu...

> We know GitLab can scale to 100k users because we run GitLab Enterprise Edition without modifications on GitLab.com

What does "modification" mean in this context? Beyond recommended specifications listed at https://docs.gitlab.com/ee/install/requirements.html#cpu and https://docs.gitlab.com/ee/install/requirements.html#memory? it doesnt list anything above 40,000 users. Beyond that, the HA documentation (https://docs.gitlab.com/ce/administration/high_availability/...) isn't _really_ active/active HA, it says it is but it's not true. True active/active would mean that you wouldn't rely on a shared NFS server, postgres, or redis server.

However, with https://help.github.com/enterprise/2.8/admin/guides/clusteri... it sounds like my organization can scale to well over 100,000 by adding more clustering nodes instead of trying to figure out independently how to scale services like redis or postgres or a shared NFS server.

> without much tuning.

I'd love to hear a comparison between the two products, and also what kind of tuning you've done on gitlab.com to support those user numbers. Would love to see that on the documentation to support the open source way!

[marcinkuzminski → sytse]

can you share how many machines handles gitlab.com currently?

[sytse → marcinkuzminski]

It is around 100 of machines. Commonly it is a couple of thousand active users per application server, but your mileage may vary based on many things.

[gjtorikian → coldcode]

> which seems to have better scalability

Out of curiosity: how many users do you have, and what measurements are you using to determine scalability?

Disclaimer: I work at GitHub.