Indeed. If there's one thing I've learned in >10 years of building large, multi-tenant systems, it's that you need the ability to partition as you grow. Partitioning eases growth, reduces blast radius, and limits complexity.
I believe GitHub is or was distributing repos instead of files. AWS code commit likely stores pack files in S3 or otherwise takes advantage of existing AWS systems..
Using a single Ceph FS is an odd choice IMO. You need to have or acquire a ton of expertise to run stuff like that, and it can be finicky. I'm not convinced you can't get this running well enough on AWS though.. So I'd be worried the move to bare metal would just cause the team more problems.
