
FreeDesktop.org financial situation regarding cloud hosting for Gitlab - mroche
https://lists.freedesktop.org/archives/wayland-devel/2020-February/041232.html
======
mrpippy
There's more details 2 messages later on what the cost is for:

> Some clarification I got from Daniel in a private conversation, since I was
> confused about what the money was paying for exactly:

> We're paying 75K USD for the bandwidth to transfer data from the GitLab
> cloud instance. i.e., for viewing the https site, for cloning/updating git
> repos, and for downloading CI artifacts/images to the testing machines
> (AFAIU).

And more details on their infrastructure in general is at
[https://www.freedesktop.org/wiki/Infrastructure/](https://www.freedesktop.org/wiki/Infrastructure/)

gitlab.freedesktop.org is hosted in GCE us-east1. One runner for CI jobs is
also in us-east1, but there are also (donated) runners at Hetzner and Packet,
and a lot of bandwidth goes toward communicating with those. Apparently even
if they disabled CI, it would still be $30K/year, I guess for hosting and
web/git traffic. Not a good situation.

~~~
Slartie
This is laughable. Just renting another box at Hetzner with the specs
necessary for their Gitlab would be around $100-$150 per month. That is maybe
1-2k per year. With 10tb/month traffic included already, IIRC, additional at a
very reasonable price, and traffic to their already existing Hetzner runner
box being non-metered to begin with.

But woooh, that's probably not cloudey enough.

------
anonsivalley652
Why don't they host it themselves in a colo?

Doing like Netflix did by paying AWS millions is wasteful and insane when
managed colo is cheaper and self-managed colo is cheaper still. ByteMark.co.uk
and pair.com are both great vendors that could save them money.

~~~
jcrawfordor
Colocation would shift a good portion of the annual expense to an up-front
investment in hardware (as well as setup, shipping, install fees). It would
also increase the required admin time.

It's quite possible that they could save some money by going this way, but
it's by no means going to just fix their problem. In a way it would exacerbate
it due to the capex for hardware.

~~~
StavrosK
They don't have to buy their own servers, renting from Hetzner would probably
be much cheaper than AWS or Gitlab, and you can automate installing the runner
on the server so your administration needs get reduced to basically "reformat
the server and run this provisioning script".

I'm sure there are things I'm not taking into account, but it doesn't seem
that bad.

~~~
jcrawfordor
Their issue is GitLab itself, not the runner. And sure, Gitlab is available in
forms like a Docker image, but as someone who managed Gitlab for years... it's
reasonably high effort.

Additionally, I would not trust Hetzner with anything that gets this much
usage. They're very cheap, and they're very cheap for a reason.

When buying colo for a busy application, it's not unusual for bandwidth to
cost more than the power and rack. All in all I'd expect them to be looking at
at least $2k a month or $24k a year for power/rack/bandwidth. This would be
something like an average, there are lower rates available in markets with
cheap connectivity but then if the equipment isn't local you end up spending
more on smart hands. In many places the rate would be higher.

Then up-front hardware purchase could easily be $50k and probably more. I
wouldn't expect to see savings at all the first year, I'd expect it to cost a
chunk more. Of course that's not quite fair, the hardware purchase amortizes
over at least a few years, but when you're struggling to pay the bills that
doesn't make you feel any better.

Renting hardware is an option, but will still cost you more than colo on year
for year for equivalent capacity. Dedicated servers from a reputable source
start at a couple hundred a month and quickly go up from there. It sounds like
their situation will require HA.

I'm just waving my hands here based on my guess of how much usage they're
seeing for the $75k they estimate in AWS spend, but I have a quote in my inbox
right now from a reputable datacenter operator for $18k per year just for
rack/power/x-connect, that's before bandwidth, which I'm going to be paying
two different providers for for diversity (and this is still worse
connectivity than AWS gets you). That's a full rack, which is a lot of room,
but the power/xconnect/bandwidth part of that basically don't change as you go
to partial racks so the savings are not really that big. Quality colocation
costs real money, and a gitlab instance with that kind of usage shouldn't be
running out of a bargain-basement provider.

~~~
StavrosK
That's very informative, thank you. It's still a third of the price, though,
if your calculations are ballpark correct, which is nothing to scoff at.

------
stonogo
Maybe it's time to acknowledge that fd.o is primarily a Red Hat operation, and
ask Red Hat to step up and serve the community?

~~~
astrodust
You mean IBM which could give this project a huge cloud services credit with
the click of a mouse.

------
CameronNemo
This is exactly why Void Linux uses GitHub and Travis. Hard to pay for one's
own CI.

------
chx
> With reasonable estimates for continued growth we're expecting hosting
> expenses totalling 75k USD this year,

How does that happen? My understanding here is the servers are doing something
ephemeral and so... uh, why this isn't on Hetzner SB? I do not know of course
whom they are using for hosting but 35 USD gets you a decent quad core 32GB
RAM server there which is very hard to beat.

~~~
earenndil
Problem is not server specs, but bandwidth.

~~~
chx
Great, Hetzner has unlimited bandwidth (well, 320TB since the port is 1gbps).

~~~
robjan
If they start using 320TB / month they will quickly enter "Fair Usage Policy"
territory. At that rate of bandwidth usage, Hetzner would be taking a massive
loss and other clients would probably be impacted.

~~~
chx
[citation needed]

[https://www.hetzner.com/news/traffic-
limit/](https://www.hetzner.com/news/traffic-limit/)

that's not what this says. Nothing at

[https://www.hetzner.com/rechtliches/root-
server/](https://www.hetzner.com/rechtliches/root-server/)
[https://www.hetzner.com/rechtliches/agb](https://www.hetzner.com/rechtliches/agb)

In fact, people have asked support before

[https://i.imgur.com/usd27c2.png](https://i.imgur.com/usd27c2.png)

------
lucb1e
Would switching from GitLab to something like Gitea/Gogs cut down on those
costs? The GitLab system requirements start at multiple gigabytes of RAM and
CPU cores for almost idle installations whereas a small Gitea I'm running
takes <100MB RAM and next to zero CPU. Most people I know that self-host (i.e.
privately, not commercially) a git server with issue tracker don't run GitLab
due to the resource hogging. But I don't know how it scales when you put more
load on Gitea vs the same load on GitLab -- does anyone have experience with
that?

~~~
stonogo
It says in the message it's storing and serving build artifacts that is
driving costs. This means the expense is in disk and bandwidth, not CPU and
RAM. Switching to Gogs would cut down on the resources which are not
identified as the problem.

~~~
tiew9Vii
They probably could use some cheap blob storage like Backblaze B2 which is
very cheap ($0.005 GB/month storage, $0.01 G download) for storing artifacts
then put Cloudflare in-front of it.

If artifacts are treated as immutable most of the traffic will be from the CDN
and as they are immutable can have a far future cache-control.

------
drenginian
Digital Ocean bandwidth is 1 cent per gigabyte.

