Hacker News new | past | comments | ask | show | jobs | submit login
Gitlab ‘rethinking’ third-party telemetry (gitlab.com/gitlab-org)
623 points by EMM_386 on Oct 24, 2019 | hide | past | favorite | 422 comments

This thread follows on yesterday's: https://news.ycombinator.com/item?id=21337594

The negative comments seem largely focused on issues with telemetry per se (which is a big deal, especially in the enterprise), but, for my company, I see an even bigger problem. There is no way that I would host a solution that includes third-party scripts. This is simply not negotiable. I don't care how good that third party's "data protection" is -- the ability to serve up that script means that anyone who compromises the third party, even just a temporary compromise of their front end, has the keys to the kingdom.

If you want to collect telemetry, fine. Do it on a first-party basis, distill it down into one file a month, and let admins upload it if they like. But the third-party script is a complete nonstarter.

> There is no way that I would host a solution that includes third-party scripts.

At least for now, it appears that the self-hosted versions of GitLab (both CE and EE) will not be getting telemetry scripts. This is only for gitlab.com (again, for now). See the table on their ticket[1] for the issue

1: https://gitlab.com/gitlab-org/growth/product/issues/164

From that link, it doesn't exactly sound like they've ruled out the possibility of adding this tracking to EE too however - and reading between the lines, it sounds like they'd prefer telemetry to be enabled by default if & when that does happen?

We are not adding Pendo or Snowplow JS snippets to self-hosted versions at this time. We are carefully considering the appropriate opt-out functionality for telemetry that will work best for our customers, and we will clearly communicate to those customers when any change is made.

Right. That basically says "We're going to back off until people stop paying attention, then ram in the anal probe".

> opt-in by default

So ... opt-out?

Thanks :) Comment edited to (hopefully) clarify the language.

it seems pretty clear they intend to add it to EE eventually. Every single time they talk about it, they say, "we're not considering changing EE at this time".

> At least for now, it appears that the self-hosted versions of GitLab (both CE and EE) will not be getting telemetry scripts.

Yes, for now. But I think everyone can realistically expect that it will be included there eventually.

Those of us who find this unacceptable should probably start planning their mitigation strategy now.

The entire nature of self hosted is that it is for companies or people who want more control. It’s Seems ill advised and unlikely for gitlab to force telemetry on those users.

> It’s Seems ill advised and unlikely for gitlab to force telemetry on those users.

I would have thought that the way they rolled this out was obviously so ill-advised that they wouldn't have been likely to do it, but they did.

Also, they initially were going to include the self-hosted installations in this as well, but backed off due to the outcry. So even though it's also obviously a bad idea, they were provably likely to do it, because that was their explicit plan.

Also, it's a red flag in any security review. Every time we answer security questionnaires this comes up. I'm stunned they would consider adding this to hosted software.

Or just opt-out, if they ever add it, and if it is not opt-in at that time.

Yes, for those who really need to stay on Gitlab.

Where my thinking is at right now, though, is that Gitlab has just tangibly demonstrated that they don't deserve a great deal of trust, and so I'm not inclined to trust any opt-in or opt-out choices.

Even if they're effective at the time they are introduced (which I don't really doubt they would be), I am not comfortable in relying that they won't change the deal in the future.

Opting out now won't matter in the future when they reset the default and then reset everyone's configuration "to let you make the choice with new information" or whatever spin they want. Then you miss that new choice in your automatically-deployed update and... well then what?

I guess if we’re making up things that could happen, why bother even giving them credit for providing a choice at all?

Indeed there is little point to care about a choice when the trust for them to uphold that preference doesn't exist anymore.

I agree completely that a company's most precious asset is the trust of their customers. If you fundamentally lose / abuse that trust, eventually you will have no more customers.

Source control is a particularly good example when it's the core IP of your company at stake, and engineers--who can sometimes be a bit prickly about these things--are making the decision on which product to use.

In this case the reaction to me seems overblown, as Gitlab AFAIK has been a well run and particularly transparent company. I felt the same way about the recent reaction to Gitlab saying they want to stay out of the politics of passing moral judgement on every repo they are hosting, which to me seemed exactly right. To be clear, I'm not and have never been a Gitlab customer, so I'm not basing any of this on any personal experience the quality of their product.

Now speculating on how they might do something in the future which impacts self-hosted instances, assuming the worst, and declaring it will mean switching providers... I think the comment would be better off just saying something like; "I'm looking at the way Gitlab is handling this issue and it makes me no longer trust them with my data, so I will be going elsewhere" -- just without the rank speculation.

> let admins upload it if they like

If that was the approach to telemetry, I think most users wouldn't have a problem. But currently the word alone makes us rethink about using the product at all.

Indeed. Back when Windows asked me if I wanted to send info about an error to Microsoft so they could improve Windows, I pretty much instinctively clicked yes. Now that they collect the information without my permission, I block it at the network level.

Take a look at your enterprise DNS and/or proxy logs and see how many times secure.gravatar.com is being referenced. It's leaking all sorts of info via HTTP Referrer (looking at you Atlassian).

Let's not call it "telemetry", let's call it what it is: spyware.

Malware is even more impactful and still technically correct since it is a superset of spyware.

I worked on a project that involved customizing an open source document management system for government use. One of my first assignments was removing the built-in/hard coded tracking baked into Alfresco:


This is spot on, in fact, GitLab’s telemetry service provider (Pendo) offers the ability to self host their scripts (although they don’t encourage this model).

It would be nice if GitLab provides more transparency on how they are deploying these third party services.

There are technical solutions that allow a site to include 3rd party JavaScript for tracking/analytics while keeping it completly sandboxed so you don't have to trust the JS.

One large website I know includes a sandboxed <iframe> from a different domain on all logged out pages that pulls in 3rd party scripts for marketing purposes. Special data you want the 3rd party to have needs to be passed to the iframe with postMessage from the parent, but the key feature is that compromised 3rd party JS won't be able to wreak havoc on your site.

Here's some info on that approach: https://cheatsheetseries.owasp.org/cheatsheets/Third_Party_J...

That makes sense but I don't think any third party would pay to be included in a iframe and fed such a limited amount of data, and if you aren't paid for it why bother making all of your users mad in order to do it?

I think we have different use cases in mind. I'm thinking of services which are free or which companies pay for to gain access to. Things like Facebook/DoubleClick for ad-retargeting and Google Analytics.

Agreed that this does nothing for users that don't want to be tracked, but at least it reduces the risk of the site being compromised by untrusted JS.

My bigger problem is that gitlab wouldn't allow me to register user account with yandex email, saying 'email is not from an allowed domain'

I wonder what was the cost difference between SaaS solution and implementing telemetry themselves, which wouldn't upset most users.

What's the difference to you between a second-party script, and a third-party script?

With the second party (Gitlab in this case) I have a contact, give them money regularly, and have other such leverage in case they screw up. Third parties generally could not case less what damage they may cause.

But what's the difference? You PAY gitlab in both instances and your leverage is the same. Do you want to be involved in reviewing their motherboards for spyware chips as well?

If you are self-hosting the product, you can control changes to the environment. If you are pulling in third-party scripts on the fly, then that control is gone.

Ah I guess the real problem is 'on the fly'.

The problem is that the resource the URL points to resides on a system that's neither under your control, not under counterparty's.

No difference really, but fourth party scripts are the worst :-/

Damn right. I think most companies should start to filter 3rd party data collection on the network level.

> There is no way that I would host a solution that includes third-party scripts.

How is this different from allowing corporate users to access web apps that have google analytics running? We've seen some enterprises block GA, but they are in the minority.

The difference is the third party script has complete control over the page it’s on. So if the page is your gitlab instance, the third party has complete control over your repositories and probably your complete infrastructure. Just introduce some malware to the build scripts and you’re in.

GA is the same in these ways, isn't it?


In what universe would javascript running on a webpage be able to access (nevermind modify) the filesystem on the server.

As the parent comment describes: You're using Gitlab as a web interface to change code on your servers. Javascript running on the page of that web interface can do almost anything you can do with that page, by producing the same events.

(actually, browsers have a bunch of tools to restrict that, so it's not 100 % if engineered properly. But if it's just "include some scripts"...)

Hey, I'm building an on prem solution and had a question regarding this. Would it still be a problem if the third party was someone like segment whose open source analytics library you're using? https://github.com/segmentio/analytics.js

Their problem was third party scripts and you are asking if a third party is okay? Am I missing something here?

Well from what I understand it's that third party scripts are a problem because they may behave maliciously and gain access to parts of the application. If the third party script is an open source project, doesn't that mitigate this?

Doesn't prevent a malicious/compromised third party from serving code other than what's in the source. I think an acceptable mitigation might be subresource integrity though, so you can lock it to a known-good version of a script?

> So, for users who have integrations with our API this will cause a brief pause in service via our API until the terms have been accepted by signing in to the web interface.

What a fucking botched way to do this. Breaking all automation at a moment notice. They should have announced the change long ago. Make an advance clear deadline for accepting the ToS and then after the deadline blocked the API.

These are the sort of stunts that happen when you relinquish more and more control of your technology and its ownership to external sources that also happen to be for-profit businesses. All hail EaaS (Everything-as-a-Service) models.

I entirely agree with you and your sentiments, but I'm not sure why many are even remotely surprised by these sort of escapades. The more dependent you are on an external business for some function and the more that business realizes it, the more they're going to use their leverage against you to their advantage to achieve their goals (not yours).

That's exactly what we see happening here. When businesses (or any entities) buy-in to external resources like this, always consider: how much leverage those managing that resource have against you, how critical the function is for you, and how many resources it will take to migrate away from that external resource at a moment's notice if you need to. This is a continuous iterative process that should be going through every developer's head with every design choice, external license, "cloud" service, proprietary internally hosted licenses, etc.

The profit motive isn't the issue. Even if you don't relinquish control to a VC, you'd still [likely] have a profit motive yourself.

The issue isn't profit. You might say the issue is greed, but more importantly, the issue is a lack of regard and compassion for your customers, and a lack of understanding of the danger this puts you in [of losing your customers and, with them, revenue and profit].

User empathy, if you will.

> but I'm not sure why many are even remotely surprised by these sort of escapades

I don't think people are surprised that these things are possible in the context of "EaaS". It's surprising because they're causing massive problems for their users by making the change so quickly. The relative gain of waiting one month before the change wouldn't hurt them that much, but it would help their customers a great deal. So the surprise is seeing how indifferent they seem to the users. (I know they're revising the decision now. Maybe they can avoid most of the bad PR.)

The free or partially free dev tool industry is extremely competitive and you need a lot of goodwill to stay alive. They're completely free to screw their users, and their users are completely free (except for some lock-in) to hop to the next provider. I don't understand why they would throw away that goodwill.

But maybe I'm wrong and the friction is higher than what I expect.

> But maybe I'm wrong and the friction is higher than what I expect.

It may be for a lot of users. But some people are GitHub refugees and only started using Gitlab relatively recently. I am one of those, and haven't been using Gitlab for long enough to have any serious friction against moving away from it.

> This is a continuous iterative process that should be going through every developer's head with every design choice, external license, "cloud" service, proprietary internally hosted licenses, etc.

The nature of tech trends will ensure that developers never look beyond marketing material that espouses how easy and cool the new shiny tech is.

I'm watching developers go all in on serverless as if the generous and copious amounts of free compute that cloud providers are granting on their serverless platforms will always flow like water.

> The nature of tech trends will ensure that developers never look beyond marketing material that espouses how easy and cool the new shiny tech is.

Not all developers. Some of us have been around the block enough times to be automatically suspicious of marketing material that leans too hard on the "easy and cool" messaging.

> ....external sources that also happen to be for-profit businesses.

Pretty sure a not for profit organization would have even less interest in pleasing their users.

Maybe, but a cooperatively owned and managed business would have complete interest in its members and their values.

What interest do they have other than that?

Depends, it could be just to disrupt all other competitors on cost and not care how miserable the actual experience is for the customers.

Many not for profits exist mainly to stroke the directors’ egos and pay them a fat salary without doing much actual good (e.g. most political non-profits like the Clinton foundation).

Generally, they do have much more interest in pleasing their users. Not only do not-for-profits typically have some kind of explicit charter, it's usually for some social benefit, like "helping users". If anything; they're too helpful, and won't bash users over the head if they're being truculent, making them slow and crufty. Additionally, not-for-profits do need some kind of revenue (typically), and that tends to come from donors, which are usually... users.

If you merely mean free stuff, not literally, not-for-profit... yeah, free but for-profit is pretty much a disaster waiting to happen.

Rant: Those business models by and large should probably be hit with the ban-hammer, immediately. Of course, given the complete and utter submission of congress and indeed the very organization of democracy to vested interests, the chance of actually seeing a functioning, efficient market in our lifetimes is essentially nil.

It's ironic that traditional communism succumbed perhaps partially due to game-theoretical inherent inefficiencies, and capitalism patted itself on the back without ever really looking in the mirror. So, yay, we get to be the human guinea pigs proving that markets aren't actually intrisincally efficient at all, only efficient within their constraints, which is pretty much worthless given how hard we've collectively been trying to saw the legs out from under this construct. Capitalism is itself one big tragedy of the commons.

What bothers me is how screwing users is becoming the new standard. For me trust in society is what enabled first world economies in the first place, instead of business being killed by people constantly looking for ways to screw you. I’m not naive, I own a business, and bad intentioned people exist and you must protect yourself, but assuming the worst in every transaction is not a normal way to think for society.

For my experience usually clients trust me, and I want them to succeed with my services, and contracts and lawyers are here only for the worst case.

I can’t put my finger on it or express it clearly but free market excuse « dissatisfied users can move, this corporation is responsible only for profit, everyone must lawyer up » don’t look like it’s the good direction for society. I don’t see why free market would be incompatible with an individual sense of responsibility towards society, instead of looking for loopholes to screw people. Maybe it’s a consequence of being global and having not community roots.

> For my experience usually clients trust me, and I want them to succeed with my services, and contracts and lawyers are here only for the worst case.

That you genuinely want your clients to succeed is probably why they trust you -- you earn that trust.

I'm of the same mindset. I genuinely care about the well-being of my customers and will always work to enhancing that (even to the point of recommending competing products if they are a better solution for a particular customer). I've never lost business in the long term by doing that -- even customers I refer to competitors remember me and have an increased level of trust in me, and more often than not will do business with me in other ways.

But I have an old-school attitude towards business. My goal is not to gain as much money as quickly as possible, but to build a solid business that is sustainable and profitable over the long term. In my experience, that attitude naturally aligns my business practices with societal responsibility.

Yeah it's bizarre. Why rush it like this? It feels like an artificial internal deadline to roll out the "feature", and they realized it violated the ToS at the last second.

I think the purpose is to force users to "agree" to the new terms due to a sense of urgency.

So this is an outage then.

Decided to post a top-level comment because I think it's very important that Gitlab do a "root cause analysis" of why they made such a poor decision. Just saying "we're holding off for now" is not going to convince me that you guys have not lost your compass.

For reference, there was a very telling comment by user jahlove about how Gitlab could be so out of touch (https://news.ycombinator.com/item?id=21349591):


> But someone up the chain of command thought we could get away with it

That person is Paul Machle (CFO):

"I don’t understand. This should not be an opt in or an opt out. It is a condition of using our product. There is an acceptance of terms and the use of this data should be included in that." [1]



That GitLab comment thread is very telling. Basically, here is what happened:

1. It's clear engineers had concerns about the feature, wanting it to at least have an opt-in.

2. But then the CFO, who clearly does not anticipate the backlash this will cause, basically just gives an "tough shit" response.

3. What's scariest IMO, though, is that the Scott Williamson, VP of Product, then replies "@cciresi if we follow Paul's guidance and just make this part of our terms and conditions, are we covered legally?"

I've seen this at many organizations in the past. It's generally called the HiPPO problem - the Highest Paid Person's Opinion. The CFO wanted this done (obviously for financial reasons, he's the CFO after all), but the VP of Product, who should be more "in the weeds" in terms of having the pulse of the customer, instead deferred to the CFO and tried to appease his desire.

Gitlab, I think your organization is off track. You need to make a broader statement that shows you have a real understanding of the problem than just "Oops".

It might have been better to simply reference the discussion without singling out the person involved. At the end of the day it was an organizational decision. It's to GitLab's credit that there's enough transparency to even see this; I don't think it's too cool to turn that into a finger-pointing tool. As a developer, we don't do this with bugs. Perhaps bad business decisions are not any different.

(For the record: I don't know this person, I am not this person, I'm pretty sure I don't know anyone at GitLab at all.)

I understand your viewpoint, and to be honest I was on the fence about posting names, but I decided to for 2 important reasons:

1. First, the info is totally public and easily viewable in any case.

2. I think there is a danger in just saying "It was an organizational decision." When everyone is "responsible", then no one is. And in addition, both of the people mentioned are senior execs at the company. Having to deal with public scrutiny is one reason they get paid more.

I do give GitLab a lot of credit for being transparent, and for (at least for now) backing off this decision. But I hope they are able to do a deeper root cause analysis of what went wrong. While obviously on a whole other order of magnitude, Boeing didn't fuck up because MCAS was a faulty piece of software, they fucked up because they let financial concerns take precedence over engineering ones.

EDIT: one other thing, since I brought up 2 folks who I think made poor decisions, I should have pointed out Lukas Eipert (@leipert), who was the engineer who originally brought up his strong objections. Even more telling, when Eipert was asked to re-review the code change, he explicitly asked to be taken off the review - his sentiment was clear.

The whole point of transparency is to facilitate accountability, so it does not help to shy away from accountability in service of encouraging transparency. Of course, transparency is definitely something to laud, but it's only half of the ideal.

You're putting pressure on the norm that organizations should be sharing this sort of thing. If the norm were strong, the pressure wouldn't keep people from sharing similar things in the future. But since the norm is weaker we need to be careful to nourish it, not pushing it harder than it can stand.

(I wrote more about this: https://www.jefftk.com/p/responsible-transparency-consumptio...)

This is a person in a C-level position. It's very much inherent in those positions to take responsibility in situations like this. If the comment had been made by a random fellow team-member without any authority, I would have been more inclined to agree.

The difference here is that bugs generally lack motive and agency. Bugs are often unintentional.

When faced with bad intentions, it is imperative that a community name offenders. It is, as others replied, an important distinction for accountability.

A CFO concerned about valuation and experienced in negotiating contracts should be thinking:

1) Fortune 1000 companies and the government will never accept this. I need to make sure the product team is designing a way to allow these customers to shut this off.

2) This is more risk than we will gain from tracking individual customers. We can't afford to indemnify our customers against these third-party tools. How will we address GDPR, CCPA and future regulations if we don't allow an opt-out?

An organisation cannot make decisions. At the end of the day, it is always a person. In this case, maybe it was the wrong person. What is the trouble in questioning this process?


Personal attacks aren't allowed on HN. They just make threads worse. Please don't.


This is not cool.

Yes, I did copy the posting of Paul Machle's name, and as a senior executive (likely worth at least 10s of millions at Gitlab's last valuation), he has a lot of responsibility here, and he should own up to his mistake.

But the point is still not to "blame" it on one or a couple people. If you look at that GitLab comment thread, it's clear that many GitLab employees had strong objections to the change: "I hereby declare my highest degree of objection to this change that I can humanly express." and "I agree with @leipert moral wise, I'm not happy with this change". So the question is why the company made the decision that ignored the right people and followed the wrong people (hint, it's HiPPO).

Firing Paul Machle won't fix GitLab's problem. GitLab needs to figure out how to change its culture so that it listens to the right people, even if they're not the highest paid.

I sincerely believe that our society would be a much more just place if C-level executives were held accountable for profit motivated decisions that harm the privacy and security of their users.

He is choosing to go override his engineers moral and legal objections in the hopes that it will make his stock go up in value by a few percentage points. This kind of greed motivated behavior is at the heart of nearly every evil brought upon society by silicon valley.

Do I expect my post to actually affect Paul Machle in any way? Probably not, but I really hope it does. Society will be better off without these greed-at-any-cost types having positions of power.

Imagine the backlash if we started posting things like "See this engineer here! They caused the bug that resulted in downtime and a loss of $10mm for the company" and attempted to Google-shame them for posterity.

I can't agree with the comparison. An accidental & unexpected software bug is not the same as an intentional, calculated, executive level business decision.

I think the officers should be scrutinized more than a lower level engineer. They are the leaders and set the tone fore the company culture

I agree with hn_throwaway_99, this is very concerning as to what path the company is on. But, I'll give them credit for realizing the error & trying to correct it.

That being said, I love GitLab's product. We use it in my workplace, but we self host the comunity edition, so this wouldn't have effected us.

I dont imagine I will ever be more than a CE level user, even in personal projects, but I can imagine the panic this caused for many companies on the hosted enterprise version.

Thanks, we started our retro 3:44pm Pacific, slightly before your post.

We looked into making it public at 4:05pm but by then it already had confidential security information and customer names in it.

There will be multiple lessons, right now we've identified 15 points to evaluate and 9 people contributed to the document.

this is about it and it's not much of a secret anymore internally, but what do you do? this is just the MR you found, but they're everywhere. unfortunately Sid has known about the HiPPO problem for some time but is preoccupied with everything else and for some reason just won't do anything about it (if it's not gotten to his level many times, someones hiding it from him), and people are getting sick of the tone deafness and surprise when things go wrong. GitLab used to be a mission and values driven company, but now values apply to directors and below and everyone else just kinda does what they want, lean on their title to pressure people, and when there's pushback they say you're blocking efficiency and results, then complain about you ¯\_(ツ)_/¯ you can see everyone was against this and raised red flags at every step for the past 5 months but of course that didn't mean anything and the rollout was horrible. the people working at GitLab are about as dedicated to your trust and success as it gets, but execs are eyeing that IPO really hard and everything else is whatever. also no offense to Scott but you can't run this like a big corp and a good place to start to improve the product as the new head of product is to fix the backlog of hundreds of functionality deficits and bugs and talk to your customers to see what they need, not get tracking. i don't know how we teleported to this place where getting tracking is a priority because otherwise you can't find ways to improve the product because the last i looked there's a backlog of really critical things going to like version 13

"Trust takes years to build, seconds to break, and forever to repair".

Why is this so important? It's important because a lot of that trust is ideological. I trust that the developers of 'git' itself won't be adding telemetry behind the scenes. I trust that the developers of 'cgit' won't be adding this stuff.

Because we're on the same page; despite the fact that theoretically the next commit that hits the repo could be the one that sends my keystrokes.

The first time you pull this sort of stunt you've instantly erased all of that good will and now you're forcing people to comb through your releases and conditions with fine teeth because you've shown yourself to act against user interests.

This doesn't scale.

That's why the backlash is bigger than you expect, not necessarily because of the magnitude of the change, but because you've positioned yourself as a threat.

The pushback from users made them rethink this change, here's a pending merge request to the blogpost: https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/...

""" UPDATE: Thanks for the feedback. There were many more concerns than we expected. We’re going to process the feedback and rethink our plan. We will not activate product usage tracking on GitLab.com or GitLab self-managed for now. We'll make sure to communicate in advance on our blog when we do have a new plan. """

You can tell a company has totally lost their heads up their asses when you get the, "Whoa! We totally weren't expecting <product decision> to upset so many people!"

Like really? Considering what people use your product for, you honestly didn't expect this to upset people? Great. Your product team is hopelessly out of touch.

That statement is for PR purposes; you can't take it at face value. Let me attempt a translation into plain English:

"I, personally, totally expected this, as did everyone who I talk to on a daily basis. But someone up the chain of command thought we could get away with it, and forced us to try and push this change. I'm going to use the pronoun 'we' to refer to the company in the abstract, and not anyone on the immediate team I work with, because I jolly well can't be throwing my boss's boss, who happens to also be a commisar from the large company that recently acquired us, under the bus."

> But someone up the chain of command thought we could get away with it

That person is Paul Machle (CFO):

"I don’t understand. This should not be an opt in or an opt out. It is a condition of using our product. There is an acceptance of terms and the use of this data should be included in that." [1]

[1] https://gitlab.com/gitlab-org/gitlab/merge_requests/14182#no...

Ouch, that whole thread under there is painfully oblivious:

> @cciresi if we follow Paul's guidance and just make this part of our terms and conditions, are we covered legally?

Earth to GitLab: You are not yet big enough to be at the "are we covered legally" stage. You are toast if that is your mindset.

When someone raises an ethical question and instead of addressing ethics, the company calls in the lawyers to address whether it's legal... your company is likely on the wrong track.

To be fair I’d say half the time I’m summoned into a discussion like that it’s because the person asking fully expects and wants me to say no to the proposal. Raising an issue with the plan of action set forth by someone more senior can be tricky to navigate politically.

On the other hand, if you need to find a hill to die on, this is probably a good one.

AKA "well it's not illegal," a lousy lower-bound.

Having been in a few of these kinds of discussions inside companies, it's super-fascinating to find one that is open to the public! It shows what is always true: that any "corporate decision", seemingly made by some sort of AI machine, was actually made by one specific (usually unidentified) person. Popcorn moment for sure.

Man, that reply in particular left the biggest impact on me. The fact that this c-level exec is making decisions like this in such a ham-fisted way would scare me away if I were a customer. There are a chain of repercussions as a result of this.

Twelve thumbs down and one middle finger. (I'm not a native speaker of emoji and I'm not sure what the 8 ball means). These are from employees?

Sure, but that just changes it to "someone up the chain of command is hopelessly out of touch". Still doesn't bode well.

Which large company was it that recently acquired GitLab Inc.? I must have missed that.

Nah, you didn't. I was mistaken - I thought I had read something about that recently, but it looks like it's just that they raised another round of funding last month.

Funding rounds are changes in ownership structure.

I suspect the think was more along the lines of: let's slip it in and hope nobody notices or at least the ones that notice don't raise a hue and cry about it.

And now that it's turned out to be false, the dominant strategy is to offer a half apology along the lines of "We're sorry, we didn't expect that it'd be such a big deal".

Incidentally, it's quite reminiscent of the Grace Hopper quote: "It's easier to ask forgiveness than it is to get permission."

> "It's easier to ask forgiveness than it is to get permission."

When people use that quote, I try to be charitable and assume that they don't intend its literal, face-value meaning.

But if they do, my response is: I will not forgive you because you're not truly repentant. This is all part of a strategy where you see what you can get away with, and you show no signs of repenting that overall strategy.

Hmmm... maybe I'm a hardass.

I don’t think that really applies to companies that are still living by the grace of their popularity.

Gitlab is heavily dependent on developers pushing their product to the enterprise, because any executive would be far more happy with Atlassian (e.g. cheap and it works)

> I suspect the think was more along the lines of: let's slip it in and hope nobody notices or at least the ones that notice don't raise a hue and cry about it.

"let's slip it in and hope nobody notices" == "mgmt heads up their asses"

That places them on the "companies I probably wouldn't be happy working for" list. Certainly there are folks there who tried to speak up about it who were likely railroaded.

That's the vibe I get on just about every one of Gitlabs new screed posts on here recently. Hopefully there actually is some employee consensus on decisions, though.

As a rule, the structures around product teams tend to discourage asking if it's better to not ship a thing. When you measure a team by what (and how many things) they ship, they are always going to default to shipping things.

The structure thing is spot on. Decisions like this tend to get made when management has found a way to essentially silence any feedback (by making feedback pointless). This is why I cringe whenever I hear the phrase "leadership team". Whenever I hear that phrase deployed it's almost always to diffuse responsibility for a bad decision so no single person has to answer for it. Which amusingly is the opposite of real leadership: real leaders actually accept responsibility for the decisions they make.

This is absolutely true. It's absolutely what killed Digg with Digg V4 IMO.

I can't possibly believe anyone who ever used Digg could have taken a look at Digg V4 and thought "Oh yes, a feed full of spam, just exactly what I come to Digg for". But the team had already invested a ton of work, and organizationally it would have been next to impossible to say "Shit, we fucked up, lets hold off". So at least I give GitLab credit that they backed off (for now).

This is... insightful.

It brings to mind a larger question of how we (as an industry) should be measuring developer and team productivity, though. “Story points” are largely designed to solve this issue, but in my experience they are quickly co-opted to mean something different to each stakeholder.

At a past employer, I experienced their trying to measure productivity by “net lines of code”. That didn’t work well, either. I prefer refactoring and simplifying to pretty much any other type of task, and it isn’t at all uncommon for me to end a day - or a sprint - with a negative NLOC. I see at a positive, as each LOC carries with it an ongoing cognitive and maintenance burden. Thankfully, that scheme didn’t last long :)

This is hitting too close to home. Currently on a team that does what’s on the task list without questioning anything—driving me up the wall.

This is a function of the organization, not the team.

If you hear ‘do it anyway’ after raising concerns one too many times, eventually you just do it.

Not necessarily. It depends on exactly what "it" is. I have quit jobs before because I was required to implement something that I considered an egregiously terrible idea.

Sounds pretty standard operating procedure for product teams in large software companies, actually.

Sad. But systemic to the industry.

Some bonehead got in charge and forced this through. People were screaming internally but ignored as trouble makers. So it goes.

EDIT: To clarify, I'm just making a generalization and am not affiliated with Gitlab.

This language is the classic "defuse, wait, and try again later" approach to shoeing in unpopular changes. They're still hedging their bets with this language, rather than renouncing the original ideas. Apologising for "bad communication" instead of bad changes is another classic deflection move they've pulled in other threads, too.

<disclaimer: founder of a GitLab competitor>

> <disclaimer: founder of a GitLab competitor>

IMO, if you want to compete with GitLab, you should introduce some higher pricing tiers targeted at businesses. These should be comparable to GitLab's pricing tiers and should not have "hacker" in their names. I also suggest that for these higher pricing tiers, you make it explicit that using the service for closed-source projects is OK.

This is good feedback, but I think the bigger issue is the alpha quality of the service. People who move now will have to be tolerant of some pieces being missing or under-documented, which often means a vote of confidence in the future of SourceHut more so than in the present. When the alpha becomes the beta, and then production, the pricing model will be re-evaluated.

Ideally, price it at the $20/month/person price that Gitlab has, because that seems to be the highest that I’m able to sell to Management.

Theoretically $100/month/person would still be a great deal, but the finance people don’t look at it like that. They just see the $97.5/month/person difference with Bitbucket.

> There were many more concerns than we expected.

Telemetry was always going to be a concern with services that promote themselves as "open-source" or "free-software". The subject is so important that it is one of the reasons that the mass exodus from GitHub to GitLab happened. So to say that "there were many more concerns than we expected" is complete bullsh*t and appears more like the VCs are puppeteering the founders with this talk here.

Like other companies that are at the mercy of VC funding, they will do anything to please them over the "community" to show that they are growing. So don't be fooled by this empty response.

Because of their fiduciary responsibility, the officers of a corporation must make decisions that increase shareholder value. Refusing to add profitable data collection to the product due to ethical concerns would be a violation of that duty to the investors.

This will keep happening until it literally becomes illegal to collect personal information.

This is a corporate lie that has been promoted relentlessly since the Reagan era. Officers of a corporation are responsible for the operation of the corporation in accordance with the law.

They are responsible to the Board of Directors of the corporation, not the shareholders. The Directors are responsible to the shareholders.

They have a responsibility of care (including a fiduciary responsibility) to operate the corporation in the corporation's best interest, not the shareholders, as directed by the Board.

That best interest can be measured in all sorts of ways as established by the Directors, which may include increasing shareholder value.

The practise of CEOs also being the Chairman of the Board, of executives being major shareholders, of bonuses being driven by share price, are all practises that should be eliminated, given that they are not in alignment with an executive's actual responsibilities and duties.

> Because of their fiduciary responsibility, the officers of a corporation must make decisions that increase shareholder value.

Drilling a hole in your ship may be immediately helpful because you are thirsty, but ultimately it’s not going to end well for you.

I’d say it’s well within the realm of reason to say a hard ‘no’ to investors in this case.

> There were many more concerns than we expected.

So they expected some, but still went ahead anyway?

There will always be concerns and complaints. I'm more surprised they didn't expect a firestorm given the way they explained the change, though. I wonder how many people actually reviewed the language.

Nothing wrong with that in general, you can't make everyone happy.

There will always be concerns from some people. Always.

Hahahahaha. I remember all the fuzz about MS buying github and perhaps forcing telemetry. Something along those lines.

People were so fast to live their stuff to Gitlab without bothering thinking about that business is business.

These fucking corporations or wannabe corporations doesn't give a fuck about you or your data. That's the fundamental mindset you need to have when sending them money for services.

Ain't no such thing as halfway crooks.

Github is already collecting limited client-side telemetry. Open the network tab in your developer tools and you'll see a request to collector.githubapp.com on every page load that includes details like your screen resolution.

If they collect it themselves, without 3rd party script, it's more secure than including 3rd party scripts in the page, which is what Gitlab seems to want to do.

Also, who knows what else they collect about you on the backend? I find it funny that people sometimes make such a fuss about front end metrics when they could be collecting way more on the back end (especially so if it's a closed source platform like GitHub, at least with Gitlab, I can look at the back end code and see what they're collecting).

Sidebar good news for anyone reading this: Since they use a different hostname that doesn't host any operational functionality, namely collector.githubapp.com, that host is already included in most DNS blocklists.

> These fucking corporations or wannabe corporations doesn't give a fuck about you or your data.

They don't give a fuck about individual users but they sure do care about their data.

> These fucking corporations or wannabe corporations doesn't give a fuck about you

Thanks for the redpill. /s I would however say that they had a good rep until now, in general, so perhaps they will reconsider if there's enough backlash.

Judging from the thread posted above, it looks like the people doing the real work actually do give a shit. It's their CFO that's the problem. Unfortunately, he and the investors seem to be calling the shots.

They give a fuck about your data, as they can make a buck off it. Your rights? Well what about their right to maximize their net worth?

Somebody at GitLab thought it was a bright idea to add telemetry to their cloud AND self-hosted enterprise versions. Needless to say, it's not about to go over well.

I currently use GitLab where I work; we chose it because our data is sensitive and a cloud service was not an option. This telemetry means that I won't be updating until this blows over. Frankly, whoever thought this was a good idea is a moron that doesn't seem to understand that users like my company chose GitLab because we didn't want this shit.

Engineers tried to resist, CFO insisted...


This was also a great comment by Candice Ciresi, the Director of Global Risk and Compliance, 3 months ago:

> Sorry, I am coming in late to the game here.

> Saying we want the UID for easy ID and deletion is one thing. If we actually intend to track and do something, then that is another issue. This former benefits the user, the latter benefits us. Because we stand to get a benefit, users should opt-in. We will also need to amp up our privacy policy. There needs to be full disclosure in what we gather, why we gather and with whom we share.


>I don’t understand. This should not be an opt in or an opt out. It is a condition of using our product. There is an acceptance of terms and the use of this data should be included in that.

Sounds to me as if the CFO is carrying water for investors.

Which points to where the problem might lie.

Why the CFO is calling the shots on critical architectural issues is a whole 'nother question.

It doesn't seem to me like he is calling shots on critical architecture. He asserted an opinion on something that would affect the revenue stream, which as CFO, is exactly his job.

Reading through the MR comments, it seems to me like that's the case with everyone. The CFO is pursuing profitable options, the legal & compliance teams are making sure everything stays in compliance, the engineers are building what is asked of them, the data analysts and product managers are asking for the data they need to get insights on product enhancement...

The big issue seems to be that everyone is so narrowly focused on just their job function that they are missing the forest for the trees. I also noticed a distinct lack of anyone from any type of customer advocacy teams (does GitLab have anything like that? Account managers, evangelists, developer relations, etc?) that probably would have been able to put forth actual data about if customers would be for/against this change.

> It doesn't seem to me like he is calling shots on critical architecture. He asserted an opinion on something that would affect the revenue stream, which as CFO, is exactly his job.

> Reading through the MR comments, it seems to me like that's the case with everyone. The CFO is pursuing profitable options, the legal & compliance teams are making sure everything stays in compliance, the engineers are building what is asked of them, the data analysts and product managers are asking for the data they need to get insights on product enhancement...

Ideally everyone should be would also be thinking about whether the feature is ethical, even if it's not "exactly their job", because there generally isn't anyone whose job is specifically to decide that.

That's precisely what I meant when I said they missed the forest for the trees.

The CFOs job is not revenue. That’s product, sales, etc.

Most parents learn this, and some dog owners.

If you want to talk to your spouse about the possibility of taking the kids to get ice cream or whether the dog has had a treat today you have to speak in code that the short mammals can't understand, otherwise you've all but promised it to them and have to deal with the consequences.

The problem is that for salespeople and business devs, the consequences of putting a bad idea in front of impressionable ears are too abstract and so they never learn. So they put an idea in front of a customer or the board about how they can make a shitload of money and the imaginary check has been cashed even before they've stopped for air.

Without some tough love these problems will be with us forever. Oh, you're going to have to walk back something you said? You'll be embarrased? Too fucking bad. Maybe next time you'll think before you promise someone $500k of work for $250k minus your bonus. Twist in the wind like you deserve.

So someone has sold an idea of making millions and once the engineers or just plain human beings get ahold of it that looks like $200k and a giant pain in everyone's ass. And everyone jumps straight from denial to bargaining with a little detour to anger to yell at the messengers for breaking your shiny dream... that you are not and were never entitled to.

Wow, that was interesting. I'm shocked they make this sort of thing public.

One of GitLab's core values is transparency, including making all of their internal communications public via public merge requests. It's kinda cool, but on the other hand, I've previously spent some time reading their MRs about security policy that made me wary about using GitLab at all.

Fortunately they've updated the announcement and are holding back on the telemetry updates. But yeah, DNT is not an acceptable option for privacy. `In order to service the needs of GitLab.com and GitLab Self-Managed users who do not want to be tracked, both GitLab.com and GitLab Self-Managed will honor the Do Not Track (DNT) mechanism in web browsers` Hopefully the update includes a promise to enable deployments without telemetry code.

https://gitlab.com/gitlab-org/gitaly/issues/2113 https://about.gitlab.com/blog/2019/10/10/update-free-softwar...

Just to point out if you care about privacy it's not recommended to turn on this feature becase close to none services support it and it's awesome fingerprinting feature for your browser as well.

There is no way for Safari users to opt-out from this because Apple has decided to ditch "Do Not Track" because of privacy concerns.

I keep it enabled for sites that do respect it, like firefox.com and friends. By removing support from Safari, Apple took this choice away from their users, but I suppose that's not something surprising.

The problem with Gitlab is that in the current state it is a dead product walking. Its existence is entirely based on VC FOMO. It has no enterprise growth to justify company valuation. And it won't have real enterprise growth either.

Github had to be acquired my MSFT because it had no path for real enterprise growth if it could not lean on an organization that already sold a pile of services and products in similar or complimentary space. MSFT was the only strategic buyer of Git hosting company. MSFT is the only circus that could use that monkey to fill the gap in its product line up. Of course it picked Github rather than Gitlab. Which is why Github would be incredibly successful, which in turn means that Gitlab in its current state won't be. Hence they would have to massively change the product to justify valuations. It would be interesting to watch.

SourceHut is in the exact position Gitlab was before VC FOMO. It can provide decent service that people and companies will pay for but they would never get a billion dollar value. As long as it works for them, it will work out just fine. I wish them well.

Sorry, this is painfully deluded.

Gitlab is the _only_ choice when your company prefers on-prem (which is the overwhelming majority of >10k employee companies) -- I work at a company with more than 15k employees and we hold an enterprise (plus) license for 8k seats.

We are definitely not alone, companies like EA/IBM/Goldman Sachs have _large_ internal deployments of gitlab.

They certainly have developer mindshare in large companies, but github is the facebook of sourcecode repositories, it will always be more popular outside (or in smaller, less self-hosty orgs).

> Gitlab is the _only_ choice when your company prefers on-prem

Is that literal or figurative? GitHub Enterprise is on premises. I used it at my last company which was on an air gapped network. It looks like it has been around since 2011.

There are countless other self-managed variations in size and feature combinations of git hosting, ticket tracking, and other things it offers.

Figurative. Obviously it’s possible to host git on your own servers. But github enterprise on prem is a comparatively weak offering. Githubs bread is made from people using their SaaS offerings and it shows. (I’ve used many different self hosted options for version control. Although to be fair my current company uses perforce for the overwhelming majority of its version control)

Nonsense. Github Enterprise at my last job (used there since 2012) was so much more refined and reliable than Gitlab EE is at my job now, which has much worse UI and no good flow at all. All Github clones (like Gogs, Gitea) lean on the Github UI for a reason, ignoring Gitlab. It's a mess.

Since Microsoft's acquisition, even github.com is progressing refinement at a higher pace than gitlab.com ever did, and I'll be thrilled when I get my company to switch to Github again. Pricewise, Github EE is much better value for the dollar than Gitlab EE; most features that Github doesn't have are rather half-assed than wholesomely done in Gitlab.

Counterpoint: My fortune 200 employer uses GitHub enterprise on site. It works great.

I don’t get how you can say GitLab is the only choice?

We use GitLab. We rejected GitHub Enterprise because it’s a closed system: we could not use it to host public open source repositories like we can with GitLab. GitHub’s sales pitch was funny, all about “inner source”, applying open source development processes to closed corporate software as if it would be a step forward, whereas we were way past that point already so it would have actually been a step backwards for us.

It’s not “on prem” anymore when part of the product is on a third party server.

Gitlab is a single product company with no sales force behind it. It already sold to all the customers that it could easily sell to. In order to sell to the rest they would need to have staffed regional offices to do really really really long term sales cycles. It is an unsolvable problem in a current state.

Microsoft is a hundred if not thousand product company with gigantic sales force and sales channels that are beyond the level of imagination of companies who say "remote only" is their differentiator. Github Enterprise exists. I know it because we used it. Github will be sold as a part of the rest of the Microsoft product portfolio, which is how they would end up in those coveted enterprises.

Microsoft is still considered a pariah by the kinds of people who would use gitlab.

It will be 100 years before my company puts their core business in the hands of a SaaS provider.

Github Enterprise is not SaaS. It is on-prem.

Sourcehut intentionally is not going to accept VC money and they know they won’t get a billion dollar value.

HN really needs to have a baseline rule of shadowbaning the downvotes by all accounts associated with a company subject of the article.

And you should refamiliarize yourself with HN's baseline rules about suggesting this kind of thing is what's happening to you.

It is just silly. HN is becoming more and more a platform where discussions are gamed by the companies that are being discussed.

What's the point of having them when Gitlab employees downvote anything that does not say "Gitlab gooood!", Google employees downvote anything that is not positive about Google and Facebook employees downvote non-positives about facebook.

Accusations of astroturfing are against the guidelines. Downvotes often signal disagreement or call out a low-quality comment.

Just wanted to point out Gogs: https://gogs.io/

It's a very light weight, self-hosted git server, the UI is very GitHub like.

It's not a replacement for all the features of a self-hosted GitLab.

But I've seen people battling a self-hosted GitLab instance when all they needed was something like Gogs.

Or Gitea, which was forked from Gogs iirc:


Alternatively Phabricator for all in one package (Trello, Slack, JIRA, Github alternatives but all open source)

Isn't phab slowly going the way of the dodo?

Where does this sentiment come from? I've seen it posted before.

I've been using Phabricator for 2 years and development seems to have picked up, if anything. They seem to be polishing what they have and making everything more robust, an odd thing to do if its going the way of the dodo.

For example kde seems to be moving away from phabricator to gitlab https://about.gitlab.com/press/releases/2019-09-17-gitlab-ad...

Combined with debian moving to gitlab, gnome moving to gitlab, ... , that was the sentiment I got.

Ah, I think that may be due to the patch submission structure of Phabricator more than anything else. I can understand why a major open source org wants to adopt something else that people have as muscle memory (ie. pull request style collaborating). I don't think that reflects the development of Phabricator at all, like I said, its become more robust and features are becoming more mature every week (and there's still some big installations out there, like Wikimedia).

Yeah Phabricator is quite good for project management, but it's a pain in the ass to manage and the CI is very meh.

Looks interesting. As if they had stolen all the CSS files from GitHub.

Gitea is also nice self-host option, but also not a complete ALM solution. We often use trac in conjunction with different source control providers, which might be ancient by now, but it always delivers and everyone seems to like it.

Gitea is a fork of gogs.


application lifecycle management

Can you really steal CSS though?

Definitely, yes. CSS (just as HTML, JS, and anything else being served) is covered by copyright. The law(s) may vary depending on jurisdiction, but generally speaking you can be sued for copyright violation. Getting caught is a different matter, but if it is obvious that you copied CSS you can be in serious legal trouble.

i mean the design of it is almost a carbon copy of the github design down to the layout, design, and colors.


if i replaced just the logo in the upper left hand corner to the github logo instead of the googs logo would you be able to tell the difference between this screenshot and github proper?

> if i replaced just the logo in the upper left hand corner to the github logo instead of the googs logo would you be able to tell the difference between this screenshot and github proper?

I think I could easily do that.

And I'm a backender-at-heart who doesn't even have full colour vision.

So, not a copy I think.

Also Pagure: https://pagure.io/pagure

Written in Python and used as dist-Git frontend in Fedora + used to host many Fedora related projects.

I believe gogs is mostly developed in China. I don’t know if it is developed by individuals or if it backed by the Chinese state. Maybe someone here knows?


It does not seem to be mostly developed in China, but there are Chinese contributors.

The feedback to this change is very negative. And some of the things we thought would help (respecting DNT) don't seems to matter much. We'll have a look at all the feedback and see what we can learn and adjust. First change to the blog post is in https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/... and we're discussing more changes.

Not sure why the "opt-in" option isnt more apparent as a less hostile UX?

I've always thought of Gitlab as a privacy minded company, until now, and the botched attempts to listen to feedback are only worsening my impression. Please prove me wrong!

"Not sure why the "opt-in" option isnt more apparent as a less hostile UX?" do you mean having to use DNT to opt out?

The issue is forcing this as "opt-out" rather than "opt-in".

Opt-out is pretty hostile UX for something you must have known would be seen as a negative by large swathes of the developer audience (and if you didn't see the negative reaction coming, then there's larger questions to be asked).

I'm a free user, so I'm "happy" insofar as I have very little say, but it's certainly a worrying step.

Thanks. We will revisit opt-in vs. opt-out, especially for self-hosted installations.

We didn't expect praise for more telemetry from the developer audience but we did underestimate the reaction. There were many more concerns than we expected. We’re going to process the feedback and rethink our plan. We will not activate tracing on GitLab.com or GitLab self-managed for now. We'll make sure to communicate in advance on our blog when we do have a new plan.

The reason why the backlash is so big:

* Devs are very sensitive to tracking. Especially opt-out rather then opt-in. We know what kind of data companies sit on. And opt-out is invasive.

* Gitlab is perceived as a developer friendly open source company. An alternative to the bloated/"enterprisey"/milk the customer stacks out there

* Self-hosting should come with a promise of privacy

* Disabling API/UI access without opt in is just a really bad move. You should have seen the anger coming from a mile away.

* The claim that you can't get insight into how Gitlab is used without client side telemetry is very dishonest.

You have a wealth of data in the DB and web server logs. True, you won't know how long a user fiddles with the mouse until they click a button. I understand the urge to optimize the UX. But user studies go a long way.

And let's be real: this isn't about optimizing button placement, but about showing some nice engagement graphs to investors.

Also, the whole response both in the issue and here was quite poorly handled in my opinion.

The back and forth here... Copy-pasting a reply to each commenter in the issue... (really?)

Just re-consider and put out a unified response.

That is a great summary of why the backlash was so big.

We have a lot of data in the DB of GitLab.com but it was hard to understand it. Using a third-party service with client side tracking saves a lot of time and effort.

> third-party service with client side tracking

That's like me inviting you to my house and you show up with a pet pig. Your pig might be clean and well behaved, but I'm not letting it my house because it's a pig.

I trust GitLab and I'm ok with the spirit of what you're trying to do, but if you want me to opt-in to client side tracking from a 3rd party (in an untrustworthy industry), you'll need to convince me the company you're working with deserves to be trusted.

Just as a data point, our organization is literally this week looking at some form of git UI tool. Due to our security requirements, cloud services are not an option.

On a smaller project, we have implemented Gitlab CE and it was/is in the lead of the various alternatives.

Telemetry to any external service from inside our VPN is a definite no-go. Not to mention that it would be blocked in our configuration, but if that meant that the application didn't run, then we need to make another choice.

Telemetry to a specific provider that we have licensing and other arrangements with would be manageable, as long as the data collected was documented and we could determine that there was no possibility of data leakage beyond that declaration. It would need to be opt-in and it would need to be under conditions that both parties agreed to.

Tell your CFO that if he wants to sell to enterprise, particularly self-hosted enterprise, then he needs to get his head out of the "SaaS" world and deal in the world of Enterprise, where things like SOX and HIPAA and GDPR and PCI/DSS and other standards preclude "collecting data on our users".

Wait, we know what kind of data companies sit on?

I've always opted into telemetry because I assumed companies just use it to improve their products. I guess I've never actually been in the inside of a big company, though. Do they do something else with it I should know about?

Thanks for the reply, and glad to hear about the review - am still surprised to hear the reaction was larger than expected though. The narrative is rather predictable - "community minded tech company raises cash, has ultra valuation, starts imposing tracking onto users etc..." Fair or not, that's the leagues now.

Also, might be a learning about engaging the community on things it understands - a common user of a SaaS platform might not have an understanding of tech, but a common user of a developer SaaS platform does.

If when I had next logged into the dashboard I got a modal that asked me "hey, here's a list of things we'd like to monitor to make your experience better, what data would you be happy we looked at?" I'd probably opt-in to some just to be helpful as I know the context.

I'm guessing a lot of your mid-market acquisition (though likely not revenue) is through word of mouth, migrations or "dark procurement" (?) - audiences that are ultra-sensitive to this sort of thing, so the misstep here is a question.

> we did underestimate the reaction

Be serious. You knew what the reaction would be. You gambled on it getting missed by the community and lost.

As said we didn't expect praise.

We published this change in a blog post and sent an email.

>There were many more concerns than we expected.

Every single one of our concerns was raised by someone in your own MR thread. C'mon, you were just hoping people wouldn't raise too much of a fuss.

I keep coming back to the thought that for companies like gitlab the 'customers' are the VC's that put up the money not people that use the product. And it's really showing here.

>We'll make sure to communicate in advance on our blog when we do have a new plan.

I would have thought this would be one of the most important steps in implementing this change. Surprises are never good for people working in this field.

Thank you for listening and reconsidering based upon feedback.

Typically when Github “copies” a feature, there’s a blog post within a couple of hours.

>for now


Not all browsers make it easy to turn DNT on (Safari has apparently removed it), so it's not an ideal opt-out mechanism.

DNT is basically a failed technology; if you really care about not being tracked you would just block the trackers rather than asking them to pretty please not track you.

(Edited to add: as others have said, this should not be opt-out anyway. Opt-in is much less user hostile, and as far as I know is legally mandated in the EU anyway.)

Using DNT as an opt out mechanism is beyond ridiculous. It doesn't impact us yet (EE) luckily. This would put us in a tough spot as we are required to control this sort of thing, but have no way enforcing it with our third party developer community. It is a worrying misstep from my perspective.

No, I mean opt-in to telemetry as opposed to having to opt-out would be more ideal

How is it that you even considered DNT a viable option when it's been common knowledge for years that DNT is a failed technology? Even Wikipedia knows that, and yet you, an expert in the field, don't?

Cynical me could imagine the conversation might go something along these lines:

Alice> We want to do a thing, and we want everyone to do it without being able to realistically avoid it while still saying "we gave them an option"

Bob> Let's use DNT, it's used by almost no-one and is largely obsolete.

Alice> Perfect!

It's a bit like web sites that allow you to opt out of tracking and targetted advertising by directing you at the cookie config settings in your web browser. A token gesture.

There's no need to be cynical, the situation you posed is 100% within the realm of possibility when you consider that the actions Gitlab took benefit Gitlab greatly.

Yeah, we need a "it's now infeasible to track me" as opposed to a "please don't track me."

> some of the things we thought would help (respecting DNT) don't seems to matter much

DNT is dead as a dodo, and many people either use browsers that no longer support it or are strongly resistance to enabling it because it's used as a signal in fingerprinting browsers.

I'm not sure why you thought DNT would help.

So first of all I’d recommend you try to understand your users.

Your EE users use your self-hosted product because they don’t want anything leaving their premises — not a single bit of data. If you require this, do you really think EA, SIEMENS, Ericsson, Lockheed Martin or Bayer are going to let any data from their instances with potentially secret and internal future projects leak out to you or your third parties? No, they won’t. They’ll require you signing a contract that you won’t track them, or they’ll switch vendors — you’re the small fish in that case.

Additionally, I’d suggest reading and understanding the GDPR — the text is actually surprisingly simple:


So you, or in case of the EE Edition, your customers, have to have explicit, clear, and freely given[1] consent to allow any telemetry or tracking which contains PII to happen at all, and especially to give any PII to third parties. Pseudonymous data counts as PII. Data which contains the IP (such as a direct connection to a third-party server to transmit data) counts as PII.

[1] Even more interesting, freely given consent is defined as explicitly opt-in, and it has to be in clear and simple language, the default (and/or the biggest button) has to be rejecting everything, and the consent can not lead to any functional changes, so you can’t require opt-in to let the user access any functionality either.

From the article:

"Enterprise Edition (EE) No current changes. We are not adding telemetry services to EE at this time."

>at this time

Which was recently added after receiving a metric ton of backlash. You have to question their initial thought process here.

Check out some more of the thought process: https://gitlab.com/gitlab-org/telemetry/issues/34

The original title was "Remove the ability to toggle sending telemetry data back to GitLab", then "Remove the ability for on-prem users to toggle sending telemetry data back to GitLab"

That thread is disgusting. It's crazy how many people in that thread seem to be suits with no dev comprehension whatsoever, talking about shit that can tank the reputation of their company.

>I'm not familiar with the term "dark patterns".

Then you should not be suggesting changes like this in the first place.

>I want to highlight again that public sector customers will be excluded from this. I've mentioned many times that it's important we respect their strict privacy requirements.

But fuck private sector customers? What?

> We are not adding telemetry services to EE at this time.

That sounds like something to rely on, especially given how they've handled this so far. Also, this wasn't originally there.

EE excempt was just added, initial version of blog post implied that only CE would stay without tracking.

> We are not adding telemetry services to EE at this time.

Emphasis mine, but the wording is notable.

That means nothing, if the owner of the instance has to sign a legal contract (which the ToS is) allowing Gitlab to track the data, then (a) the owner won’t upgrade, or (b) they owner becomes GDPR incompliant immediately.

I can't believe any organisation can be this tone deaf considering the awareness of privacy abuses that are practically in the news on a daily basis.

Respect your users and respect yourself, roll back the change, make an apology PR piece and move on.

I personally don't have a problem with telemetry when the product is free and what you collect and how it's used is clearly defined. If people are paying, it should be reduced or eliminated if the monthly fee is high enough.

I think a big problem is that it was so unexpected! Especially the API changes, where people probably are wondering why things aren't working today. Some advance notice would have helped, probably.

I've used Gitlab in the past and will continue to use it here and there, though I host all my own repos at home with just regular old git and use Github for work, so take what I'm saying with a grain of salt because I'm technically not a heavy user of your service.

Part of the issue is that Gitlab's approach signals a "we don't know what we don't know" problem.

Enterprises have been dealing with GDPR, CCPA, and data privacy issues for several years now. The apparent fact that Gitlab doesn't recognize when they're running afoul of opt-out standard practices mechanisms, and has those vulnerabilities appearing to be not caught during the SDLC is probably causing a lot of second guessing of competency by your more mature customers.

edit: This isn't a problem unique to Gitlab. Microsoft, for example, has encountered and dealt with this problem (telemetry privacy issues) as well (https://docs.microsoft.com/en-ie/DeployOffice/privacy/overvi...). Search for "Microsoft Dutch DPIA" for all the sordid detail.

Please make this telemetry opt-in and optional, even on GitLab.com.

Judging from the MR thread it seems that almost everyone was in favor of opt-in except for the CFO.

The code review worked. Then he got shouted down. That’s a big corporate culture problem.

And why did the lead email mention SOC2 and not CCPA and GDPR?

You can't measure something without impacting it. Adding spyware to products make them slower.

Here is an alternative solution I begun using for one of my projects:

There is no telemetry like grep in nginx logs with carefully fetch()'ed paths.

Put this in JS code:

Then you can find the information you need with:

    grep -o ".*/stats/[a-z_]*" /var/log/nginx/access.log
Now you can count how many users reach features and checkout steps.

That is dozens of time more performant than using third party scripts. No extra library.

Yeah, but they want heatmaps and attention spans. Optionally eye tracking, if camera is available.

I don't like the stated justifications for enforcing telemetry — even if it is genuine, it is basically self-deception.

In my experience, access to analytics is a quality of life improvement for programmer, that does not actually result in better products. Having a thousand times more crash-reports wont's cause your employees to grow extra hands and brain-cells to act on all of them. Having a great insight in user behavior does not automatically translate in great managerial decisions.

Personally I would love Gitlab to stop overusing Javascript and fix performance of their backend instead of trying to conceal issues by refusing to show big files and abusing lazy-loading. But judging by their recent actions, they are more likely to copy more anti-features from Github than work on actual hard problems.

There are so many issues in play here. Some plain legal, others based on expectations.

1. DSGVO means opt-in when collecting private information not necessary for the usage of the product (and even then it's dicey). Making the use of your product dependent on accepting private information gathering is illegal. DNT is logically a valid way to opt-out (apart from browsers dropping the feature and users not knowing about it), but that's not enough here. It would be different if DNT was enabled by default (I assume), but all browser makers dropped the ball here years ago.

2. You built a reputation of the people's git provider, albeit in conflict with the completely free alternatives. And now you make a post "we will change it, you have to accept the new TOS, and there will be a service interruption", like a generic US company might do. It's unfitting to you. It does not matter that the products concerned are not free.

3. The right solution exists and would not hurt you. I assume you want telemetry data to improve the product (if not the issue is even bigger). So you add that option and ask them whether he would be okay to activate it, like Mozilla is doing in products like Firefox. This scheme works and data gets collected anyway, a bit less, but that should not be an issue. Careful: Admins must be able to remove the prompt for all their users if it arrives in the self-hosted products.

4. Pick a data collection destination that is the least controversial. In your case that should be inhouse. Alternative might be a privacy focused organization in Europe, but it has to be known to your customers.

5. If you think DSGVO does not apply to you realize that it does not really matter since the sensible parts of it are also the ethical correct way. So in any case it is good to follow the spirit of it.

6. Note also how matter of fact and corporate the language used in the blog post is. That can only worsen the impact on those that care about you and the topic.

>It would be different if DNT was enabled by default (I assume), but all browser makers dropped the ball here years ago.

Browser makers did the logical decision as they have no control over websites adhering to DNT. If they made it default, every site would promptly ignore it as it'd impact their business too much to follow it. On the other hand, if they made it opt-in then sites might follow it for the minority of users who used DHT. In the end, it was a bad idea as it had no teeth at all to force adherence on websites.

I've been a huge Gitlab supporter for the past 5 years, and we have all of our repos there. Have had their paid service for a while.

This feels gross, and I'm going to look at moving.

Yeah, that was my feeling as well. I can forgive them for being careless enough to delete their production database (2 years ago?), but not for the dysfunction in the company that was necessary for this debacle to happen.

Something I have noticed several times and I can’t understand from an EU perspective is this: in the US a company you have a contract with (ie a company you are paying for a service for a certain amount of time) can change their tos / contract with you without prior notice and without having to at least honor the previous agreement until your current billing period expires?

For example if I paid for 1 year, I would expect the contract available at the time of payment should apply for the entire year or offer a refund if you don’t want to accept the changes. And always with at least two weeks of notice

Usually they have a clause in the contract stating that they can change the terms. Github provides 30 days notice, Gitlab has an email with instant effect and a vague thing about checking their website. I've seen sketchier companies who have instant changes w/ no notice and that state their published TOS is inaccurate.

I am altering the deal. Pray I don't alter it any further.

According to my, possibly incorrect, take on the situation this does not really make that much sense.

As I understand it, like GitHub, GitLab offers relatively permissive free plans to individuals and projects in their early stages. Both also offer pathways to graduate to commercial plans that are also enterprise friendly.

It sounds like GitLab planned telemetry for the enterprise-managed plans (not the individual-managed plans). If GitLab was seen as an alternative to Microsoft-owned GitHub, the plans for telemetry undermined this reason for developers to start their projects with GitLab instead of GitHub and advocate its use to managers, investors, and legal departments.

If that is the situation, where does SourceHut stand? The site makes little sense to me. The URL is sourcehut.org, but there is no statement about it being a non-profit, or any discussion about a board that manages it. There is a pricing page, but no contact us page. I don't know where it is based, and I can't find a reference on Crunchbase to get any idea of who owns the company.

If it is a competitor to GitLab, maybe it is a competitor in the space for individual developers, but there wasn't a significant problem for individual developers when it comes to GitLab (or Github). If the problem with GitLab's announcement was that it made the upgrade path less enterprise friendly, how is SourceHub an improvement if by all appearances it is incompatible with any sane enterprise.

I don't have a reply for most of your comment, but with respect to this part:

>If that is the situation, where does SourceHut stand? The site makes little sense to me. The URL is sourcehut.org, but there is no statement about it being a non-profit, or any discussion about a board that manages it. There is a pricing page, but no contact us page. I don't know where it is based, and I can't find a reference on Crunchbase to get any idea of who owns the company.

I own SourceHut as a sole proprietor, you can reach me at sir@cmpwn.com with private questions or ~sircmpwn/public-inbox@lists.sr.ht with public questions. Archives here:


The business is operated transparently, here's the latest quarterly financial report:


This whole mess (gitlab really fumbling the ball) seems like a great opportunity for sourcehut.

Even though I may not especially enjoy email-patch workflows, I still wish you success, because of your user centric ethos.

Going to github isn't better I guess, or has a bad outlook at least with MS's lust for "telemetry". Last I checked, they blocked indie search crawlers.

The recent Github repository exodus has now been in vain since GitLab is taking action similar to how MS is famous in developer circles for telemetry and not respecting your privacy. Whenever VCs are involved, the community always finishes last.

For individuals sourcehut[0] may seem to be an alternative. Open source orgs might be better of self-hosting. In either scenario I would rather go with the community than to be in the mercy of a VC-backed corporation for hosting my work in this case.

[0] https://sr.ht

If you use something like ublock origin to block the third party telemetry scripts on github I don’t think it breaks the rest of the site no?


Very disturbing that this is a discussion you need to have now as opposed to already knowing this considering how privacy sensitive the dev community has always been.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact