I’m convinced 90%+ of telemetry is simply to have usage metrics to show managers, rather than actually being used to improve products by identifying inefficiencies and bugs. And it’s a malevolent spectre if you’re the 1% of users who actually use a niche feature in something, since developers will use that knowledge to take it away.
Despite a 100-fold increase in hardware power, and languages/frameworks that handle 80% of the workload a developer 15 years had to think about, applications these days have fewer settings, open and run slower, and improve far less rapidly than anything written in C/C++/Java from 1990-2005 despite phoning home with novels worth of telemetry every few minutes.
Monitoring network traffic and DNS requests through a firewall and pihole, it’s downright hilarious that it’s the worst products that top the charts in both, even when being online is not a necessary component of the software. Meanwhile the software that just works, and works well, might check for an update once a month, and the changelog generally reveals new useful features and mentions performance increases.
> And it’s a malevolent spectre if you’re the 1% of users who actually use a niche feature in something, since developers will use that knowledge to take it away.
Yes, but also no. If my resources are not infinite, knowing that an "expensive-to-maintain" feature is used by just 1% of users, it is an incredibly important piece of information when I'm gonna have to decide where/how to recalibrate my (team's) effort.
Product managing is also this: I have to know what works and what doesn't. Maybe I will use that knowledge to understand how to increase the 1% to 5% with a specific business case. Maybe it won't make sense and I will have to decide if I can keep it or not.
I agree that vanity telemetry is bad for users, but killing ALL telemetry is bad for everybody.
> Product managing is also this: I have to know what works and what doesn't.
You can't use a blunt usage metric to tell whether a feature works or not. It only tells you that the feature is popular for some reason. That reason might be that it works, but often the reason people use a feature a lot is it's shoved down their throats.
It's a vicious cycle. Say you are Product Manager for Feature X. Your bonus/promotion/career progression depends on some vanity usage metric for feature X. The more monthly actives or new users for feature X, the better it is for you. So, naturally, you're going to do whatever you can to funnel users at that feature. Stick it prominently on the main screen. Highlight it in flashing red. Spam the user with notifications begging them to use it. Make competing features low-contrast gray on gray. Whatever you can get away with. Then the next monthly metrics report is in: The feature is now being used by +15% of users since last month! Incredible! Your bonus buys you a Tesla. Management's conclusion is: "This feature really works and users must really like this feature." So, the product focuses more on developing that feature.
Now scale the above out to every feature in the app, each with their own competitive and motivated Product Manager. This is how we get apps that are all-telemetry dumpster fires and with more and more flashy features fighting each other to grab your attention and fewer and fewer useful 1%-features.
I've personally witnessed meetings with multiple furious product owners literally shouting at each other about where their feature's button gets placed on the home screen because that placement made the difference between 5% feature usage and 25% feature usage. Where was the end user in all these discussions? Not present. End users are to be milked for metrics gains.
> You can't use a blunt usage metric to tell whether a feature works or not. It only tells you that the feature is popular for some reason. That reason might be that it works, but often the reason people use a feature a lot is it's shoved down their throats.
I never said that was the only source of information, but indeed a useful one. Thinking a good PM only uses that one bit of info means you have known really bad PMs in your work.
The product managers know this, but the higher-ups (who control the money flow) tend to get tunnel-vision on a single metric, and that metric is often going to be the one most understandable to a layman. So that's the one that gets juiced.
Well, tracking users by cookies or fingerprinting or whatever can tell you that by combining data from different sources. Now we've looped back around to being in favor of the most intrusive methods.
As I said that is a "piece of information", not the whole story. Making an axing decision with ONLY that information would be outright idiotic.
Telemetry has to be designed as well, to piece as much as you can, and then you complete the full picture with other sources. Maybe not everybody does this, but I don't think that this practice should be "shocking" for anybody.
yea you sound like someone who is competent and will actually thing things through and make a rational decision. Many people are not and will latch on to the information "only 1% of people use it!" and refuse to listen to why that may not matter"
Well, in Europe, if you want to know how much a user pays, you need to tie telemetry to their account containing personal information, which is forbidden by GPRD unless they give their explicit consent and they have a process to delete/rectify/download all of it.
Since most telemetry are optout and don't bother about giving users access to their data, they can't legally do it.
That's my point really: useful telemetry is really, really invasive. So just ask consent, and accept most users will tell you no.
If you chose what's convenient instead, you will not only have a mixed bag of data, but also you have decide to basically force yourself on people, in an age where a part of your userbase is getting upset over things like using the wrong pronoun on them.
I tell yes to Firefox and VLC to report on my usage. It tell no to Google and Microsoft. They want their cake and it it: behave badly, and then have we trust them.
Right. A mention in the ToS does not equal real consent. I would only add that it also has to be informed consent. Saying something like "can we collect data from you to improve our products and services?" is also not soliciting real consent, because it tells the user basically nothing about what is being asked of them and how it will be used.
You think that keeping track of "This paying user uses this and that feature, and is thus on the correct price plan" is a violation of the GDPR, assuming it's mentioned in the privacy police and maybe ToS?
I don't know what does or does not violate the GDPR, and I'm not talking about metering services. I'm talking about telemetry for installed software.
I'm not even talking about what the law says, because the law is very inadequate on these matters (in the US, anyway). I'm talking about, from a common-sense point of view, what "consent" is.
In my view, it can only be counted as "consent" if I have been informed exactly what it is that I'm consenting to, and I am asked for that consent.
ToS don't count for a number of reasons, starting with the fact that nobody reads them, it's unrealistic to expect people to read them (because everything comes with lengthy ToS documents and if you really read them all, little time would be left to do anything else), they are generally difficult to properly understand if you're not a lawyer, and they are are usually intentionally vague of this issue -- which means the "informed" part of "informed consent" isn't satisfied even if you do read them.
I think the Principle of Least Surprise should apply here. If software is going to do something that users won't notice happening, is intrusive, and isn't pretty obvious from the nature of the software, the right thing to do is to tell the user about it and ask for permission.
Software that doesn't do this is adversarial to the user.
That seems interesting, when thinking about pro rated subscription fees -- if a feature isn't being used, it'd cost nothing. But with telemetry disabled by default, the company couldn't detect that the feature wasn't being used, and thus the customer would pay more. And if opting in to telemetry, the customer pays less and the company gets telemetry.
Ok, but if we focus on results, why has software usability and functionality declined with the rise of telemetry.
I personally think it is because program managers rely on telemetry instead of stakeholder research and industrial design, either because they don’t know better, or because management thinks it will cut costs.
Besides, telemetry is a feature that 0% of users use, so it should be the first to be cut.
Subjectively speaking, while software I use these days has a lot less settings, I also tend to need to change settings a lot less than back in the day, defaults tend to be just right a lot more often. So at least for me it seems to be working, at least somewhat.
> Subjectively speaking, while software I use these days has a lot less settings, I also tend to need to change settings a lot less than back in the day, defaults tend to be just right a lot more often. So at least for me it seems to be working, at least somewhat.
But there's a trade-off there—as, ideally speaking, most software has become easier for most people to use mostly satisfactorily right out of the box, it becomes all the harder to adapt if you're one of the people for whom it doesn't work mostly satisfactorily, since there's no real culture of design around the idea that a user might want to use software in the way that they, rather than the designer, intended. Even if it's fair to write off those niche users, there are still the most users for whom the software works mostly satisfactorily, but can't be modified to be perfect, again because the culture of user customisability just isn't there.
Right. There used to be a design philosophy of making the defaults fit the majority of users, but to expose plenty of settings so that more advanced users, including those who have grown into the software, can still do what they need to do, and can work around design decisions that are bad for them.
In fact, it's something the US Air Force figured out after spending substantial time and resources.
That is to say: There is no such thing as the Average Man.
Make something that will generally fit most people, but then also add enough customization features so each man can tune it to his liking. There is a reason car seats and steering columns are adjustable.
The dichotomy isn't between telemetry and no options vs. no telemetry and all options. Somehow a decision needs to be made what to include because everything has at least an opportunity cost, and historically that has usually been made by "common sense", i.e. what whoever is in charge feels like is the right set of options. That can work out great if you're very much like that person or able to influence them (and they want to listen), but if you aren't, too bad, and the vast majority of people out there is not in either position. Late 90s/2000s me was probably a huge lot less like those people than many here on HN, and I guess that explains some of the difference in experiences. I generally seem to be quite a lot like "most people", which makes an approach that targets the middle of the bell curve work well for me. I'm not a big fan of embedding surveillance technology in apps, but I'd say the people who do are on to something and what's missing may just be a more transparent and privacy-preserving way of doing it.
My subjective experience has been the opposite. Modern software tends to be inflexible and limited, and those limitations absolutely adversely affect me. There isn't a day that goes by where this sort of thing doesn't interfere with what I'm trying to accomplish.
If you only care about users for which your software works, then most users spend almost all of their time as “power users” (= ones that have used the software for more than a few days)
Focusing purely on out of the box experience looks good for quarterly metrics (subscriber growth with low initial churn) but you end up with a population of users that are desperate to switch away.
At that point, you are one viable competitor away from being screwed. Worse, as users defect, telemetry will tell you that usage of the features associated with switching is declining, so you’ll double down on stripping out the functionality that you should be investing in if you want to defend your market share.
I don't think that a good case has been made for this. Telemetry does bring benefits -- it's cheaper than the old way, and captures useful things that couldn't be captured without it. But it also comes with some costs that are far from insignificant, especially for users.
I don't see any real reason to think that on the whole, telemetry has been beneficial for users. I think a stronger case can be made that it's beneficial for software makers.
The problem isn't that you have a feature used by 1% of the population. It's that you have like 60 features each used by 1% of the population, but with not much overlap. So all the 1% features get axed and suddenly you have only 50% of the users.
> So all the 1% features get axed and suddenly you have only 50% of the users.
In practice things rarely get that far, thanks mostly due to the complete farce that is "data driven design". Such an approach to design would naturally lead to the axing of every unpopular feature if it were followed mechanistically, but in reality "data driven design" actually refers to the practice of mining to support whatever you (the designer/programmer/manager/etc) have already decided to do, and the data is ignored for anything else.
It actually works like this:
I form a subjective judgement against some feature which I dislike (maybe because it's a pain to maintain or just because it doesn't suit my preferences for any reason.)
Then, I look in the telemetry and start making charts of things until I can find some combination of the data that seems to support my prejudice against the feature. I bury all the presentations of the data which failed to support or even refuted me, I pretend I never saw them.
Then I write up my report claiming that the feature should be removed for objective scientific reasons, and my fancy charts and graphs prove it! It's totally not just my subjective prejudices driving the design, no Siree...
Besides the privacy/consent concerns, this is why telemetry is bad for users. It devalues dialogue between the users and developers and replaces it with pseudo-scientific arguments based on nothing but the developer's preferences. That's what it's really for, an excuse to never talk with the users.
If you axe 60 features and there are 20 features left that are used by that 50 percent, then that might be a legitimate tradeoff from a business perspective.
I don’t disagree with anything you’ve said, and this is only tangentially related, but I wonder how much the long-tail of usage is impacted by the likely correlation of power-users/niche features, and people who firewall/pihole telemetry.
I also wonder how much of a chicken-egg effect there is on the trend of developers hiding niche/power-user features that they would prefer to sunset deep into settings menus and then that setting seeing negligible use.
That is usually information you already know from other sources
(e.g. is the 1% feature a feature that is pivotal to getting new users?)
(e.g. is the 1% feature a feature that is key for a specific audience within my total addressable market?)
But you have know from somewhere that the 1% is, in fact, 1%...makes sense?
You know that a lot of legislative and regulatory effort goes into making sure 911 works on as many phones as it does right?
> If this is the type of PM you are accustomed to, fire them.
People's expectations about how projects treat 911 are born from those hard requirements. If it didn't have those requirements, you would find that every PM is the kind of PM that would let the service suffer. Maybe not scrap 911 altogether, but certainly deliver a less robust and reliable 911 service than they do now.
the example I like to always use is when EA kept abandoning Madden (especially online franchise mode and gameplay) for like 10+ years straight (while making money hand over fist on MUT).
One time there was this new mode, for like 1-2 years, of 3v3 online. it was probably the most fun mode madden has ever had. it just needed a few tweaks. sensible stuff. But it would've been amazing!
but no. because it was difficult to get to in the menu system, fewer people used it and few people knew about it. so of course telemetry is going to say few people used it. but it had a TON of potential! a product owner made the wrong decision to cut that feature, and guaranteed they used misleading telemetry to justify it.
You cannot separate the two. Telemetry is primarily about the user. What features do they use? What is their pattern of usage? Etc.
So it's about the user as well as about the software. If it weren't, the only telemetry that would be of any actual value to you would be crash reports.
I think that 90% of telemetry is useless (eg. how often a feature is used says nothing about how important it is), but telemetry related to errors and crashes is extremely important.
My experience with shipping software is that if 50 people encounter a bug, maybe 1 of them will click the "feedback" button to report it. If you don't have a feedback button, it's less than that. If you don't have a technical audience, it's even less again.
So if you want to ship high quality software, then automatic error reporting is essential.
It’s telling that you have products absolutely jam packed full of telemetry, yet their “community support” forums will have years old threads with hundreds of replies about specific unresolved bugs and small tweak requests that go completely ignored despite getting bumped weekly.
This disconnect is common, stark, and very obvious to all. I've long been genuinely amazed that companies allow this state of things to persist. It makes them look really terrible, and certainly doesn't help to make the case that users sacrificing some privacy to the software maker actually results in benefits to the user.
And to add to the general disdain for users, many of the larger company support forum are staffed exclusively by volunteer “experts” that aren’t even associated with the company or have a direct line of communication to it. Heaven forbid someone working on the product actually sees what their users desperately want them to know.
I would be much more inclined to click a "report bug" button if software didn't put itself into an adversarial position by engaging in shady telemetry tactics.
But, especially these days, you pretty much have to regard all software as being a bit hostile.
On a related note, exploiting security vulnerabilities is often non-deterministic. There might be thousands of errors or process crashes before the exploit is successful, and if at least some of those get reported to your telemetry servers it becomes a goldmine for discovering zero-days which are actively being exploited.
Microsoft has people who hunt for evidence of zero-days in their telemetry, for example.
I think you’re correct in that the telemetry is to show managers, but once it’s in their central log aggregator, there’s another team that then uses that and other aggregated logs from other teams to make business decisions
Making telemetry opt-in would not have fixed this issue, the author is simply using this example to make an unrelated point about telemetry.
The real solution here is to remove all optional code from the critical path - this not only includes telemetry, but also things like in-app tutorials, automatic updates, crash reporting... If some code is not implementing a core function of the application, it should be allowed to crash and the application continue working. This is a core principle of good software engineering and it needs to be talked about more.
Absolutely this. The iOS mobile client of a system of I was responsible for once spontaneously stopped working, dumping users back to the login screen. The Android client was fine, and we couldn't see any issues in the backend logs.
After quite a bit of digging into the problem we eventually found that the iOS app was using a single error handler for all HTTP requests, including the one being made to query Status Page for any active incidents so they could be shown to users - our Status Page account had been erroneously disabled, which meant the requests were returning a 401, which the app interpreted as the user needing to re-authenticate. I'm not sure I've ever been embarrassed than having to report upstream that the cause of an incident was our solution for flagging incidents to customers.
> our Status Page account had been erroneously disabled, which meant the requests were returning a 401
Part of the bug here would seem to be this returning a 400-class status code: it should pretty obviously be a 500-. The most suitable standard code is 503 Service Unavailable.
If we're going to pedant about status codes, it seems like 401 is correct. 500 is a "server error" range, 503 would imply the service is down due to server side issues.
401:"not authorized" seems exactly correct for status page saying "I know who you are, and what you want, but you can't have it because you chose to disable your account."
I am assuming account-specific endpoints, which on reflection isn’t guaranteed, but is I think reasonable to expect. On a shared endpoint, 401 would be reasonable, but on a dedicated endpoint I maintain it’s unreasonable and incorrect.
401 is only permitted if an Authorization header is missing or insufficient, and the server “MUST send a WWW-Authenticate header field containing at least one challenge applicable to the target resource” (https://www.rfc-editor.org/rfc/rfc9110#name-401-unauthorized). Presuming an account-specific status page endpoint, this cannot be satisfied; nothing the client can send will work, therefore it’s far more reasonably treated as a server error than a client error: the server has been instructed not to service these requests. “Service Unavailable” is clearly suitable. (Refusing the connection would also be reasonable, and have certain technical advantages for machine use.)
It's more common than you think. The Liftmaster MyQ app for my garage door openers does the same thing: Any kind of error and it knocks you back to the sign in screen.
Wish there was a good solution for this sometimes.
I maintain some small open source projects (which I wrote for my own benefit), and judging from GitHub issues and stars, get some use.
But I have absolutely no idea who's using the software, or which subsets of features they care about.
I'd happily put work in to improve things users cared about, but have no idea how to get that info (I don't know who they are so can't ask), or using telemetry (which I'm loathe to add due to its perception).
Just ask users for feedback. We run a zero telemetry web browser and have one of the busiesies feedback boards [1] I ever seen for a browser. Never in a single moment I wish I had more data, and actual user feedback is so much valauble than any telemetry.
If your telemetry is human readable (like a dialog that says “can I send details with the features you’ve used” where it actually shows what is sent, you should get SOME response. Those kinds of telemetries I love.
I know that I'm jaded (and I would argue not without reason), but it's hard for me to trust that software is fully and correctly disclosing what data it's sending.
What I've seen some software (including some enterprise software) do -- and what I do with my own products -- is generate a human-readable text file that the user can email to the company. Then the user can be 100% certain of what data is and is not being reported. It also gives the user the opportunity to redact any data they are not comfortable sharing.
I'd rather have my bugs fixed 1000x faster with opt-out telemetry. Most people don't change the default settings, so you'd get significantly less crash reports with opt-in telemetry.
This is, really, the heart of the problem. There is very little trust. And the software industry, as a whole, is the reason. The amount of abuse we as an industry have engaged in, and continue to engage in, is remarkable.
Repairing that trust is something that will take years of good behavior. We as an industry haven't even begun the first steps down that path, though.
> Use a trustable third party telemetry service
Does one exist? And if so, how would you or I -- let alone an ordinary user -- know it was trustable?
"There should be more trust."
Nope. Trust is always earned, never given.
Why? People, Corporations, NGO, Governments are all broken in some way and are basically as evil as you allow them to treat you.
I recently opened Minecraft for the first time in several years to smack a few blocks. Their “telemetry settings” were my favorite. Options? “All” or “minimal”. Unsurprisingly, the game still ran after I disconnected from the Internet to disable all telemetry.
My favorite is when my young kid old yelled that there was an error screen about a minecraft profanity filter, but “fixed it” before I could read it.
I foolishly set up my microsoft account, github, and linkedin using the same email address, so now it is just a matter of time before Microsoft crosses the telemetry streams, and I get insta-cancelled for naming something “kil the pooppy crepers!”
Yes, Microsoft also created a social media profile for a kindergartner without any sort of notification or parental consent screen. I don’t see why they get a free pass to ignore all the “protect the children” laws.
GDPR deals with user identifiable data or that can be tied to the identity of a user through partially anonymous data.
It should be fine to collect "telemetry" as long as the data itself isn't personal and that it also cannot be tied in any way to a particular user.
At least, this is how I've interpreted the GDPR rules.
On the other hand if you tie telemetry to a user ID, fingerprint, unique device identifier, etc, then all the data is personal, and since it's not obvious and necessary, you also cannot argue its a legitimate interest (note that most GDPR popups arguing legitimate interest are absolutely not valid).
I can't believe how much BS we have to hear about consensual sex these days to the point people are suggesting contract before making love, and yet the virtual signalers in the valley can't still see the problem with forcing people to be spied upon even when told to piss off.
Maybe too many new web framework turn over also burnt out their common sense?
Silicon Valley still seems to have a huge problem with the concept of user consent and giving users meaningful choices. If "Silicon Valley" was a human being, it would be that creepy guy in the club hitting on everyone, saying "Do you want to be my girlfriend? [YES] [Ask Me Again Later]"
You could also show the same person, 10 minutes prior, telling their locksmith they should say "main-key" instead of "master-key" because that's inappropriate, otherwise they won't call them for more gigs.
>people are suggesting contract before making love
I would unironically require this simply because the legal liability risks are too bloody high otherwise.
Not that I would get myself into such a situation in the first place. In this day and age there are many other avenues of entertaining oneself without the ridiculous liability risks.
I disagree. For example, Jetbrains asked me nicely if they can enable telemetry for their IDEs. So I did. Windows didn’t ask. So I use O&O ShutUp to make it stop.
Assuming that's true, then you can't collect telemetry. What's the alternative? "We're going to spy on you against your will because it's profitable for us."?
Although it is non-trivial work that you need to do with each update, but you can still remove all telemetry from Firefox through `about:config` tweaks
I don't think this is always true. When I get an error message or some sort of error while using a program on my private PC I don't mind sending the error code and telemetry data to the devs, because it hopefully helps them to find and hopefully fix errors in the software I am using and therefore is a win-win situation in my opinion at least.
It is always true that you should ask. In this case, you have no qualms living with opt in, versus opt out. In this case, you can live with the minority rule [0] with no impact to your function.
Amos (Rust's fasterthanlime) has pointed out that opt-in causes a problem which the "Eh, they can always opt in if they care" crowd don't spot until it's too late.
The telemetry is collected at time T, and your decision that you care must come before time T if it's to have any effect on the outcomes. After the telemetry is collected, and decisions are taken based on that telemetry, it's too late to opt in.
This reflect a common problem with humans that what they don't like is regret, but what we offer them as a fix is informed consent -- it turns out what they wanted to be informed about was really whether they would in future regret it, which we cannot know.
Now, as a democracy we "fix" this by just deciding that even if you refuse to participate in the process we don't get to claim that your interests don't matter. That feels like a reasonable price to pay for very important decisions like "Should we build a freeway across this land occupied by the people from this weird religious sect?" but it feels way out of proportion for something like, "Should I add more animated Emoji reactions to version X, or should I work on enabling the text-to-speech feature in group chats as well as one-to-one ?"
Laws like GDPR can help organisations stick to collecting only what they actually needed to know and not just "Everything because why not" - GDPR is why not. Did you just collect the full name of every user? You'd better have a damn good explanation of why you did that. Whether user has <4GB, 4GB-8GB or 8GB+ of RAM? Very easy to justify.
In this worldview, it is the Boss who makes the Worker work, because the worker is lazy and cheats at every opportunity. It is closely related to "objectifying the Other" where character traits and behavior patterns are cast upon the Other in interaction, without regard to detail.
I think a slightly dark pattern of making a "Yes" more visible (or default on enter in a CLI like Y/n) is an acceptable compromise. Informed consent, but biased.
Indeed, as it should be. You should always ask the user when you collect their data, for each transaction. If your software then is annoying as it performs, that's on you as the developer. Fix your processes.
Your test harness should cover edge cases. Invest in that, versus the externalized cost of collecting telemetry.
Your users are not your labrats. Quit pushing your testing costs on to them.
Despite a 100-fold increase in hardware power, and languages/frameworks that handle 80% of the workload a developer 15 years had to think about, applications these days have fewer settings, open and run slower, and improve far less rapidly than anything written in C/C++/Java from 1990-2005 despite phoning home with novels worth of telemetry every few minutes.
Monitoring network traffic and DNS requests through a firewall and pihole, it’s downright hilarious that it’s the worst products that top the charts in both, even when being online is not a necessary component of the software. Meanwhile the software that just works, and works well, might check for an update once a month, and the changelog generally reveals new useful features and mentions performance increases.