Hacker News new | past | comments | ask | show | jobs | submit login
Despite having just 5.8% sales, over 38% of bug reports come from Linux (reddit.com)
1292 points by otreblan on Oct 24, 2021 | hide | past | favorite | 255 comments

Notably, of those bug reports, fewer than 1% (only 3 bugs) were specific to the Linux version of the game. That is, over 99% of the bugs reported by Linux gamers also affected players in other platforms. Moreover (quoting from the OP):

> The report quality [from Linux users] is stellar. I mean we have all seen bug reports like: “it crashes for me after a few hours”. Do you know what a developer can do with such a report? Feel sorry at best. You can’t really fix any bug unless you can replicate it, see it with your own eyes, peek inside and finally see that it’s fixed. And with bug reports from Linux players is just something else. You get all the software/os versions, all the logs, you get core dumps and you get replication steps. Sometimes I got with the player over discord and we quickly iterated a few versions with progressive fixes to isolate the problem. You just don’t get that kind of engagement from anyone else.

Indeed, although my kneejerk reaction to the subject line was to expect another sad article about how it's not worth it to support Linux, this author seems very happy with the decision to do so.

> Worth it? Oh, yes - at least for me. Not for the extra sales - although it’s nice. It’s worth it to get the massive feedback boost and free, hundred-people strong QA team on your side. An invaluable asset for an independent game studio.

Maybe there's a better term to throw at this. "Even though just 5.8% of userbase, Linux users contribute 55% of helpful QA feedback"

Makes total sense. You're unlikely to use Linux if you're not something of a power user, at least. I would guess there's a lot more developer type people who use Linux, and they are the people who appreciate a good bug report.

Same people also have a better idea of what might be causing the bug, how to get more information, logs, and so on.

They're also probably more used to the development process. With proprietary software, if you don't pay for a support contract, you basically can't submit a bug report - not really. You just ignore the bug and hope it's fixed in later versions.

This applies to big AAA games too, so games aren't necessarily special.

But indie games are in a weird spot where:

1. They're clearly artists. Nobody who want to make tons of cash becomes an indie game developer. If you have the aptitude to grind through the docs and make a game, you have the aptitude to grind leetcode and get a comfortable job somewhere that pays benefits and cash. And since they're artists, and they're making games, people empathize deeply with them.

2. They get a bit of a pass on the whole free software zealotry thing. I'm sure some people are still hard-line about it, but like, if I had to rank software in terms of freedom-threat-if-proprietary, something like Office or a compiler and towards the top, and Valheim is rather closer to the bottom.

3. They really have to listen to their players to survive, so they end up operating a lot more in the open.

So they end up looking a lot more like an Open/Free project than you would expect.

I think the title was intentionally a little clickbaity but I'm not exactly bothered by it. Clickbaity is a negative when the bait-and-switch robs you of the pay-off. In this case, I think it was overall a more interesting story and so for me subverting my expectations was actually a pleasant experience.

OTOH, it could have a negative effect if people only read the title and not the actual content.

I didn't read the content, but I trust the HN voting process to get the juicy comments on the top :)

Fair point. Unfortunately, there's no helping some people :)

Reminds me of a tweet from Yann LeCun a while back where he bragged that they take down more reported content than other social sites. While at the face of it, this seems good, what it really means is that their system is worse at proactively taking stuff down before it gets reported. If they took everything down that violated their terms of service before it was noticed by someone else, they'd have a 0% follow through on these take down requests.

Multicapitalism might be the more academic term.

A developer invested time to support linux not necessarily for financial return but social capital returns in the form of more helpful QA feedback.

Wow what an interesting term. Any idea where I can learn more?

There's a few forms I've bookmarked:

- Follow Bill Baue and the Reporting 3.0 folks - https://twitter.com/bbaue/status/1342101339120279552 - https://www.r3-0.org/wp-content/uploads/2020/12/r3-0-White-P...

- Joe Brewer on Multicapitalism and producing ecological flow: https://earth-regenerators.mn.co/posts/the-hidden-flow-of-re... skip to 16minutes for this model, references "the atlas of economic complexity" - https://atlas.cid.harvard.edu/ (bias: I sorta collaborate with Brewer on Earth Regenerators)

- Ethan Roland & Gregory Landua on 8 forms of capital : http://www.appleseedpermaculture.com/8-forms-of-capital/

The worst part? I've seen developers and companies actually complain about this. Just 6% of sales yet look at the huge amount of work they cause us!

This is the case with people selling business software, meaning bugs would mean wrong accounting or worse outcomes for the customers. Yet all layers in the company would treat these reports as an annoyance.

Huge amount of work they do for us

Marketing/PR depts never see it that way. They open up new tickets, and that's the metric they care about. Not justifying it, but marketing depts rarely get it right.

Marketing of Facebook was of no use when Facebook was laying flat not so long ago. It shows which department should really have a definitive voice in product metrics.

> marketing depts rarely get it right.

Depends on whether you feel selling people stuff they don’t need is ‘not right’.

Are there any good articles on writing helpful bug reports? Whenever I submit a bug I try to give as much detail and be as specific as possible but I always imagine some dev somewhere reading it rolling their eyes at the unnecessary paragraphs I’ve submitted.

It's interesting to compare that guide "How to report bugs effectively with ESR's "How to ask questions the smart way" [1]. They both cover similar material, but the styles are extremely different. The latter is rather hostile to the reader: "If ... then you are one of the idiots we are talking about." "If you decide to come to us for help, you don't want to be one of the losers." It's also heavy on "us vs them", how we are experts and you need to treat us properly.

You know, it just occurred to me that the "How to ask questions" document is targeted as much at hackers and how they should maintain their "standards" than at users who are asking questions. For example, the document has approving examples of "logically impeccable but dismissive" hacker answers; these make more sense as instructions on how a hacker should respond than as something relevant to a user.

I guess my point is that I had read the "How to ask questions" document for decades and viewed it as an objective document, not realizing how arrogant and "gatekeeping" it is.

[1] http://www.catb.org/~esr/faqs/smart-questions.html

I agree, that "how to ask questions the smart way" article always left a bad taste in my mouth. Perhaps there should be a "how to answer questions" companion piece.

esr seems to have a pretty big ego, just based on his writings. (e.g. at one point he declared that he can tell if someone is smart or not just by looking at them)

He actually at one point wrote that he was a reincarnation of the god Pan, so he has/had a very literal god complex, too. It's such a shame that he was the person to interpret hacker culture for those of us in the future, because he's hardly a good lens

I don’t disagree with your sentiment, but that wasn’t meant as a literal “I am the god Pan” — it was figurative, hyperbole. At least, that’s my recollection from when I read it a couple of decades ago.

I've read it more recently, and he did mean it literally. I've cut this together because it's far too long to quote en masse, but the full thing is at [1]

    Desperate for something to feed my jones, I snaffled my other sister's 
    abandoned flute. And wow! I was a natural...immediately better with it than 
    with the guitar I'd been hacking at for months. [...] This was delightful but 
    mystifying. All I'd had to do was learn to play a scale, and this amazing river 
    of music poured forth with barely an effort on my part. It seemed almost as 
    though my hands and lips had always known what to do, had been waiting for me 
    to pick up the flute. [...] I got these stunning rushes of pure timeless joy, 
    when my consciousness seemed to expand outwards from the limits of my skin to 
    fill the universe and I could no longer tell whether I was playing the music or 
    the music was playing me. Nor were these effects just going on inside my own 
    head. [...] I was walking home, idly puzzling over this peculiar incident, and 
    damn near fell over when I finally got it. That girl had been trying to cope 
    with a theophany; she had looked at me and seen a god. A particular god. And I 
    knew, suddenly, with utter shattering certainty, which one it was. And that it 
    probably was not the first time I had inadvertently triggered such an 
    experience, and would almost certainly not be the last. [...] Not that I took 
    any of it seriously as a description of the real world. It was an intellectual 
    chew-toy, perhaps at best a way of understanding the pathologies that prevented 
    human beings from living the infinitely more desirable life of reason and 
    science. Until I realized, finally, belatedly, what had been happening to me. 
    Until the Great God Pan reached out of my hindbrain and thundered "YOU!". And 
    his gift is music and his chosen instruments the pipes and flutes. And his, too 
    the power of joy; magic so strong that when it flowed out of me, even before I 
    knew what I was doing, it amazed people into awe and incoherence and poetry. 
    [...] (And, oh, yes. The first time I handled a set of pan-pipes I could play 
    them. Fluently. Effortlessly. And knew I could before I touched them.) 

[1]: http://www.catb.org/esr/writings/dancing.html

Strange, he's old enough to have watched the first TV run of Lt. Columbo ...

The https://WhatHaveYouTried.com guy backtracked after a few years in the linked follow-up article, ashamed of giving the geek world another way to gatekeep and tell unworthy people to get lost.

Gemmell made a mistake in assuming the shame and guilt of the people who abused his article. The article is just a tool. We should hold the abusers responsible for their gatekeeping and telling others to get lost.

Thank you! I’m definitely guilty of mongoose-ing and diagnosing in my bug reports so this is very helpful!

> diagnosing in my bug reports

It's probably fine, if you put it in a separate section, clearing indicating that you are guessing.

At work we have an optional "diagnostic" section in our ticket template, mainly intended for the team, but I'd be happy to see a user try to fill it. At worse it's harmless, at best it gives the actual reason, somewhere in the middle it can give ideas even if it is wrong.

> At worse it's harmless

I wish. It happens very often that such speculation derails the whole investigation. It really shouldn’t, but people are people. If the title of the bug report says that the foobar is broken it might take weeks until someone realises that they were wrong and actually the problem is in the frobnicator which feeds the foobar.

Especially when because of this we assing the investigation to the wrong team. The foobar people don’t want to appear as if they don’t take the bug seriously, but all they check everything looks normal on their side.

As a dev, I think

STEPS TO REPRODUCE Open foo click bar


ACTUAL RESULT crashes with error x912344

MISC <any other logs, details here>

is the best. It gives just the right amount of detail to get started, in a good, actionable format.

Yeah, that was basically most of the bug template we were filling when I used to work QA at Warner Bros. Games.

    Summary :
    Steps to reproduce :
    Expected result :
    Actual result : 
    Build/Version :
    And a few more fields about platform, severity, etc. on Jira.

Lucky you to have that standard enforced within the org! We don't so I pretty much invariably have to respond to "bug reports" ("foo doesn't work pls help") with "can you please format your bug report like this?" haha

Out of curiosity, what does deving for Warner Bros mean? Like working on their internal systems and stuff? Netflix integrations or something? Sounds cool, bet you get to see movies before they come out too!

Sorry for the delay!

I was working at their Montreal Studio. It's exclusively a video game studio. I was mostly working in QA on Middle-earth: Shadow of War.

Even if the game is technically developed by Monolith Productions, Warner Bros. Interactive Entertainment (WBIE) helps with their ressources (Artists, QA, Devs).

There's also alot of internal tooling for Builds automation, Automated testing / QA Aids tools, etc that is handled by WBIE as the game editor.

We didn't see movies before they got out but we had access to a 30-50 persons theater room that I was able to "rent" for free outside of business hours to watch movies with friends and show them the non-secure areas of the office. (Cafeteria (Free drinks!), gaming room, etc.))

I'm not really into commics but we had a room with thousands of comic books that we could take and read anytime we wanted (WB owns DC Entertainment).

I worked there 6-8 months when I was out of school before getting my first dev job.

No sweat, I'm glad to see the reply at all :) those perks do indeed sound really cool haha

Very important to also ask: Which application. Get an URL if at all possible. If someone says 'the calendar program is down', I have at least 5 candidates, and 50/50 chance something I've never seen pops up.

Users tend to rename programs, even give them names of other programs. Our entire finance department can't wrap their heads around the fact that 'Oracle' is not a bookkeeping application. Worse, they are now convinced the new version is called 'fusion'. Oracle financials? Never heard of.

Use this template:

1. When I do this

2. Here is what I expect to happen

3. But instead, this happens

One more tidbit: In the "When I do this", I find it useful to omit stuff that isn't interesting. Does the thing fail if I do something immediately? Or does it only occur if the application has been dormant for a few minutes? Does it only occur using the mouse, or can I use the keyboard?

Get a good set of steps for reproducing the problem, but wander a little off those steps to see what is actually necessary. I've been surprised more than once when something that I thought was incredibly complex only required a few steps. And also been surprised when a complex repro was actually complex (e.g., if I drag quickly it works, but if I drag slowly it doesn't? what?).

It's probably better to err on the side of having too many steps rather than missing cruicial steps to reproduce the problem.

...4. And this is my software version and Linux distro.

Also useful to run the app from the terminal, replicate the bug, and see if any logs pop up.

Ugh no thanks, I'll just send them a screenshot and say "it's broken, please fix this!"

If it's good enough for my colleagues to submit tickets like this, it's good enough for me!

From "Post-surgical deaths in Scotland drop by a third, attributed to a checklist" https://westurner.github.io/hnlog/#comment-19684376 https://news.ycombinator.com/item?id=19686470 :

> GitHub and GitLab support task checklists in Markdown and also project boards [...]

> GitHub and GitLab support (multiple) Issue and Pull Request templates:

> Default: /.github/ISSUE_TEMPLATE.md || Configure in web interface

> /.github/ISSUE_TEMPLATE/Name.md || /.gitlab/issue_templates/Name.md

> Default: /.github/PULL_REQUEST_TEMPLATE.md || Configure in web interface

> /.github/PULL_REQUEST_TEMPLATE/Name.md || /.gitlab/merge_request_templates/Name.md

> There are template templates in awesome-github-templates [1] and checklist template templates in github-issue-templates [2].

> [1] https://github.com/devspace/awesome-github-templates

> [2] https://github.com/stevemao/github-issue-templates

bug reports and communication in general: If you've got a good summary/abstract more detail is always better. I don't know if I've ever had to talk to someone on my team about how they "over communicated" (not the same as TMI!). Make the receiver responsible for saying "OK, that's enough, got it!"

Conversely I think you should generally write the smallest amount of text you can. It maximizes the chance someone else will actually read it.

I sometimes try to follow the journalistic reporting way of writing. (Not the clickbait style mind you.)

So I start it off with a few brief paragraphs of text explaining what I did, what happened and what I expected to happen. Each paragraph only a couple of sentences long.

Then below that I fill in with more details.

Of course this still means that if you glance at it you still see a lot of text and may not bother reading. But my hope is that if by chance the recipient does read the first few paragraphs then it will provide the necessary motivation to either read on or to begin investigating on their own.

It's best to do something like

1. config and versions

2. the bug (failure to see expected results

3. how to reproduce in as simple terms as possible

4. any data such as screen shots, core dumps, logs, etc

Think about what you would want to know if someone was reporting the bug to you. Provide at least that much information.

Usually sending just "help" and no other data is preferred. Don't send screenshots, don't make use of the ingame reporting tools (pressing F11), don't send steps to replicate.

I’ve often thought that game developers, esp indy studios, would benefit from doing first beta releases on Linux exclusively - wringing out a bunch of bugs, then doing a wider release.

Well that might be one way to _accelerate_ your dev process.

maybe alpha, beta level you gotta be on all platforms.

>"we have all seen bug reports like: “it crashes for me after a few hours"

Here is a gem from my collection of user "bug reports". The email with this exact words and nothing else - "something is wrong, what do I do?". I was biting my fingers trying not to reply with the first thing that came into my head.

"We are sorry to read that. Please clearly describe your problem to us so we can help you. Feel free to call us on this phone number: XXX-XXX-XXX"

People don't always understand our perspective, we should guide them into doing the right thing and everybody will be happy.

Save your generic answer so you don't waste your time doing so, but being pedagogical should be part of our work in my opinion.

>"We are sorry to read that. Please clearly describe your problem to us so we can ..."

What makes you think I do not do those things?

I don't :-)

I was speaking to everyone. Apparently I got your last sentence right.

Interpreted literally it’s a reasonable question: “What process should I follow when I encounter a problem?”

Well try to visit car repair shop and ask the same question as in the original email and then explain them what you really meant.

I don't hold myself and frankly answer that my crystal ball is on scheduled maintenance today.

So when will your crystal ball be ready, could you then asap adress the issue? Also keeping spare crystal balls would be really helpful.

We are a small company supporting a free opensource product, we can't afford spare crystal balls. Even the one we have was bought on a secondary market, warranty expired in late 14th century.

Conclusion: want good testing and bug reports of your game? Support linux.

Or have proper analytics. If a user emails you and says "It crashes after a few hours" rather than feeling bad for them, you ask them what their steam/game id is and you look them up in the crash reporter system and see exactly what happens rather than relying on a few programmer consumers doing the work for you.

But not analytics that are too good, or certain people will write a lot of inflammatory articles about how your software is nothing but evil spyware.

Seriously, though, having a robust and automatic crash reporting system very much helps track down bugs you're never going to get a good report on or be able to directly reproduce.

It's not really that hard to do this: ask bug reporters to attach their logs. Don't just take them preemptively "just in case".

If I asked users to attach a bug report file, most of them would complain and tell me to stop waisting their time and get the problem fixed. I have many times explained to users that I found their issue in our reporting and have not ever got a negative response.

The average user does not give two shits that you can see exactly what line of code their instance of your game crashed at. They want to play a video game and not play find the bug with you.

The vast majority of users would not even be as nice as to email you saying there was a crash. They would just immediately complain on twitter or refund the product if it wasn’t resolved without any interaction.

You're in a thread talking about Linux users who file good bug reports.

The average user also has no idea what/how much information is contained in those automated reports. If you are just sending a build ID and a stack trace, sure. If you are sending memory dumps (even partial) or associating those stack dumps with identifying information (e.g. steam IDs) then you really should be asking for informed consent before you do that. And no, it being a common practice to just collect as much information as you can does not make it an OK practice.

Possibly, but how many of the bugs are rare edge cases (i.e., attempting to break the game) versus problems players actually face in normal gameplay? Without knowing what priority or severity these bugs take, it is difficult to say how useful the increased volume of feedback is.

Even if someone is not attempting to break the game, they might accidentaly hit those rare edge cases.

Of course, you could infer the usefulness of the increased volume of feedback from the fact that a game author is extremely happy about it.

Overall, I don't see any particular problems with having reported issues. We definitely don't pay cloud storage per open issue or anything like that. Initial assessment of a bug report is not particularly difficult either.

I'm not sure if your concerns translate to actual negative impact.

Depending on what kind of game you have, those exotic edge cases can come to be the dominant behavior in games, particularly multi-player where slight advantages can lead to some players dominating every one else.

I'm not a gamer, but from what I've read, the whole industry has been transformed by the incredible lengths people go to, to cheat.

So I would expect that edge cases might be really relevant to mainstream use because everybody is trying to break the game.

FTA it sounds like basically all of them were normal gameplay that affected all platforms. only 3 were linux specific.

Yep, it’s a great point to raise in support of releasing software for Linux. Adept users who can produce high quality bug reports and help solve bugs across all platforms provide a lot of value, perhaps enough to justify “only” a ~5% market share.

Just from the title, I went in expecting that percentage to be much higher, but still expected I'd get a chance to complain how report quality was not taken into account (I expected Linux users to submit better reports)!

Pleasantly surprised on all accounts :D

Sometimes I forget my detailed bug reports with steps I took, even if I can't reproduce it--since it might provide a hint to someone with insight into the programs internals--, are a weird outlier.

more likely s/linux/developers, which i suspect represents the majority of linux users

God damnit... Clickbating is ruining any shred of useful information...

I am not sure why would one expect more linux bugs in a game made using a game engine (I couldn't find any info and I am assuming this game is one, correct me if I am wrong). If it is not working in linux, then it must be an engine bug, not a game one. If I am wrong then props to devs for having such a consistent game engine

Perhaps other option is using cpp, and having UB that behaves differently in different compilers. But I doubt that as well

OP was not saying that 38% of the bugs are Linux-specific, he was saying that Linux users report many more bugs per capita than windows users (and he sees that as a good thing).

Yep we find a variant of this with our python open source data science users who normal businesses would hate:

- Across all Python envs (Jupyter, streamlit/plotly, flask/Django, ...), they report way more bugs, and often from our free tiers, which sounds low ROI.. but our community caught all but ~2 significant bugs we fixed this year that our internal automated testing missed and would have otherwise gone to our Enterprise and gov users . They are happy to exchange a tiny bit of GPU or viz burps for new features faster, as long as we are responsive to their reports! That also pushed us to a better dev cadence which wouldn't have worked for our Enterprise users. More subtle, we have more confidence for deploying wider features/APIs which would otherwise be too underused + risky for the same reason.

- they are often doing innovative new things with our stack, which originally meant feature requests for non-repeatable sales.. but ended up stretching our flexibility into wider applicability, and they can clearly tell us areas to advance the visual, analytic, etc sides of our R&D and package it up for the rest of their internal analysts. Almost like a multiplier for conversations with our more operationally focused no-code/low-code users, and helps us see around the corner for them in the AI+API+customization sides!

- most of our technical partnership discussions now go through our OSS client repos. Rather than PowerPoints with BD people that don't have staff to do things, a product dev or sales engineer/architect or devrel blogger will think, "oh our customers would benefit from better interactive visuals on our data, let me whip something up quick with that package" and we help them take it from there

- open source data users like PyData developers and data scientists, compared to say the # of SMBs out there or general enterprise employees, is small and almost by definition doesn't pay for most software... but they end up being attached to projects that do, and influence much of their organization that definitely does!

We're just used contributing bug reports and pull requests, sort of our thing. It's not as prevalent with Windows or MacOS users, just not part of that culture.

Great article, though he's benefitting from a specific slice of Linux users that are great to work with.

On the other end of the spectrum, try releasing a php plugin for Wordpress, OpenCart, Magento, etc. I released a mildly popular, open-source, free add on in this space. The Linux users there are decidedly less technical, but still doing it themselves.

What I got for "bug reports" was mostly terrible, few details/logs/etc. And a weird entitlement thing where people had really high support expectations over something that was open source and free. Even a few expletive-laden emails. I did receive a few really helpful reports, pull-requests, and thank you emails...but they were very much the exception.

I think the difference is:

With a game that supports Linux, you get people who have chosen to run Linux instead of Windows because they prefer it. They have also worked enough with their system to get their Linux system capable of running games well, which may not be trivial for the given distro.

With a Wordpress plugin or whatever, you get people who are running Linux because they "have" to. Trying not to sound "true Scottsman"y here, but most of the time these are not actually Linux users, these are Windows users who are further confounded by having to use an unfamiliar platform.

There are even a few php users trying to run windows server...

Yes, you get users who have linux as their 'daily driver', as opposed to those to dabble in it a bit on the side :)

For similar reason, I think the level of experience is probably different. Meaning those could be the same person and time is the important variable. After they’re used to developing, they better understand what useful feedback looks like.

This is actually how I got started with development. I learned how to use Linux just enough to set up WordPress (and SMB) on an old computer. This got me into PHP development (I'm recovered now) which turned out to be unbearable on Windows. I started running Linux in a VM for development, but my pathetic hardware couldn't really handle it, so I tried it on bare metal. I'm now running some flavor of Linux on my workstation, laptop and multiple servers and am exactly the sort of crazy person who will spend hours of their unpaid time compiling a detailed bug report with stack traces and all for even fully proprietary software.

In addition, in my experience maintaining a developer tool, an outsized source of bug reports is Linux users reporting already fixed issues in their outdated distro packages, or even problems introduced by their packager (e.g. flatpak/snap permission problems). And they typically won’t identify the version despite clear instructions in the issue template asking for the debug output containing everything I need, including the version.

If you support distro packaging of your software (which you should, since it should ideally reduce the load on you when it comes to triaging bugs) and users are coming to you with distro specific bugs then you should:

- Immediately close issues where the user has indicated that they did not test against the latest version or where the user has not answered the question "did you reproduce this issue with the current version of the software built from source?" (You should just automate this)

- If users continue pestering you and they seem to come from a specific distribution: complain to the maintainers of the distribution and ask that they inform their users of the proper bug reporting channels.

New users are coming to linux on a regular basis and they're coming in with a windows mindset. It is important for the smooth functioning of both linux distributions and software projects to make users aware that the proper channels for reporting bugs they experience when using software which was packaged for their distribution is the distribution's maintainers.

Users need to be made aware that unless they're compiling directly from source and using a supported version then they have no business going to upstream with their bug reports.

You may think this is harsh but if you get backlash, distributions should have your back on this (they usually do have information somewhere to inform users that bug reports should go to them first). I recommend any open source project take this stance when it comes to bug reports. If someone finds a real bug and they are certain it's not one caused by their distribution then they can easily build your project and reproduce the bug there.

(Aside: If you are providing a library then an appropriate level of API stability is a must have if you want people to be able to actually test bugs in a newer library version.)

I strongly disagree - I'd rather all my users filed their bugs directly on the upstream issue tracker for my projects so that I can get a proper view of what bugs are being encountered. If there are steps to reproduce then most of the time it will be a lot easier for you as the developer to just check if the bug still happens with the latest version than having the user or packager build it themselves. Don't expect users just have a build environment laying around or can quickly set one up. Don't expect that building the latest version is always easy on every distribution.

If its not easy to reproduce or if there are no reproduction steps then sure, ask for more information. It's also OK to refer to a distribution bug tracker if the user is not able to produce a useful report or if you suspect there is a packaging issue, but doing that for all reports of problemsx with distro builds, especially when using automation, is disrespectful of the time the user has spent on reporting the bug. So sure, you can do that (you don't owe users anything after all) but I think its rude and will only turn potental future contributors away. I'm definitely glad that that is not how most projects handle user reports.

If you are being overwhelmed by user reports then maybe you can find some dedicated people from your userbase that would be willing to help with bug triage?

> Don't expect users just have a build environment laying around or can quickly set one up. Don't expect that building the latest version is always easy on every distribution.

In the case when a user is not technically skilled enough to test for the bug in the latest version themselves they should go to the maintainer whose job it is to be able to do that.

> but doing that for all reports of problemsx with distro builds, especially when using automation, is disrespectful of the time the user has spent on reporting the bug.

What? How is it disrespectful? They're helping their distribution and their distribution should be forwarding the bug onto you if it's not their fault.

In fact, if the distribution is aware of it then the user is more likely to get a hotfix sooner than if they go straight to you.

> So sure, you can do that (you don't owe users anything after all) but I think its rude and will only turn potental future contributors away. I'm definitely glad that that is not how most projects handle user reports.

It is actually how most old style open source projects operate. If you're happy to accept direct bug reports which are either already fixed in the latest version or which may be distro specific issues then go ahead. Please don't burden every other project with this kind of noise by making it clear that this is a project specific policy.

Don't blame the user here - your bug reporting template should have users declare what package version they're using so that you can easily tell them they're on an outdated package. Flatpak is usually a boon in this regard since you can ensure your users are all on the latest packages, though you're making the tradeoff of having to deal with any Flatpak-specific bugs (which I'd say is a decent tradeoff for solo maintainers).

> your bug reporting template should have users declare what package version they're using so that you can easily tell them they're on an outdated package

Of course I do that, and I said so. I do everything I can to make it trivial for users to give me all the information I need. They just don't give a shit. Some people won't read.

(And the extra irony here is that my original comment already mentioned issue template asking for everything, yet you didn't read it and jumped in to tell me "don't blame the user".)

Edit: I should also mention that this is FOSS work. I’m doing free support for these users who supposedly shouldn’t be blamed for wasting my time.

Bug templates help, of course. But people do still just dump the wrong stuff in until the report is accepted. Like some kind of OS version in the App version field, for example. Or some ambiguous thing like "latest". Or copy/paste from an existing bug report from some other person. You can see this on many public repos.

> Like some kind of OS version in the App version field, for example.

Oh, I would be grateful for even that. Often enough, the bug template is simply deleted, the last line of the error in non-debugging mode is pasted in along with a "doesn't work"; or just "doesn't work".

While I don't do this intentionally, I do use a native Android app (fasthub) to submit issues. So far as I know, there isn't any GitHub app that has support for issue templates. Occasionally I'll submit an issue without remembering to check for a template first, though I usually remember before actually submitting.

Congratulations for acting exactly like his bad bug reporters.

Most of the Wordpress developers I know are Windows users. (I don't know too many of them, but still.)

That's interesting. Most of the WP devs I encountered are deploying on old-school shared Linux hosting, like BlueHost, GoDaddy, etc. Maybe some use Windows for local dev stuff, but I doubt many deploy their "prod" there.

I think that's the issue though: You're seeing "Linux users" that only use Linux on their servers because they are somewhat forced to do so.

I would actually expect that almost all of those people use Windows or MacOS for their local development.

And those hosts often are not using up to date packages and don't even have up to date security fixes at times.

I recently moved a wordpress site I was working on from a local dev setup onto a live bluehost server and was immediately hit with a bunch of out of date package warnings. As the customer was using some cheap shared hosting service, there wasn't much I could really do about it.

Shared hoster here. More likely the app arbitrarily decided 'these versions of PHP are EOL', where the reality is that they're being maintained by a number of OS vendors.

Too many customers run really old versions of code that need older versions of PHP and so on. We can't not provide them without losing the business. We do of course, make newer versions available.

The other fun one we periodically get is customers doing PCI scans or similar and getting warnings for out of date versions of OpenSSL, Apache, etc. Nope; just backporting showing the old version numbers.

That's sort of funny, as one of the remaining few value pitches for something like Bluehost (versus a $5/month vps) is "somebody else does apt-get update && apt-get upgrade for you".

Keep in mind that WP has a vast installation base, so your local sample is not necessarily representative. I only know WP admins that a fairly comfortable with Linux and use it on a daily basis.

Not a lot of Windows in the shared hosting space. And what little there is tends to be more expensive, from what I have seen.

And, perhaps other than knowing about permissions, you don't really need any Linux knowledge to use these hosts.

Back in 1996 I had a website running on a shared unix of some sort. I could barely upload files in the correct binary/ascii mode, or copy and paste a “chmod” command.

My 1996 self would be the exact nightmare wordpress user today, although I guess I wanted to learn even though I didn’t know what I was doing, which is different to many comments I’ve seen on places like GitHub.

Glad to see that it was a positive view of the Linux bug reporters, rather than "Bah, I spend all my time fixing packaging issues from entitled Linux users who scream at me that the game doesn't work with their obscure, home-spun distro."

In a similar vein, I think the biggest value-add that Arch has over other distros is that it turns out having the filter of "can follow well written instructions through mildly tricky commands well enough to result in a bootable system" results in a community with a base level of competence, care, and patience that puts it at least two standard deviations above the other distros and at least four (I know how small the percentile is now) above just the general wash of garbage that you get when you Google for Windows issues.

It creates similar effects to the different credit card companies. Why would anyone accept Amex and its higher fees? Because they bring you higher value customers, sometimes dramatically higher value.

>In a similar vein, I think the biggest value-add that Arch has over other distros is that it turns out having the filter of "can follow well written instructions through mildly tricky commands

What is the value of "is competent enough to copy paste commands from a wiki?". Honestly I think the best bug reports might be because some Linux users probably understand C/C++ and can understand crash reports and error messages because they understand the system.

  > What is the value of "is competent enough to copy paste commands from a wiki?".
Because without that filter you are getting feedback from, at best, people who cannot even copy paste commands from a wiki.

You might be surprised how many people are out there which can't even read a wiki close enough to follow instructions in it.

Plus in my experience a lot of Arch users don't just "copy past instructions" they also somewhat understand why this instructions are needed, the Arch Wiki is grate as a resource for setting up things when you understand what you do, but it's often terrible when you just want a step by step guide.

Any way the main benefit of Arch is that it's close to stable upstream repose, instead of sometimes lacking not just month but even years behind wrt. the version of libraries they ship.

>Any way the main benefit of Arch is that it's close to stable upstream repose, instead of sometimes lacking not just month but even years behind wrt. the version of libraries they ship.

There is a downside that most Arch users omit intentionally, when you get latest GNOME/App with the cool new bug fixes and cool new features you also get the new not cool bugs and the new redesign/feature removal. This can cause the system not to boot if the kernel/graphics driver is updated in incompatible ways.

LTS distros with years behind features is in many casea a feature many appreciate. 2

I've been using Arch since ~8 years and at least for me updates breaking stuff is rare, and every time it happens there was a simple easy work around the problem (like downgrading for a day or two at which point the bug was fixed).

Through without question a major reason why the (few) problems I did ran into haven't been a problem was due to my understanding of Linux.

The is the misconception that Arch is bleeding edge, it's not. It's the latest stable releases of the software it composed of. And at least for my use cast the amount of headache it reduces by doing so far outweighs the amount of problems I ran into (which are in my experience few, and iff you have the necessary skills normally easy to work around).

I believer you, what I dislike is those people that push everyone into Arch and omit to add the things you added.

I bet that a vanilla LTS where you only update for security reason is more stable and risk free.

Tbh. I would never directly(1) recommend arch to anyone for a simple reason, it requires some linux/unix skills to be worth it.

If you have the skills I don't need to directly recommend it to you, you already know it.

If you don't know it you likely don't have the necessary skills.

Through by being opinionated about the choice of packages and way of setup and adding a QA Team, more CI and slightly delayed (non-security) updates (like 1 day or two) you probably could produce a grate experience even for casual users. Hm, but then as a company producing a custom Linux distro is rarely worth it and often special purpose enough to not care about the benefits this approach would bring.

(1): But still indirectly advertise it.

I don't think a rolling release will ever work for casual users, there are too many configurations of hardware and things you might not know people are using, some package update could affect someone printer and you cost this person a job opportunity.

You would at least have to block the AUR on such a distro, there are too many patched kernels/packages there and too many recommendations to install X from the AUR.

Then you have the GUI chnanges in apps, it is too frustrating to make the user learn weekly some new GUI workflow because app X decided to improve stuff.

Arch has a great use case for people that actually need latest stuff and for the people that just really want the latest stuff to satisfy their appetite of checking new features.

Arch probably has a lower share of GNOME users than your average distro.

I’ve been using the Arch Linux Archive as a way to stick with a known stable system for a few weeks until I have time to dedicate to a system upgrade and correcting any issues that arise.

How do you undo an app or subsystem after an update you don't like?

For example I upgrade my IDE but not in place, I keep previous version just in case the new one is buggy or they again moved shit around. For my main system I am on LTS and I upgrade if there is a need and not to get high on version numbers. For example I tested new versions of kernels and video drivers and end up on what feels right for me and stopped there. Sometimes the new video driver is more unstable then the older ones so I would never do a driver update without having a good reason and time to evaluate it.

Arch keeps a package cache of old packages, if you want to forever.

So installing old packages is trivial (iff you had them installed before).

>So installing old packages is trivial

I don believe this is true for ANY package(like install an old KDE4 app on your KDE5 system)

It depends how much older.

Like just a few days, weeks or continue to use a older no longer maintained version of a package because you don't like the new version.

I (now) guess you mean the later in which case, yes it's not trivial as it's not really supported by anyone anywhere. Neither the original software developers (which replaced it with a newer version), the package maintainers/Linux distro (which also moved on as the older version doesn't get updates), other packages interacting with ti (which expect a newer version), etc.

So while it might suck, I would recommend to simply not do so as it's not worth the risk and headaches it brings you. At least not as long as you don't find enough people to do a fork and maintain that fork.

Still undoing you last update is often as simple as just installing the package from the cache, and then you would need to pin it/make pacman ignore it (and then it probably will brake sooner or later depending how much it relies on specific system libraries in specific versions being available).

I disagree, when you release a new version of something distros like Debian or Ubuntu will backport the security fixes but not backport the new features and the new bugs.

As an example we are a group of people with some eye disabilities but not completely blind, we use an old application Jovie for KDE, this app was removed from latest KDE and replaced with something with lot less feature. We are compiling this old app from source and at least for this old LTS distros we can get both Qt4/KDE4 and Qt5/KDE5 libs installed. So imagine people with disabilities doing an update one day and poof your must have thing is gone and you can't use your system anymore and some person on the internet will tell you that you are doing it wrong and should not use anythng else then the latest version of stuff because the developers do not support old stuff.

Arch Wiki is so great enough that even users of distros other than Arch read it.

That is still a pathetic filter , "I know to read and I know to copy paste", I still believer that the good reports are not from the copy-paste ,I run Arch to be cool group but from technical people that run Linux(any distro).

> I know to read and I know to copy paste

You'd be surprised. I've seen many arrogant users try to install Arch and fail because they were literally incapable of reading and understanding the simple instructions written on the Arch Wiki. Even technical users of other distributions. Sometimes they use an installer and manage to get a working system with no effort... Only for it to stop working because they failed to read the news and some update required manual intervention. If I remember correctly, users are required to provide pacman output as a form of CAPTCHA in order to create an Arch Wiki account. They must prove they installed Arch in order to edit the wiki.

A certain degree of elitism is great, no matter the community. Arch Linux users are expected to be responsible for their own systems, it's expected that they will take care of it and maintain it. People who can't or won't do this are better off using something else.

The bar can be as low as it wants to if it effectively filters people out. It's like how fizzbuzz is a terrible stupid test until you realize the shocking number of people who claim to be developers but can't pass it.

We would need to do a rigorous test though to see if this low bar is significant, you would have 4 groups of people

- Windows tech people(c/c++ developers) - Linux tech people(c/c++ devs and sys admins) - Arch non tech people that copy pasted instructions - any other OS or distro non tech people

then see if the Arch copy paste-rs are closer to non tech people or closer to tech people. The assumption is that since they can read instructions would create better bug reports, but is also possible they would be over -confident elitist-ic dudes that use weird packages and broke the program with their copy pasting of commands and installing garbage from AUR;

Yet, something like 95% of all desktop users can't do that. You, presumably, disproportionally interact with tech people. Typical office worker might encounter problem that requires terminal once in a year at which point sysadmin is called. And it`s not only about reading and copy pasting it's about ability to devote pretty large amount of attention to a task.

For someone to find the wiki, read it and then attempt to solve their problem in a rudimentary fashion gets you towards the tail end of the bell curve. They might even follow up with you if you have questions.

Nah, is super easy to find the Arch wiki and the install instructions, Arch users will push you hard to it so is impossible not to find it, maybe you might find some honest person though to explain the downsides too.

I'm assuming you don't generally do front-end work, because many many users are complete [insert PC word for zero capacity for rational thought].

I do interact with customers, but I would rather get a bug report from a Kubuntu users that knows the difference between a compiler and a linker then from a 12 years old that managed to copy paste instructions into a command prompt.

And I would rather have either of those than a bug report from anyone I've had to teach to use a browser.

Being able to read and follow instructions is the main skill required to be a good bug reporter.

> What is the value of "is competent enough to copy paste commands from a wiki?"

Very high actually. You can tell them things like "please run in debug mode" or "please run with this command line flag" or even "please change this setting and retry". Even more basic, you can tell them to restart the game/program, or to reboot their computer, and then you can trust that they actually did it.

When dealing with a regular computer user, you can't assume any of these things.

From my real world experience on making desktop apps you want to get all the bug reports from all users, so you need to make it "1 click", This means add a menu /button to submit a bug report, then you popup a form where user fills stuff, you also send the log file where you collected info live OS version and other useful stuff plus the crash logs).

Same if you need to have the user run in debug mode you make it 1 click to enable/disable debug mode, usually though developers don't work directly with customers so then they don't put the effort to streamline the collection of quality bug reports.

> What is the value of "is competent enough to copy paste commands from a wiki?".

For a while, NixOS had examples throughout its manual, in the installation section, which did not together form a usable installation script, or even snippets within one. If you read the prose in the manual and used the examples as examples in the context of the prose, you'd be fine. But if you blindly copied and pasted all of the example snippets, the install would not complete.

You can watch someone ‘get filtered’ here: https://www.youtube.com/watch?v=QujRHErFG4w

The documentation has since been revised to make the examples copy-paste safe, which is a change I endorse because I see NixOS is a tool whose adoption I want to see grow and whose community I want to welcome and educate people rather than function as a super duper cool kids club whose that makes me feel special inside. But it does show how you could up the ante from the Arch case, if you really think exclusionary obscurantism is the way forward for projects you care about.

There is a even simpler reason:

A lot of people gaming of Linux are either software developers or system administrators.

People buying a "Linux gaming system" are the rare exception, instead it's often "buy a powerful computer for use case X, and hey why not go for a slightly better/tweaked spec and also also game on it".

> buy a powerful computer for use case X, and hey why not go for a slightly better/tweaked spec and also also game on it

I'm a programmer and this is exactly what I do.

> Why would anyone accept Amex and its higher fees? Because they bring you higher value customers, sometimes dramatically higher value.

Why are Amex customers higher value? Is it because they're typically business cards rather than personal?

> Why are Amex customers higher value? Is it because they're typically business cards rather than personal?

That's probably a part of it, but the real answer is because an Amex is a charge card, not a credit card, which means that whatever is spent on it MUST be paid off in full at the next billing cycle. The net effect of this, is that if your customers are shopping with an Amex, they're people who /have/ money, or in the case of business cards they are acting on behalf of an organization that has money. Without writing an expository essay, it seems to be simply true that those who are more affluent also have networks and can bring people by word of mouth if you meet their expectations.

Amex cards allow balances to be carried as long as a minimum is paid each month, just the same as with any other credit card. https://www.americanexpress.com/us/benefits/payment-flexibil...

The primary reason Amex customers are more valuable is because most Amex cards carry an annual fee, which only more affluent folks can afford.

Pay Over Time is a new "feature", which traditionally was not possible on an Amex card. Technically speaking, if we want to be pedantic, American Express is a banking and finance entity and offers many account types including typical credit cards (American Express Blue, any of the branded Amex like the Delta or Hilton Amex).

The traditional American Express card, though, is a charge card and not-withstanding the new Pay Over Time "feature", requires the entire balance to be paid in full every billing cycle. The entire reason they're popular with businesses is it provides an easy way to effectively do Net 30 payment terms without having to deal with POs and invoicing.

WSL is somewhat similar in that installing it was harder than installing most other things on Windows. They've made it easier in Windows 11, and will probably continue that direction. I wonder if MS knows where that leads for them.

You can literally just do `wsl.exe --install` on the newest builds of Windows 10 and it enables the hypervisor feature and downloads Ubuntu in one command. AFAIK it's the same in Windows 11 (I'm still on 10 and don't plan on upgrading for a few years at least).

Yes, as I mentioned, they made it easier only very recently. 09/27/2021, actually. And you still have to open cmd.exe or powershell with "run as admin". It is not as easy as installing, say, GitBash.

Also, some things are still under documented, or hard to do. Like that "wslg.exe" can be used to make Windows shortcuts to launch graphical programs. Or how to make various systemd things work (pid 1 is not systemd), and so on. Or how to add all the Windows fonts to the Linux distro. I'm saying they will likely improve all this over time.

TBH, as a non-ubuntu (gentoo) user, i'm pretty sure, that all the packaging isues (for non-gentoo packages) are something that I, personally have to deal with, and not bother the original developer with.

On the other hand, a Windows user is trained not to report bugs, especially in games. There's rarely a channel for reporting things & simple bugs go unfixed unless the community can work out some hack to do it for themselves. Games are abandonware monoliths on day one.

Secondly, if a program crashes in Windows and you try to force kill it, you get blocked waiting for the 'report' to be sent do Microsoft. Users are trained to click off this, so the OS will finally do what you asked it to.

Do you think this has something to do with how many Windows programs (not just games) handle errors and crashes? Even Windows still uses the "Error code: <bunch of meaningless numbers>" scheme. I can't even think of the last time I even attempted to get crash logs for a Windows program; however, on Linux I can usually find where logs got dumped and debug without as much of a headache.

... and if you search for that meaningless number and manage to find a proper support page, you are usually left with as much information as you started with. I don't understand why open source developers are so much better in this respect, but the diagnostics usually leave me feeling as though I could solve the problem even when I don't have the technical skills to do so. Being able to comb through the logs to pinpoint the problem is a major contributor in this respect.

(The only counter example I can think of at the moment is Samba, and guess what ... it interfaces with Microsoft products!)

I've noticed this too, compare any forum thread for a AAA Windows game and a Linux game, the Windows game thread is going to look like a bunch of chickens running with their heads cut off wondering why the 0x00423 error is causing the game to crash. Head over to a Linux thread and it's a usually a much more productive conversation because the errors are just clearer.

And when you can find a bug reporting channel the issue gets answered by an ‘MVP’, usually with some sort of inapplicable canned response, and then closed as ‘fixed’, without actually addressing the problem.

> On the other hand, a Windows user is trained not to report bugs

Decades of dealing with open source projects hasn't been much better in my experience. I don't usually bother to submit bug reports because my experience is that they'll be ignored for a few years and then closed unceremoniously.

That depends. I've dealt with open source products where I've submitted a bug report, submitted a bit of code, had the developer say that the patch wouldn't work for a good reason, then push out a proper fix nearly immediately. On the other hand, I have heard many stories that reflect yours. I suspect the size of the user base is a major contributor. Popular projects have many people submitting "bugs" that are little more than low priority (to the developer) feature requests.

For an audio app that I work on, it's similar, but different: around 1% of the users are on Linux, and around a third of the bugs / complaints come from Linux, but they're virtually all Linux specific.

We're having to seriously consider dropping Linux support because the support / maintenance load is so high. (This comes significantly from the libraries we're using having worse Linux support, and a lot from the Linux audio ecosystem in general not being very mature; and then a whole bunch of it is people wanting packages for dozens of different systems.)

I would think you can deal with platform-specific issues by saying "we support Ubuntu [or whatever you're willing to do]; anything else you're on your own" up front. But yeah, audio is a general weak spot.

I won't buy closed source programs with "Linux ports" anymore for this reason: it seems to always mean "some dev got it working on one Ubuntu version once". The last bug of this form I hit was a program trying to read a hardcoded path that apparently exists on Ubuntu and blissfully segfaulting if it didn't exist...

What you may be missing is that that comes from necessity -- again, for an app like ours, 1% of our users are on Linux. Producing packages for Ubuntu along takes about as much time as producing for Windows and macOS, where almost all of our users are.

Testing and packaging for, say, the top 5 distros would get the effort-per-user to an even crazier extreme -- we'd literally be at the point that supporting a user on Linux takes 200 times more effort than supporting a user on Windows. You either need a very large number of users, or a very large value per use for that to make sense.

For non-OSS projects, most of the time the smart decision is just not to support Linux. I've worked at other pro-audio companies, and they had Linux versions internally that they didn't release because of the support costs.

have you thought about offloading that work to the distributions themselves? I'd be curious to see what say ubuntu would charge for them maintaining your packages for you. you just need to provide a tarball to them.

I assume that doesn't work for closed-source software.

I mean, "Linux" isn't enough to target, and in all fairness, if they say it's for Ubuntu then that's what they support. One would hope that they do say they're targeting Ubuntu, of course.

We do that. In bold. Still doesn't stop people from writing to us at least once a week asking for something else or reporting that the Ubuntu package doesn't work on their distro.

That is unfortunate; it sucks that the cost of WONTFIXing a ticket isn't zero:\

With Pipewire and containerized distribution (eg. AppImage or god forbid, Flatpak), distributing Linux software can really be as complicated as you want it to be. Granted, audio software is still a bit confusing compared to Windows or MacOS, but with the advent of PipeWire I think most people are ready to put those days behind them.

Another alternative is to drop Linux support and let your users make a Wine installer if they really care about your software. Most VST plugins and audio software works boringly well through Wine, even scaling up to be a viable route for supporting games.

Yes, it is a VST plugin (and standalone), and amusingly probably works better in Wine than Native, aside from the fonts being craptastic. Containerized distribution doesn't really work for plug-ins.

This is the first time I'm reading of Pipewire, and it sounds promising, but will need to have host support before it becomes a reasonable VST / LADSPA replacement. (I didn't see if that's among its goals, but such would seem reasonable.)

> Yes, it is a VST plugin

Shouldn't that be the easiest way to distribute audio software for linux? It's just a static shared library and maybe some data.

> This is the first time I'm reading of Pipewire, and it sounds promising, but will need to have host support before it becomes a reasonable VST / LADSPA replacement.

I'm pretty sure they didn't mention pipewire as a replacement for vst or lv2 (or ladspa lol). It's benefit would be for your standalone since it supports alsa, jack and pulseaudio clients and can get decent latency.

> It's just a static shared library [...]

No. First, "static shared library" is self-contradicting: libraries are either shared or static, not both.

The longer answer is that plugins have to be compiled with the `-fPIC` flag to be loadable. The `-fPIC` flag basically says that it has relocatable symbols. You can't link against system-provided static libraries because they're never (by design) compiled with `-fPIC`. It might be possible for us to configure our build system to recompile every library we depend on as static and relocatable, but that'd be quite a bit of effort to set up.

As such, we have to link against shared libraries, and you can have conflicts with the host if it is also pulling in those shared libraries. Ardour, on Linux, specifically, in their own packages, pull in a bunch of non-system shared libraries that conflict with the system ones, which breaks our plugin. There are other shenanigans with other hosts.

> It might be possible for us to configure our build system to recompile every library we depend on as static and relocatable, but that'd be quite a bit of effort to set up.

Sounds like you already know the solution to your problems. Link all your dependencies into your plugin, hide all symbols except the plugin interface and base system ones and you can make your plugin distro-agnostic. If you depend on external shared libraries then yes, your interface to the system becomes a lot more complex and thereby brittle.

There are 53 shared libraries that our small app depends on (this is not strange on Linux). Getting them set up to all be built independently by our build system would be a large task, and a large thing to maintain. Technically it's probably the best way to go, but it's probably a week solid of work, and probably not worth it to support our small number of Linux users.

> but with the advent of PipeWire I think most people are ready to put those days behind them

Maybe. On the otherhand I heard similar things about Pulse and Wayland and a decade later they're still problematic.

Linux audio support is generally a pain. Have you considered officially supporting one audio library and just saying YMMV if you use something else? For example, only support ALSA.

Packages aren't supposed to be submitted and maintained by the developers themselves. The distribution maintainers and community is supposed to do it.

That only works for open source packages.

Ah, I know that one!

Easy! Just make it open source and get rid of your livelihood. Make it open core so people will hate you just as much (bonus points if your software can be ripped off by a hyperscaler) or go Red Hat and be 100% open source and basically turn into a consultancy which is not what a lot of people want to do with their lives.

I'm just being sarcastic, in case anyone was wondering.

I don't know about you - you might be insane for all I know - but I'm MUCH more likely to report a bug if I think there's any chance of it being fixed

Which is why I've stopped reporting LibreOffice bugs. I have had literally hundreds of bugs filed, with test files and reproducibility instructions. Then they changed bug trackers - twice - and all that is lost.

With KDE and Mozilla, I'm still seeing bugs fixed a decade after I stopped reporting them. So I might be inclined to file egregious bugs to those two projects, if I feel inclined at the moment to wait a decade for a resolution.

I saw recently twitter complaint about emacs bug filed almost 15 years ago finally getting a wontfix responce(bug was about undo history loss or something). And then they write articles about their new and shiny bug closing process[1]. [1] https://lars.ingebrigtsen.no/2021/08/14/10x10/


Copy paste the link. Or:

> I'm so totally impressed at this Way New Development Paradigm. Let's call it the "Cascade of Attention-Deficit Teenagers" model, or "CADT" for short.

> It hardly seems worth even having a bug system if the frequency of from-scratch rewrites always outstrips the pace of bug fixing. Why not be honest and resign yourself to the fact that version 0.8 is followed by version 0.8, which is then followed by version 0.8?

> But that's what happens when there is no incentive for people to do the parts of programming that aren't fun. Fixing bugs isn't fun; going through the bug list isn't fun; but rewriting everything from scratch is fun (because "this time it will be done right", ha ha) and so that's what happens, over and over again.

In your case, replace version "0.8" by "new bug tracker" or "new bug handling process".

> Then they changed bug trackers - twice - and all that is lost.

When precisely did this happen?

Within two or three years of forking away from Open Office. I was a Go Office user, when they forked to LibreOffice I verified and copied all my Open Office bugs to the brand new LibreOffice tracker. About a year later they switched trackers, then maybe two years later I think they switched again. So maybe 2012 or so.

Knowing a bug will be investigated would be enough, never mind fixed. Sadly I would assert that the vast majority of open source projects do not bother to respond to reports or even patches. The result of being ignored enough times is that I have become much less motivated to report bugs on projects that might actually care, unless there is clear and present evidence (openly available) that their issue tracker is sufficiently active. If I have to wait more than a week for a reply, you have lost me for good. If I can’t see evidence of the process working, I will not waste my time.

I had much better results with one-man projects, than with projects which have a small army of volunteers and/or of paid employees. In the later case, it feels exactly the same as when I try to get in touch with the various administrations in my area, meaning that I can achieve a similar result by printing my missive, setting it on fire and dancing around it naked.

I've reported bugs to FF and Chrome several times because they were in things I cared about. Developers tend to care more, know that there will be a place to report bugs, know how to do it and so forth. Obviously Linux gets more bug reports.

(I use mac, the m1 is just great, but I would never go through the effort of reporting a bug to mac [unless serious security hole] because I don't care about them, and because I have the impression from having tried at one time some years ago that mac bug reporting sucks.)


"Do you know how many of these 400 bug reports were actually platform-specific? 3. Literally only 3 things were problems that came out just on Linux. The rest of them were affecting everyone - the thing is, the Linux community is exceptionally well trained in reporting bugs."

The whole thread was interesting but this comment had me cracking up:

>Same experience here. Linux folks not only are our most frequent bug reporters, but they're also the best at documenting their issues and sometimes just... give us code for UI improvements?

Another example of "they'll never tell you": https://pointersgonewild.com/2019/11/02/they-might-never-tel...

One thought: if merely 'being a Linux user' can enrich bug report rates by a factor of 6.5x, how many bugs are still not being reported...?

It's worth noting that this game uses Godot engine, which is open source and built with Linux in mind, and most platform compatibility complexity was shifted to engine. Experience is quite different for big game developers that have their own engines and can't offload complexity to another layer of abstraction, supporting a lot of hardware/software combinations on Linux for them is quite hard. I presume that's why there are games that didn't get PC Linux releases while technically having Linux port(Cyberpunk 2077 would be prime example).

> Experience is quite different for big game developers that have their own engines and can't offload complexity to another layer of abstraction, supporting a lot of hardware/software combinations on Linux for them is quite hard.

Oh but they can offload that complexity to e.g. SDL [0]

As for why games like Cyberpunk 2077 are not on Linux, we can only speculate. Remember however that for a profit-focused publisher/developer the Linux port not only has to be profitable to make sense, it has to be more profitable than other things they could spend their time on.

[0] http://libsdl.org/

We know for a fact that Stadia runs on Linux. We have leaked documents from CDPR hack earlier this year with details of Google's deal with CDPR. From these documents we know that Vulkan renderer exists and even how much Google paid for that.

For most AAA titles the reason is simply 3rd-party DRM and/or Anti-Cheat not working on Linux.

I'm talking about native Linux ports, not Proton compatibility. There is Linux port of Cyberpunk 2077 and we know it because there is Stadia version of game. And Linux versions of Hitman 3, Control, more than a dozen of Ubisoft games(they basically have Linux versions of all their major engines), two latest Doom games. And Denuvo DRM(which is basically only game in town right now) doesn't affect compatibility with Linux that much, DRMed games like RDR2 work just fine. Anticheats are bigger problem, but they are being dealt with. Biggest anticheat providers are now Proton compatible.

I think Windows users are used to live with bugs and work around them. Linux users prefer to adapt a system to their needs, Windows users adapt to the system.

Or maybe it's much simpler than that: Linux attracts people who are more tech savvy and like to get involved. Windows users are often incapable of understanding even basic OS concepts.

There's often no need to go any deeper than that. Same with Mac users: there's a considerable demographic to whom an OS is simply just an app launcher (which IMHO is fine), maybe even just the browser.

There's a reason ChromeOS has been so successful.

Negative feedback is insanely valuable. I write about that here:[0].

Here's what I wrote:

> It can be mighty unpleasant to read negative reviews and comments about our work, but it’s worth it to do so, as long as we’re doing it to improve said work. I often say that positive affirmations feel good, but negative feedback is required to improve a product. Negative feedback, even if it’s uncouth diatribes from unpleasant people, is far more valuable than the most glowing praise (unless said “glowing praise” comes from Consumer Reports).

Simply put, negative feedback is a goldmine. DON’T WASTE IT. Steel yourself. Take a belt of absinthe, if you need, and open the “Comments” section. Read everything. It sucks, doesn’t it? Is that even physically possible? Don’t you wish you were in good enough shape to do it? Do you remember your mother ever saying anything that could indicate this were true? Wait. What was that they said about the communication error report alert?

[0] https://littlegreenviper.com/miscellany/the-road-most-travel...

There's another piece of advice I've heard along these lines.

> Listen to the users complaining. Users are very good at identifying problems that you missed or can't see because you're too close to the project. But be careful listening to users about solutions to the problems. They can't see the whole picture.

I think this is good advice. You should want to make the best product you can. Some users have good ideas and can see things you can't. But it is hard to get the signal out of the noise and this can be very frustrating.

Agreed. Like Henry Ford is quoted as saying:

    "If I had asked people what they wanted, they would have told me 'A faster horse!'."

Yeah and it takes skill to distill what they actually want out of that comment. People don't want a faster horse, they want a faster method of transportation. This is easy to miss, especially the more technical jargon people use when explaining what they want.

>even if it’s uncouth diatribes from unpleasant people

HUGE Caveat: negative feedback is not always constructive feedback. I imagine the kind of feedback devs fear isn't

>There's this weird clipping problem with the character on this level that can rarely cause you to get stuck in the floor


>OMG why can't I even move in this game, did the dev even playtest this shit?

Even if they are technically saying the same thing, the latter makes me want to discord the "feedback" as some dramatic user. Even if I ignore the language, there's not much action I can take to address the issue, whereas non-diatribe negative feedback user gave me an exact level at the bare minimum to start off at.

And it's the internet. It may not be the most common feedback, but in this day of "likes", people sure do like to hoist the latter kind of comment up for devs to see first.

Wine's crash handler provides a lot more details and diagnostic information if you know what you're looking at. "Oh RAX was 0 and de-referenced. That looks like a trivial memory error in SomeComponent.dll+0x302223."

Other platforms are infantilized: "Oopsey doodle that crashed :(. Look for problem online? Oh no solution found. Sorry about that. Got it?"

That's not platform specific at all. If your program spits out proper crash logs (which most games do), it's a matter of locating and examining the log file.

The "if you know what you're looking at"-part is pretty much nonsense as you're basically saying "if you are an experienced programmer with intimate knowledge of both the system and the hardware architecture".

This is not something a regular user should be expected to know. A good program should simply provide a proper feedback-channel and collect and send the appropriate information if the user choses to do so.

>The "if you know what you're looking at"-part is pretty much nonsense as you're basically saying "if you are an experienced programmer with intimate knowledge of both the system and the hardware architecture".

Same thing.

> This is not something a regular user should be expected to know. A good program should simply provide a proper feedback-channel and collect and send the appropriate information if the user choses to do so.

Then why do all these bugs fall through the cracks on other platforms?

> Then why do all these bugs fall through the cracks on other platforms?

Two main reasons:

1) Other platforms are primarily used by non-technical individuals who simply don't care about the OS and who more often than not are unable to tell the difference between bad software ergonomics, user error, and software bugs in the first place.

2) The Linux platform is built around the concept of OSS, where user participation and direct feedback are an essential part of the ecosystem. Most OSS has bug tracking and Wikis that describe how to report a bug and quick release cycles (e.g. fast feedback from the user's perspective). Linux users make a conscious decision to use Linux and some level of involvement in the OSS community is often part of that decision.

It's not just the community mentality of the users. Just by raw numbers there are going to be more "technical users" running on other platforms because the number of Linux users is so vanishingly small.

Most of the people trying to game on Linux are "non-technical" with regards to being able to debug random errors in some random binary. You can only really assume they have enough skill to put an Ubuntu image on to a flash drive, boot it, and follow a few prompts.

The software tools are just better for locating, reporting, and communicating these errors are just better on Linux. Not treating your users like morons and giving users actual feedback about why something crashed instead of writing out crash reports to disk that never get sent anywhere is apparently a more effective way of tackling software defects.

> Just by raw numbers there are going to be more "technical users" running on other platforms

Observation bias. The vast majority of desktop users in 2021 are found in a professional setting (offices, etc.). Those users don't use computers at home, whereas the vast majority of Linux users do.

> You can only really assume they have enough skill to put an Ubuntu image on to a flash drive, boot it, and follow a few prompts.

I'd like to see some statistics on that. Gaming has shifted to consoles and gaming on PC has been the domain of MS Windows for decades. Survey¹ says that >96% of users are on Windows and the few Linux gamers (~1%) are just as likely to run Arch as Ubuntu (0.26% vs 0.29%).

The choice of using Arch over Ubuntu is a strong indicator in- and of itself that the user made an informed decision (e.g. is tech savvy as opposed to "insert media and click 'next' a few times).

> Not treating your users like morons and giving users actual feedback about why something crashed [...]

As I mentioned that's not up to the OS but to the software in question. Also you're contradicting yourself here: on the one hand you argue that users aren't morons and have technical knowledge, while on the other hand you assume they're basically trained monkeys that can flash an image and follow prompts.

Which is it?

1. https://store.steampowered.com/hwsurvey

> the thing is, the Linux community is exceptionally well trained in reporting bugs

I don't have any stats to back up my claim but I believe a good chunk of those Linux users are tech folks - well, we submit bugs all day long at work, of course we are trained to do so!

Interview with the author, mostly touching the Linux development of the game: https://www.gamingonlinux.com/2021/05/an-interview-with-kode...

Kudos to them. I remember reading a report from a game dev a few years ago that said basically the same thing and came to the conclusion, "... and that's why we're ending our Linux support," which utterly baffled me.

Appreciate the old reddit url, so much more readable imo.

I am more impressed by 5.8% of sales being from Linux, it was less than a percent in my experience

I'm reminded of Valve talking about supporting Linux for their half-life2 engine. Their experience was that they were able to find some fabulous engine speed up improvements that they then backported to the windows version.

The common thread with supporting Linux seems to be that it's not necessarily worth it from a revenue/market size point of view, but there are all kinds of intangibles that you only get by doing cross platform development.

At the coprorate level, intangibles don't help you move up the chain, unforuntately. I imaigne there are many well-intentioned devs that lack the choice to keep working on these things, or have to make other choices in the grand scheme of things.

Looking at the numbers, and the fact that most of the bugs affect all OS's playing the games, the title should probably something more like "Linux users file more reliable bug reports"

That's basically what's being said here, but you just know flag wavers are going to take the title to mean that Linux is more buggy.

Most Windows users don't even know what a bug report is.

> Most Windows users don't even know what a bug report is.

... or a telemetry.

To be fair, given a well-known application (paid especially), users may be more inclined to try to complain about some annoying/unexpected behavior. Just too often it goes onto a forum, so noone knows if this results in a bugreport somewhere.

Yep, sampling bias.

It's a sci-fi 2D space mining simulator. It's already going to attract the type of user who files detailed and specific bug reports. If the author made a Call of Duty clone for Linux, would they get as many great bug reports? I don't think so.

When I submitted a bug to Linux distro Pop OS, it was fixed and deployed within days to all users.

Can you imagine anything like that with Windows or Mac OS? We have reports how Apple silences zero day researchers instead.

Microsoft/Appple don't have nightlies. For systems as complex as that, it can take days just to land it in some dev feature branch, let alone get into into a production branch scheduled by product managers to deploy whenever.

A good trick for removing unintelligent content from google search results on any general computing issue is to prefix the search terms with "linux". And I'm a Mac user :)

As an Ubuntu and RHEL user, I tend to prefix my searches with "arch", for the same reason (although if you have a subscription, and your issue is with a RHEL system, RHEL is worth searching first, as they often have a bug report with resolution for your exact issue).

I bought this game because I appreciate good devs and want to encourage linux support in gaming.

This game is phenomenal, I can't stop playing. Intuitive, good UI, tons of fun. Thanks!

This is great. By the way, one of the things that has often discouraged me from filing bugs is not really knowing how to do so. Sometimes, looking at issues on github, I find people provide super structured and clean descriptions of bugs. Then there are the other kind, that the author describes: “it crashes.”

Is there a standard way of describing bugs? What resources/standards have people found useful to refer to when reporting one?

There are a bunch of open source projects on github with issue templates. Have a look at a couple of those and pick one that suits you.

I wonder how many people will read just the headline, think, "I knew it!" and move on without actually reading this insightful article.

It just means that Linux users are more technically savvy I would say.

If you've installed and use Linux then 90% your interests are in IT / software / hardware design.

So you know concept of a bug. And it is a good chance you've produced and fixed those by your own too.

The best person to report a bug is a programmer really.

the titled should be updated to say "the Linux community" at the end, as on the original page

The original title was too long

The Mac and Windows using developers at work never know how to file a bug report. I have to play 20 Questions to get them to give me the information I need to isolate the problem and fix it.

The Linux-using developers fix the problem for me.

This article does and doesn't mirror my experience.

When the pandemic hit, I helped my now 15 year-old godson build a video game add-on as a way to stay close since we couldn't travel. For the last two years we've had a blast working on this together. The add-on has grown in popularity, and has a few dozen thousand users.

Where I agree with the article, many of the complaints do come from Linux users.

Almost without exception though, the bugs they reported were Linux-specific, or related to old-versions of the add-on that we no longer supported.

A lot of times the Linux users were a release version or more behind because the emulation software they were running hadn't been updated to run the latest version of the game. They'd update the add-on, before updating the game, and then be upset when it didn't work on the slightly less than new version. Building a work around for this was a valuable lesson for my godson, in that we had to plan for the imperfect-path and make sure the add-on had a lot of edge cases baked in.

Some other things I've noticed...

Linux users do tend to report more bugs, but mostly because Linux users tend to be quick to anger. Customer support for Linux users... it was never pleasant. Someone would say something like, "Don't you test your shit?!" and like keep in mind this is a free add-on for a video game... and they'd be irate that we hadn't tested it on every system imaginable before releasing it.

Linux users don't accept reality. Often they're aren't running the "real" instance of the game, they are running some hacked together thing running on some emulator... it's not going to be perfect. Or they were running the game on unsupported hardware that they rigged a work around for. A lot of times they'd swear the bug only came about due to our add-on, but countless hours wasted trying to appease folks like this and 99% of the time it was a bug on their end. We'd probably have had 20% more "real" dev time if we didn't ever engage with those people in the first place.

Linux users tend to be more European, and there was certainly a language barrier at times. Made supporting the add-on for them worse because bug reports would come in bad-English, or German run through Google Translate to English... or just in German. There seemed to be a large number of mid-30s Germans playing a game that I'd say was aimed more for teenagers... Without sugar coating it, bug reports form those people always felt like, "Doesn't this loser have anything better to do?"

Linux users do tend to be more technical, and often they have suggestions for improvement in mind. But rarely do any of them take the time to crate a PR. It was infuriating. We'd have one guy complain every week on an open ticket... we'd say, "Hey mate, we're backed up... we'll get to this when there's time. Feel free to submit a PR!" and instead he'd just paste 80% the code he wanted updated in the GitHub ticket, and then nag us every week to fix the bug. It would have been an easy enough fix, except it required us to install Linux to verify the fix. Oof.

Broad ways I wish every community was better?

Don't be a dick. So many of the complaints came from these man-children and every other word was a profanity. Especially on open-source projects, just understand how nice you are directly correlates to how fast your issue gets worked on.

If you can fix something, don't bitch about it... just push a PR.

If you push a PR, test it first. Don't assume it's on someone else to do more testing than you'd care to do. "Oh man, that would require me to do X, Y, Z... that would take hours!" Yeah, it would require anyone testing to do X, Y, Z... if you want it fixed, the fix includes testing on X, Y, Z... not just writing the code. Way more testers are needed for all open source projects, but most end users -- even the ones who can't code -- don't really want to help the project by volunteering testing time. If there's a project you like, and you can't code, you should still reach out to the creator and say, "Hey, how can I help test?"

That sounds a lot like badly managed support TBH. Complaint about having to install Linux to check fix indicates that you weren't even trying to run build on Linux before releasing it, I'm not sure why were you supporting Linux at all with that kind of attitude. And I presume that vanilla game doesn't have common way to install mods and Linux version is non-existent.

>I'm not sure why were you supporting Linux at all with that kind of attitude.

it's a free add-on made alongside a 15 year old for bonding purposes. For all we know, it wasn't. And I don't think they have a duty to provide any labor that goes beyond their bonding to ensure this, let alone be yelled at for it.

Correct, we never set out to run a software company... just build something my godson found fun. Figured since he was interested, he'd be motivated and he'd learn a lot along the way. A lot of what he's learnt has been social skills... not programming skills. But there was no intended support for Linux, he's 15... like he finds it fun to build things, and I'm doing the best I can to limit the tedious "edge cases" so he stays motivated to be a dev later in life.

>Someone would say something like, "Don't you test your shit?!" and like keep in mind this is a free add-on for a video game.

that's always the worst. If they paid some money for this, then I can understand some anger in their investment. Here, it's literally goodwill providing this tool the complainer chose not to make themselves and they instead throw a fit like the world owes them this free labor.

I don't like using this retort, but this is exactly what the "why don't you make your own steak" sorts of replies are for.

I love that this links to the legacy reddit ui.

Just for kidding but... do you think windows users are able to make/know what is it a bug report?

Hrmmm. This appears to be a trend. Moving averages suggest in some short time 13% of those sales may account for 50% of the bugs.

They should drop Linux support and save themselves the hassle.

Despite being 13% of the userbase

TL;DR: reason number one million why you should be using Free software whenever you can: you don't feel helpless about bugs.


There is for all practical purposes a singluar linux.

Why? Because you can bundle your own "userspace" to support your game, and that's what steam does for you with it's runtime: https://github.com/ValveSoftware/steam-runtime

It has to pick up a few things from the surrounding environment, but steam entirely standardizes the vast majority of it, and the rest of it is really really similar between every linux-running OS.

TBH this runtime can be replaced, and Arch ships a package which makes it very easy to do


Sometimes it works better than what Valve ships. It saved me a couple of times.

The value of the steam runtime is not in how well it runs for users, but that it provides a singular target for game developers. I'm not sure if you witnessed any discussions around why most game developers are not willing to port their games to linux. From what I've seen the main complaint is that linux is too fragmented and nobody wants to package a binary for X versions of glibc, Y versions of sound libraries, and Z versions of graphic APIs. Having the steam runtime to target solves all of these issues, even if it doesn't represent the best that a user can run on their individual configuration.

> nobody wants to package a binary for X versions of glibc

You only need to build against the oldest glibc version you want to support - and the runtime does nothing to change that since you can't bunle your own libc (if you want any dynamic linking at all, which you need fro e.g. OpenGL/Vulkan).

> Y versions of sound libraries

The steam runtime makes sure that there is a libasount/libpulse* present so that your program can start if it links against those, but it does not provide anything for compatibility between them - that can be achieved by e.g. SDL or OpenAL Soft (which load them dynamically which means there is no problem if the libraries are missing and you won't even need the runtime). It's also ok to only support one audio system then then have users figure out the compatibility on their system (e.g. PipeWire supports the ALSA and Pulse ABIs).

> Z versions of graphic APIs

Same, the runtime ensures that a vulkan loader is present but does not actually do anything for compatibility. Both OpenGL and Vulkan also provide backwards compatiblity so you again just need to decide on a minimum version.

What the steam runtime mainly does is provide a bunch of libraries that are present on Ubuntu that games might link against so that everyone has them. Nothing stopping the games from shipping those libraries themselves though.

Are you sure glibc versions (or libraries in general) are compatible between distributions? That's what I meant by "different versions", sorry for not making it clearer.

> Are you sure glibc versions are compatible between distributions

Yes, the only difference (between backported bugfixes) really is which glibc version is included.

This does not apply to other libc implementations like musl which only tend to be used in lightweight focused distros intended for embedded systems or containers like Alpine Linux. With these even the steam runtime won't help you since it (and Steam) also requires glibc and it's needed to use e.g. flatpak to provide a glibc userland - that's on the user/distro though: https://wiki.alpinelinux.org/wiki/Steam

> or libraries in general

Libraries generally fall into two categories: system libraries required to interface with hardware or other OS components and general user libraries. System libraries (e.g. libasound, libpulse*, libGL, libvulkan, libX11, libwayland) do provide a stable ABI with strong backwards compatibility and you can and should rely on the system verisions (again, decide on a minimum version, e.g. by compiling on Ubuntu LTS or something or use SDL to abstract this for your). Don't include copies of these unless you know what you are doing.

Other libraries (for example zlib, libpng, ...) do not generally provide any ABI guarantees (at least not long enough ones to matter for binary distribution) so you should ship your own copies or use the steam runtime to do that for your. If you can, statically link them into your binary and hide the symbols so that they don't clash with any system libraries that may be loaded - this provides the best future compatiblity with current and future distros.

There are some interface/abstraction libraries like SDL and OpenAl soft that both provide a stable ABI and themselves build on other stable ABIs. Here the best path is to distribute your own copies but do not statically link them.

A pain point is the C++ standard library libstdc++. Like glibc it currently provides a stable ABI with backwards compatibility. Unlike glibc it tends to be much closer linked to the compiler version so it will be harder to avoid requiring a newer version than what your oldest target distro comes with, unless you are fine with restricting yourself to an older compiler and C++ version. It might be tempting to include your own copy but this will cause problems because graphics Mesa (the open-source Vulkan and OpenGL implementation) also uses libstdc++ and requires a copy at least as new as the one it was built with - which any copy you bundle won't be for long. If you can statically link libstdc++ and libgcc (and all your C++ dependencies) and hide all C++ symbols then you are fine. Otherwise, rely on the steam runtime to get your a new enough version.*

Does whatever work on Android too? Because that's technically Linux, that's what I believe GP was referring to. I believe Android actually has a totally different graphics stack.

The point is that people who choose an OS other than Windows or Mac tend to choose something based on Linux, and are usually technically skilled and submit much higher quality bug reports, more often. Linux users just care more about the experience of using a computer, otherwise most wouldn't have jumped ship.

Also, technical skill aside, Linux users are used to dev <-> consumer relationship being more symmetrical and more of a "two way street". The few non-coder Linux users I know prefer it for this aspect --- similar to why people might shop at a local neighborhood co-op instead of Walmart/Target/Microsoft/Apple

talk about missing the forest for the trees! _goddamn_

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact