Hacker News new | past | comments | ask | show | jobs | submit login
Rockstar thanks GTA Online player who fixed load times, official update coming (pcgamer.com)
1032 points by mmhsieh 32 days ago | hide | past | favorite | 321 comments



The original fix’s author updated the post about this: https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times...

Rockstar awarded them 10k for it.


Recent and related:

How I cut GTA Online loading times by 70% - https://news.ycombinator.com/item?id=26296339 - Feb 2021 (697 comments)


It's funny because reducing load times by 70% is almost certainly worth way more than $10k if calculated.


I can see both sides of this.

On one side, yeah - the work he did will be worth a lot more than $10k to Rockstar. They've had lots of people complain about & stop playing GTA online because of the slow loading times. And this issue would probably never be found or fixed otherwise.

On the other hand, he did it for fun (without any expectation of reward). $10k isn't massive and we can quibble over the reward amount, but its delightful that companies are rewarding people at all for fixes like this. A few years ago he'd be lucky to get sent a GTA t-shirt. Its definitely the right direction we should be moving in - bug bounties are a thing in security, and we should be celebrating any movement toward rolling them out in other circumstances.


And also they have to balance out internal politics as well. Putting massive incentives on the discovery of bugs could really mess up the motivations of internal staff.


Indeed there were stories on that previous thread of people being banned for 'hacking' when they've pointed out flaws to game developers


> On the other hand, he did it for fun

This might be an argument for him to console himself (or just keep smiling) in the event that he received nothing. It isn't a valid consideration for Rockstar, which should take heed of the calculus you stated with your first hand only, because for the beneficiary of a windfall, karma awaits.


If some engineer at Rockstar had done this work, they would have gotten their regular salary and probably not much else, which probably would have worked out to less than $10k for the amount of time spent on this issue.


And yet, in practice, none of the engineers at Rockstar has done this work in 7 years and a half. I do not believe that it is fair to value *a posteriori* a one-million-dollar fix with "if" sentences.

Let us spin it differently: contact Rockstar before publishing the details of the fix, and offer to help them on this issue. How much do you believe that Rockstar would value the fix *a priori*?

The truth is this kind of fix is worth a lot, but it is always tempting for the company to be penny-pinching (and come up with justifications for being so) after the fact.


Engineers don't get to work on whatever they want (usually). The described problem doesn't stop people buying shark cards, the fix is not new content and players leaving the game because of it don't affect the bottom line significantly. This could be why the priority was low and why you should prioritise tech debt


> none of the engineers at Rockstar has done this work in 7 years and a half.

Its probably because some project manager would have told the engineer who wanted to fix this that its not important.


A Rockstar employee would have had access to the source code probably. Running a profiler would have taken them significantly less time than the author.


Further, a Rockstar employee would just isolate the parser, feed json files, graph processing times and look in awe to the resulting plot.


Yeah but to be fair, that's what he's actually supposed to be doing.


As a company that's technically true. As an individual engineer they would probably have been sneaking time to work on it and have to be a pet project of someone senior enough to get it merged and released.


not really, there's a lot of great work i could have been doing on my past projects that wasn't a management priority


The author also publicly mocked them and forced their hand in changing 7 years old code. They didn't even mention this $10k reward in their PR issue.


It's a really nice and wholesome use of HackerOne. Nice that they acknowledged and awarded as opposed to some companies that try their best to push a security disclosure out of scope and not pay out a bounty [0] [1]

[0]: https://news.ycombinator.com/item?id=9278832

[1]: https://news.ycombinator.com/item?id=16000550


That's fantastic. Hopefully a few coffees as well, from that amazing write up. :)


Hot coffee is banned at Rockstar


That's awesome! I didn't think Rockstar would act on it.


Note: he lists his address as living in Latvia.

Still I’d give him more for the significance and PR. But these things tend to be fixed rather than subjective for a reason.

https://github.com/tostercx


>Rockstar awarded them 10k for it

that's insulting tbh


How is that insulting? They are under no obligation to give them anything at all.


I honestly expected Rockstar to do nothing at all about the situation. I’m genuinely surprised they’d implement his findings and give him some money as well.


At this point, I wouldn't have been surprised if a video game company sued someone for DMCA violations for something like this. Rockstar and Take-Two in particular are no strangers to handing out DMCA take downs, cease and desist letters, or lawsuits. I'm glad they're being cool about this.


The added value of the fix is in the upper 8 figures due to enormously reducing friction for players. If a consultancy was hired to fix the performance problem the cost would run over $1M. $10k is funny money to Rockstar.

In normal people's terms, if somebody did hours of labor that gained you >$10k, would you give them $10? That would be insulting, yet that's what Rockstar did.


Maybe you were not aware that GTA has taken in a billion dollars? Or don't know what that number means?

Most people don't: <https://mkorostoff.github.io/1-pixel-wealth/>


Maybe you don't realize that taking in $1B means they were doing just fine without this bug fix. Faster load times amount to "Oh isn't that's nice".

After reading his account of finding this, how long do you think it took him? 10 hours? 20? 40? That's $250 to $1000/hour for something he did out of curiosity.


250$ a DAY would be a nice salary here in Latvia.

I don't think anyone here can get anything close to those numbers you specify... but you could try to hire him and make a nice offer :)


Oh my good lord that was a serious eye-opener. I’m not American but can’t imagine it’s drastically different in the UK. I’ve lost a huge amount of money in the last year and also my car decided to die on me, I’m currently scraping together 3 grand or so to get a new one at the moment. So many people I know are struggling at the moment too. Meanwhile Jeff Bezos is probably sleeping like a baby.


> Rockstar awarded them 10k for it.

The original post, analysis and the actual fix of this bug is beyond impressive. Strangely, Rockstar awarding $10k for that is like someone searching for lost treasure, identifies historical references which leads to finding a ton of it and somehow leaves with 1/8th of it, even though they did 98% of all the work.

For that detailed effort, analysis and with a fix, I'd say that is worth around $80k. At least Rockstar was generous enough to award them in the first place; so I'll give them that.


> For that detailed effort, analysis and with a fix, I'd say that is worth around $80k.

No good deed goes unpunished. Rockstar went out of their way to thank this person and send them a check equivalent to several weeks of median developer salary. Can we not shame them for this goodwill gesture?

$80K would be on the order of half of the annual salary of most developers outside of SFBA.


>>$80K would be on the order of half of the annual salary of most developers outside of SFBA.

Lol, I'm a lead C++ programmer in the UK and I don't make that much. It's American salaries as a whole that are completely stratospheric.


I'm sorry to hear that you're not better compensated. I also emplore you to reframe your perspective. Consider the possibility that pay in many places outside tech hubs in the US is outrageously low. There's a race to the bottom wrt exploitative commodification of developer talent as the 3rd world comes online, and I find it abhorrent. This "SFBA / FAANG companies are the sole exception" narrative is incorrect, and harmful to you and to me and to everyone in our field. If you're living in a 1st-world economy, have solid technical chops, can communicate reasonably well in written and spoken English, working for a for-profit business, and have a few years of experience, and you're not making over $150k/yr, you're almost certainly underpaid.

Know your worth! Demand more! Capture the value you produce!

- Chris in Duxbury, MA

PS For context: I've been building web-related things for a living since 1998, and have always been paid fairly for it. Because I insist.

PPS Ofc YMMV... circumstances, personality, ambition, etc etc. But there is so much more opportunity to create and capture real wealth than you might imagine.


Then I need to provide more context

"and you're not making over $150k/yr, you're almost certainly underpaid."

Those salaries don't exist in the UK, no matter your position or the company you work for. I know senior programmers working for Faang companies in London and they don't make this much. Like, in general you won't see a position offering above £100k/year. I know someone with 25+ years experience in the industry and they are on an (incredible for the UK) £90k/year.

The only way to reach the equivalent of $150k/year in the UK is to work as a contractor and be good enough to justify billing that much. As a normal employee you won't see that.

My point is - it's impossible to judge everyone in the world by the American pay standard. Would I make more money in the US? Maybe. Would my standard of living be better? Also maybe, but I lean towards no with this one. But it's almost certainly not true that by making less than $150k/year you're underpaid in a place like the UK.


Where did you get that idea?

I worked for a US/UK company with its engineering HQ in London until recently and was earning drastically more than that. Now, I was an early stage employee and doing unique work for the firm. That position wasn't advertised - I created it. But some of the more senior engineers on my team (in the UK) were earning about $150k/yr GBP equivalent, a bit more in some cases.

Are salaries in the UK lower than in the USA, yes, absolutely. They are not that much lower than non-SF jobs in the USA.


Putting aside the question of whether there's an objective standard for whether someone is underpaid - it is not true that nobody in the UK makes $150k/yr as an IC SWE without being a contractor. Filter on the UK: https://www.levels.fyi/comp.html?track=Software%20Engineer

You'll see that mid-level (L4/E4) engineers at Google & Facebook easily break 150k; senior engineers (L5) start at ~225k and hit 300k. That's about 30-40% less than what Google & Facebook pay engineers in SV.

I also saw multiple other companies where engineers with less than a decade of experience were approaching or breaking 150k - Transferwise, Improbable, Palantir, Morgan Stanley, Booking.com, Cloudflare, Bloomberg, Yelp, Babylon Health, Goldman Sachs, Stripe, etc.


My entire network makes more than that in Dev and Arch roles in that part of the world. None of them have to live in London for them either.

Either you're underpaid, or C++ has been commoditized to the same point as PHP. If you value money over language, I'd switch. But given quite a few in my network are Java developers, I doubt language is the issue here.


Well then I have no idea how you and your mates do it. I know people who are in very senior positions at FAANG companies as well as couple big financial firms in the UK and not a single one of them makes over £100k/year. Two of them with PhDs in big data analytics and working in the industry. My own salary might be excused by being in the North East, but they work all over the place, in and out of London.


It sounds like you've got an academia background so if they're research or R&D positions, that might be why. Because they're competing in an already saturated field. Or there aren't many jobs but lots of graduates keen to work in your subfield who will do it for lower pay because they enjoy the work.

If you moved into in-demand fields, service delivery or consulting, you'd see 100k GBP as normal for seniors. The fields only require mastering the tools or domain and the ability to work with business stakeholders. That is from my experience, the majority of the software world.


Again, senior technical contributors at UK FAANG offices do not make under 100k/yr in total compensation; see: https://www.levels.fyi/comp.html?track=Software%20Engineer

I don't know what your friends told you, but either you misunderstood them, are talking solely about base (and see my other comment for why >100k base also exists in those roles), or were given inaccurate information. 100k GBP would be 45-60% of the credible range for senior engineer compensation at Google/Facebook in the UK (and maybe 55-70% for Amazon). That's so far outside of the standard compensation bands that other explanations are more likely.


You’re framing it the right way, and this is what we should work towards. But man, I long for the day when what you’re describing here is realistic.

In Norway, country of «holy crap, I’m an employer lobbyist and our salaries are so high we can’t compete in the export markets», you’ll only be around this level if you’re a freelance contractor billing 100% and doing all the sales work yourself.

There are some CTO and probably a few lead roles paying this much, but for an IC you’ll top out around $120k. Prove me wrong if other Norwegians have a better view :)


If you're reading this and are an startup founder hiring software engineers, one way to help kickstart the trend in this direction is to hire fewer engineers than you otherwise would have, and become _much more selective_ with who you hire, but pay them salaries that are competitive with not your local market, but with the hottest markets in the world (SF, NYC, etc), and actively advertise that you're doing so.

This will give your company a unique competitive advantage in capturing the best talent your market has to offer (and it likely has plenty of world-class talent to offer, but many of them end up getting brain drained away to those hotter markets today). It will also allow you to stay small for longer, dramatically reducing communication/coordination overhead and delaying the need to introduce management hierarchies, which is actually a substantial competitive advantage especially for early stage companies where agility and flexibility is the key to success. If all that's not enough, it will also do wonders for retention since it's basically a semi-permanent golden handcuff.

Over the long term, other companies in your market will end up having to compete with _you_ for talent, which raises the boat for everybody and helps your local market flourish with engineering talent and reverse the outward brain drain trend.

The unwillingness of employers in other markets to compete for talent outside their local maxima, optimizing for short-term self-interest over the long term health of the local talent market that they depend on for global competitiveness, is a true tragedy of the commons situation, and is a huge part of what makes Bay Area startups so dominant.

It's also what forced my hand into moving to SF myself. It would have been extremely financially irresponsible of me to stay. By leaving 5 years ago, I tripled my salary compared to competing local offers, and one job change later I'm now making more than double of that (at a startup, not at one of the big tech corps).


This is completely unrealistic.

Nowhere in the world outside SF/NYC is there so much VC money that can support paying developers those salaries.

I would say over 90% of tech companies make their money from local markets. You can pay your developers as much as you make from your local market.

As for hiring. You have no idea how well someone is going to work out. Some people are really good at interviews but turn out to be terrible employees and vice versa. This is why the hiring process in tech tends to be pretty convoluted and broken in general. No one can figure this out. The laws regarding hiring are also much less liberal outside of CA. Hiring and firing is severely restricted in many places.

Very few companies are in the position of being "unwilling" to pay developers more. Some with plenty of cash are even simply limited by the tax laws how much they can pay. The cost of paying someone too much over local market rate incurs obscene tax costs.


> The cost of paying someone too much over local market rate incurs obscene tax costs.

This is fascinating to me, do you have any examples of jurisdictions where this is the case?

Here in Germany the “taxes” (they’re technically mandatory insurances) than an employer will have to pay on top of the gross salary all have a salary cap, the largest one being around €85k. After that any increase in gross pay has no additional cost. Of course my personal tax rate might take a hike but that’s not the concern of my employer.


For slovenia taxes are progressive, above 10k a month is the highest bracket and taxes are 61%. If you want your employee to make 10k a month, your company pays 24k. If you want your employee to make 5k a month, the company pays almost 12k.


I think that’s a foolish comparison. When talking about these super high SF salaries, folks are talking about _gross_ salaries. Nobody will reject a higher salary because they’ll have to pay more taxes.

A lot of the conversation is about the value of the developers work being high enough to warrant higher salaries. With that argument we should look at gross salaries + taxes/fees paid by the employer on top of the gross salary.

Your personal tax rate is between you and the state.


I was not talking about a personal tax rate. My point was that employers can not pay anywhere near SF salaries because some places tax high salaries so severely. It is not a matter of employers being unwilling. A rich company can't just decide to pay people at SF levels and expect to create a local silicon valley.


One key part of the suggestion that might have been overlooked was to hire fewer engineers than you would have otherwise.

This could mean hiring 1 really amazing engineer instead of 2 or even 3 average ones depending on the realities of your local market, and pay them based on what it would have cost you to pay the 2 or 3 engineers combined.

This is always a choice you have available, unless you're bootstrapping and can't even afford to make the first hire at those rates, at which point you're probably not looking to compete on the global stage to begin with, so this suggestion doesn't really apply.


If a company knows that it'd actually cost 300k to hire the local kind of engineer that could make 150k in the Bay Area, that company is either going to move to the Bay Area or hire Bay Area-like engineers as remote contractors to get their 150k worth of engineer for 150k.


In Romania I'm paying about 45% of the gross salary to the state, and about 55% to my employee. The higher the salary, the worse that ratio gets.

Of course, the employee then has to pay VAT (19%) on everything they spend those 55% on.


The "able to pay" part will resolve itself if you manage to start a sufficiently successful startup. Eventually through high enough profits, since the startups that turn into global leaders in their field are more profitable per employee than anything a local non-tech company can compete with.

Getting to that point is a bit of a chicken-and-egg problem, though. And firing policies is a political barrier that's hard to avoid. I suppose you could keep everyone on as contractors, paying them standard rates but going to them directly, not through an agency. That would land the salaries in the region suggested, and you'd still have the ability to fire without cause.


Lack of VC money makes starting a sufficiently successful startup a bit difficult. There is a reason there is only one silicon valley.


There are more silicon valleys, like Tel-Aviv or Shenzen, but arguably the Bay Area is the only one merit-based. Good luck getting funded at the other two as a foreigner.


Interesting thoughts. I think you're onto something.

Going off on a tangent: I personally won't leave for California/SF since I have family here and won't put the obviously massive financial advantage ahead of that.

But I did consider this when I was in my twenties. It wasn't a super easy choice. Interesting to hear that someone have taken the leap. Not multiplying one's salary by 3-6x could indeed be described as quite financially irresponsible, and the alternative has to be pretty important to pass on that.


Sadly, you're absolutely spot on.


> I'm sorry to hear that you're not better compensated.

They are in the UK with a somewhat decent healthcare system, somewhat sane rents (notable exception: London) and a somewhat sane pension system.

US salaries are so high because US salaries have to cover for all three points additionally.


Companies that pay well in the US also provide good healthcare benefits. There are some weird setups like HDHPs where they give you free money at the start of the year, then immediately charge you so much for prescriptions that it gets used up again, but I think this is a way to let us farm credit card points.


That’s just silly.

I lived in the UK with all those things you describe, but I still made a point of billing my Bay Area rate because that is what the work product of a software engineer is worth on the market.

So, no. Even if you live in your parents basement and they cook your meals and take are of your other expenses, you should still approach your work life as though you were an adult.


Never lived in the US (and haven't lived in the UK for years) but the idea that you need a SV salary to get health care which meets or exceeds that of the NHS sounds a bit suspect.


From what I know, moving to US from a country with good socialised health care is a lot like selling a lot of put options on your health with different strike "prices". You might get rich, or if you are very sick, you are bankrupt.


Multiple studies have shown that Kaiser Permanente (an integrated health insurance/care provider—basically a non-governmental equivalent of the NHS in comprehensiveness—that is available in many US states) is more efficient and effective than the NHS for about the same cost.

Examples:

* https://www.bmj.com/content/324/7330/135

* https://www.bmj.com/content/327/7426/1257


You seem to have some simplistic view of how global economy works if you think all around the globe people should be payed the same for the same profession. Unless you work as freelancer in which case most of the time you have to pay your own taxes(which in most countries will amount to a big percent of your paycheck), you will get salaries according to the local market price standards regardless of what your peers in another country get. There are at least a few reasons for that one of which is that the market would not allow for huge disparities in salaries compared to other professions and to the same profession with local funding. You can try to negotiate good bonuses and/or equity but that is not mandatory and it is as much as you can get.


An honest question... By now you could probably retire a few times over. What keeps you going?


Actually, I have no intention to "retire". (In recent years I've taken the month of July off to take my family sailing; I can easily imagine taking longer stretches off, and/or shifting my time further from paid work towards hobbies... and at some point maybe even pivoting to writing and making music full-time.) My "ikigai" is, thankfully, mostly decoupled from making money per se.


When you realize how high cost of living is in cities paying those salaries, how much education (for potential kids) is, how you must save for your retirement entirely on your own... and then of course insane out of pocket medical costs... you soon realize that when you adjust for lifestyle you may be in fact much better off than the person with stratospheric salary.


I'm in the bay area. For what it's worth, my personal out of pocket max for healthcare is something like $2500/year, and they were only half-joking when I got the "million dollar work-up" last year at UCSF. I spent 3x that when my dog ate a corn cob... There is the additional cost of premiums for the rest of my family, but it's not absurd.

Real estate and property tax is rediculous, state income tax is high, and restaurants around here charge $18 for a hamburger these days. We're going to have to move once the kid(s) are old enough to be hindered by the bad schools. It's enough to perhaps make the high salaries not quite so competitive, and especially so if you need to pay for private school.

Salary is only a partial picture of compensation in the bay area, though. If you're a software engineer at a FAANG type company, you're potentially getting 1-2x your salary between bonuses and equity. If you joined early on at a start-up that's doing well, you're perhaps getting set up to spend your 40's mowing lawns like Forrest Gump if you want. It's enough that, even if you take very conservative career choices, you could work hard out here for a decade while living comfortably, then move somewhere else, buy a nice house outright, and work for fun while not stressing out about money ever again.


Right. A very, very ordinary apartment in NYC is $6000/mo. Figure 30-40% of your salary for taxes. If you made only $240K in NYC, your apartment would use up fully half of your take-home.


Even assuming this is true ($6k a month sounds absurd), this is a bad argument. The old "rent should cost no more than one third of your income" rule is basically for people around the poverty line. For them it's a question of whether they can afford to eat that month.

Just to give an extreme example: if Jeff Bezos uses half his income this year to build the largest estate the world has ever seen, he's still sitting pretty on the rest of it and has the largest estate the world has ever seen.

At a 35% tax rate, your take home income is $156K. At $6K per month ($72K/yr), you have $84K left over after rent. If this is you then YOU ARE WEALTHY. There is simply no getting around this. You are making more alone, after taxes and after rent than my partner and I have ever made together total in a year in our lives, and we live in a city with a relatively expensive rental market and are not "poor".

The absurd talk about "only" making over two hundred thousand dollars a year (plus whatever your partner makes if you have one) needs to stop. It's so out of touch with the norm in virtually every city in the world that it would be laughable if it weren't kind of sad.


<https://mkorostoff.github.io/1-pixel-wealth/>

People taking in $200k are not in any sense "very, very wealthy", except from a third-world viewpoint. $200k in today's dollars approaches what would have been called "middle class" a few decades ago, when there was one. Now you are only low, below $300k, and high, above $1M, with very few left between.

(In countries with regular health care and education, life costs less, so this about the US.)

The plutocrats have been very successful at deceiving American voters that they are not almost all now lower-class. The Powell Memorandum, 50 years ago, laid out the plan. It has been followed since then, and Americans now are routinely persuaded to vote against their own interests.

Wealth produced in the US has been rising as fast since 1975 as it did between 1945 and 1975, but normal Americans' income since 1975 has been flat. The whole difference has gone straight into the pockets of the extreme rich, and they are who are very, very rich. You and I have no concept of how different their lives are.

In 1975, regular parents could pay for putt their kids through college while paying their mortgage.


>>People taking in $200k are not in any sense "very, very wealthy", except from a third-world viewpoint.

Mate, I don't know what to say. You live in a bubble made out of solid stone granite. Literally anywhere else outside of US that kind of salary makes you "very very wealthy". Within top 1% definitely. So I don't know if you think anything outside of US is third world? Or what?

>>(In countries with regular health care and education, life costs less, so this about the US.)

So how does this work with your statement above?


Simple: Outside the US, $200k is very comfortable, except in certain places like Moscow and London.

In the US, it would be lower middle class, if there still were a middle class. So, instead, it is high in the lower class. If you consider it wealthy, then congratulations, the propaganda is strong with you.


No "very, very ordinary apartment in NYC" costs $6000 a month.


What would be a very ordinary apartment anywhere else is considered palatial in NYC. NYC residents have got used to much reduced standards.


Where, in the middle of Manhattan?

I'm living in NYC right now and paying less than half that in monthly rent.


Yes, in the middle of Manhattan. A substantial number of people do live there. For reasons.


You are doing it wrong then. I was making 100k GBP a year in 1998 in London as a C++ developer, the jobs are out there if you have the talent.


I'd love to know where and how, because I have never seen a position offering £100k and above anywhere in this country. I have friends working as seniors for Facebook in London and they don't make £100k.

I have two guesses - either you were a contractor, or worked for a bank where I've heard it's possible to cross £100k salary. Or third option, you mean <£100k of salary + bonuses?


FAANG pay terribly in the UK.

The one I worked for wanted me to move to London, and I said fine, as long as you give me a comparable standard of living.

Nope, they took my Irish salary and converted it to GBP, and that was it (this was post Brexit, btw).

Obviously, I didn't move. And working with any engineers in the London office was a massive pain as anyone without family commitments worked in London for a year, and then transferred to the US for a massive salary bump.

The core issue is that FAANG don't benchmark against finance in London, which means that pay is pretty out of wack.

For instance, I got a 150K offer from an investment bank in London around that time, and I'm definitely not as useful in finance as I would be in tech.

It just blows my mind that they haven't fixed this to be honest (I suspect that they'll have to, now that Brexit has actually happened).


Almost certainly the conversation is about total comp rather than base salary; at FAANG the difference is marginal (especially since at Facebook, there's no 1-year cliff; your equity starts vesting immediately every month).

But even putting that aside, most senior (E5) Facebook engineers in the UK make 125-160k base salary in the UK. Even if that's denominated in USD, that still leaves a significant % breaking 100k GBP base.

Source: https://www.levels.fyi/company/Facebook/salaries/Software-En... (filter for "United Kingdom")


> I'm a lead C++ programmer in the UK and I don't make that much.

Normal software engineers make that in Sydney and Melbourne. You need to negotiate a higher salary or switch jobs.


As I explained in other comments, salaries that high don't exist in the UK. And you can't just negotiate to have your salary be double the market rate where you live, no matter how good you are. Also, isn't Sydney the most expensive place to live in the world? My mortgage where I live is £600/month for an actual house, what good does making 2x the money give me if I have to pay 4x the money for rent?


It probably would've cost Rockstar at least $100k in employee time to have one of their teams find, debug, and fix the issue.

The extra time can now be spent playing the game instead of waiting, increasing revenue.

So relatively speaking, it's a low payout for a huge company like Rockstar to give on their game grossing billions.


$100k pays a dev several months. Someone who has source access and needs more than a few days to fix this problem doesn't deserve that kind of salary.


I said employee time as in it being a dev team effort not a single developer.


Yet, instead they took years and failed to fix it.


Which shows you that before that blog post they valued the fix less than that.


Rockstar will make tens of millions of additional revenue from increased player retention over the years (GTA Online is a massive cash cow, it is pretty much the original hyper-monetized game-as-a-service as a continues to fountain cash to this day), slow load times are an extremely commonly cited reason for marks (excuse me, customers) leaving the casino. This is a product that had made six billion dollars as of two and a half years ago, it is likely the most profitable entertainment product ever created, $10k is an almost insultingly low fraction of the value he created for the company.

Better than nothing I guess, but suggesting that it's OK to exploit people because of where they live is kinda crass. This is an industry where you are (should be) valued based on what you can create, not where you live.

To flip this around, if this guy lived in Haiti would you say it would have been fine to give him $1k because welp, cost of living is even lower there? Insulting.

And again they had years and years they could have fixed this themselves, at in-house prices if they wanted to. They had to be shamed into it by a third-party developer pointing out their "could be fixed in a day by a single developer" tier mistakes. O(n^2) algorithms causing minutes-long load times, repeatedly reported by thousands of customers across the game's 10+ year lifespan.

It's like giving a waiter a 1% tip. Is it nice that you gave them a tip at all? I guess. Technically nobody asked them to get you your food. Is it a reasonable fraction of the value delivered by customary standards? Not really. And in this case it's like a 0.01% tip.


"It's one banana Michael, what could it cost -- ten dollars?"


"There's always money in the banana stand."


> $80K would be on the order of half of the annual salary of most developers outside of SFBA.

If the author of the bug fix is indeed from Latvia (.lv), the median annual income of a senior software engineer in the Baltic states is currently around 30K-40K after taxes.


Sorry is that number in Euros, USD or another currency?


It's in EUR, since all the Baltic states are already in the eurozone.


Half? I live in the Midwestern US. Have been dabbling in software for most of my life, have some decent working examples and am finishing my BS:CS this year.

I can't even get close to my current salary in my current unrelated field ($85k) to even consider a career switch. Are the jumps from junior to mid-level/pm that high? I am a PM in construction so that may end up being my path.


Junior to Mid level is pretty significant, even outside of the bay area. At a finance company, Junior to Senior for me was about $45k base salary over ~5 years, with another $20k in bonuses annually. If you can only find say a $70k job, I would think its not unlikely that you would be able to work 2 years at that, and jump for $90-110k.


As someone searching for employment in the Midwest, I feel your pain. I saw a position yesterday that wanted 15+ years of experience and paid, I kid you not, $17/hr.

Compensation in the Midwest is a joke. If my family weren’t here, I’d return to one of the coasts.


They're looking to fill the position with a foreign contractor, almost certainly. I wrote some PHP sites 20 years ago and get contacted every now and then about taking a 2 month contract for $16 an hour writing PHP in Pittsburgh, which I'm sure has its charms but is also halfway across the country from me.


The terrible smells, pretty sunset, and constant threat of petty violence in SF is offset with significant extra hiring fees.


Dude, I'm currently living in Japan and $80k is almost double my salary, and I'm already above the national average and live pretty decently here.

Disclaimer: I don't live in a big city like Tokyo.


80k is about the ANNUAL salary for most devs outside of SF.


> $80K would be on the order of half of the annual salary of most developers outside of SFBA.

Wow, I need a raise.


Edit: Nevermind misread it.


we’re punishing rockstar for giving this dev $10k when there was no bounty program in the first place


I think this is a great example of a fallacy that's pretty popular on HN: That work should be rewarded based on "value created" rather than effort.

I don't think this line of thinking is helpful. Should Amazon increase the price of their VMs if you made more money with them? Should a doctor charge a lawyer a higher price for a flu shot because their time is worth more?

Of course not. Because in all these cases, the clients would just go buy someone else's service.

Of course it is helpful to consider the value of the work you produce, and you can use that in negotiations. But you are not entitled to a higher rate just because your client is big or successful.

If the developer had offered a two day consultation for $80.000 to Rockstar they would most likely have declined and said, sorry, for that money we can hire someone else to do a lot more performance improvements.

If you want to get paid based on value created, you need to negotiate. But you don't have a moral right to get more money for the same effort just because your client is rich.


>For that detailed effort, analysis and with a fix, I'd say that is worth around $80k.

Do you think the fix is actually worth 80k in terms of additional revenue for Rockstar? I wouldn't be so sure. If not though then no it's not worth $80k.

"I'm a developer and I'm paid a lot to do my job, therefore everything I do must be very valuable" is not the right line of reasoning to use here.


It’s absolutely worth that much and more. GTA5 Online pulled in well over $500 million in 2019, and even more in 2020. It’s now the 3rd best selling game of all time, behind only Tetris and Minecraft. https://www.tweaktown.com/news/69666/gta-made-half-billion-d...


That's not a relevant number.

The only relevant fact to justify $80K is -- would slow loading times have resulted in at least $80K of lost sales going forwards? Or at $50 a copy, then starting today would there be 1,600 people who would buy GTA V, but won't because they know loading times were slow? Maybe because a friend or reviewer mentions that as a con? (And you can also add lost DLC revenue from people who gave up playing earlier than they would have otherwise.)

Honestly, that's an incredibly hard question to answer. But it is in no way obvious that it is "absolutely worth that much and more".

In contrast, $10K is a nice round number that demonstrates gratitude for their charitable work. And keep in mind this was probably a weekend's worth of investigation or so. Not many months of dedicated work, which is what $80K buys on the job market.


If the 500 million figure is correct, you would need only a 0,02% (1/5000) increase in spending for a change to be worth $80.000.

Diminishing the notoriously bad load times of GTA:Online by 70% could probably do that. I actually never got into the multiplayer because at the time my HDD was taking several minutes to load the game.


Just a note, it doesn't apply retroactively.

It doesn't matter how much money GTA V made in the past. This applies only to future profit that would be have been lost going forwards.


It's not about $50/copy, it's about lost revenue from GTA:Online which makes money off micro-transactions and continued spending from existing customers. And yes, I think it should be obvious that very long load times has made a significant number of people spend less time and money than they would have with fast load times.


Hard to say, but definitely more than the 10k, all the people that just clicked off before it loaded.


This reminds me of my 13 year old sibling yelling at our mother when we were teenagers for not buying them the exact item they wanted, but something pretty close. Disappointing.

One should not hold people responsible for acting "non-perfectly". Any behavior exceeding "average" should be met with admiration and gratitude, average behavior with "respect and appreciation", and anything somewhat sub-average with "mild disappointment but no surprise and some understanding".

Having a company spend money they are technically obligated to allocate to share holders, to a developer who, without any pre-existing contract or even verbal or commonlaw understanding, helped them, is way beyond "average response" for modern businesses.

Please, don't hate on the vanishingly small number of counter-examples blazing a trail in this space.


True. I’m glad he got anything at all. Game improvements generally don’t. Think of all the mods that exist just to fix bugs in say Skyrim


They tend to get a lot of community accolades but yea - spending time fixing someone else's intellectual property isn't a great place to start your proposition if you want to make money.

Even traditionally mod friendly companies will tend to sweep in and either block any derivative work that's monetizing their improvements or else negotiate with the creators to take the lion's share and republish the content under their company label. There are a bunch of examples of this but Counter Strike is a pretty famous one.

That all said, it's pretty hard to argue that making minor alterations to a product should allow me to resell it as a novel work. Fan Fiction exists - but it doesn't (legally) exist as taking a novel and taking on a few new chapters then reselling it without giving a portion to the original author.


He got paid a good month salary for spending a bit of time fixing an issue (which is tricky but not extreme if you know re). I feel like people having strong opinions that's not enough should just open a GFM campaign to give him more and add their own money. It easier to say how others should spend.


> even though they did 98% of all the work.

You are forgetting the game which Rockstar made. A loading times fix is not 98% of the final product.


That's basically 6x the average Latvian developer salary. The country the original developer is from.


6x their monthly salary*. In a lot of European countries, salary is always referred to by the monthly amount, not yearly.


they could've given him nothing tho so im glad they gave him something. sure it would've been nice if it was more


The only reason he got paid at all IMO was the widespread exposure the original article got. When you find something this significant you’re almost obligated to do a song and dance and tell the world about it, so you can get the attention you need and be paid.

Frankly for an introvert with no public online presence or audience that’s just not very appealing. There’s easier ways for software developers to make $10k: just wait for two paychecks.


I think the author was motivated by intellectual curiosity rather than money. Thankfully he was enthusiastic enough about it to share it with the rest of us.


Good on them for acknowledging something a little embarrassing and responding with gratitude instead ignoring to try to save some face.


I'm not sure if "embarrassing" is the right word here.

This bug has existed for years and has lead to hundreds, if not thousands of people quitting the game citing this exact reason yet Rockstar did nothing.

I think "shameful" is really more appropriate. It's not like they simply didn't know this existed. I think they have known all along it existed and didn't give 2 shits to fix it.


Watch out not to condemn them no matter what they do. Sure, they messed up in the past, but given the situation they found themselves in when they saw this report, this response is still good.

You risk making discourse difficult when you continue punishing people who are attempting to do the right after doing wrong for a while.


I disagree. They ignored the issue for years and that part won’t be erased from memory. They should be given credit for finally fixing it, but nothing more. What happened before this week was and remains forever shameful.


Very likely that internally there were people who had an inkling this was going on, but could never convince product to devote time to put it in the backlog. This change is a win for that engineer.

This example will be ammunition in internal debates about how much to weigh product pressure vs engineering suggestions that improvement is technically possible; the next time an eng says "let's look at this slowness one more time, it may be fixable more easily than you think", they might be listened to.


> What happened before this week was and remains forever shameful.

lol without context you'd think something tragic actually happened here.


Losing customers over such a simple, known problem, over years… is tragic for a company like Rockstar. This is like Microsoft ignoring a bug that made Windows take 10 minutes to start and just letting that happen for 5 years. Now tell me that’s not tragic.


Anyway GTA5 is just a game.


To you it's just a game, to Rockstar GTA is literally a multi-billion dollar product with a household name that politicians talk about.

I can't believe you're missing this huge point.


> You risk making discourse difficult when you continue punishing people who are attempting to do the right after doing wrong for a while.

On the other hand, a few days of doing the absolute bare minimum of quality control doesn't undo a decade of actively hostile practices. They're still very very deeply in the red when it comes to their relations with the playerbase and for good reason.

Paying $10k to the person who documented this bug is a good start, but you need to do a whole lot of work to make up for the impressions caused by that kind of prolonged systemic negligence.


Right. I'm not saying eval(their net behavior)>0; I'm saying that if your response function is always negative, you lose the ability to influence their response.

Have you heard the one about the country with the death penalty for all crime? It just leads criminals to maximally try not to get caught, without regard to damage done [i.e. removing all witnesses to even the smallest crimes]. This isn't what you want; even within "negative" relationships both sides lose when you don't have a gradient of responses.


The crux of this is that most readers presumably found out about both the bug and the fix in relatively short proximity, and their response is negative to the combined events, not to the knowledge of just the patch.

A criminal prosecuted for murder and petty theft simultaneously can't really go around complaining that they got 25 years in gaol for stealing an apple, but if everyone had already known about the murder and dealt with that accordingly it'd be pretty harsh.

(not that fixing a bug is exactly a criminal act, but the general principle remains)


An alternative possibility: there were 100 bugs of similar consequence, all of which they cared about, and they got 99 of them. This one escaped, because stuff happens.

> I think "shameful" is really more appropriate.

When you make an inadvertent mistake in the future, karma is coming for you: people will speculate that you "didn't give 2 shits".


I’m sorry but no.

This bug existed for a very long time and was the #1 outcry from gamers. It is well documented all over the internet.

Every company I’ve ever worked at and every code base I’ve ever worked on would have made this a #1 priority. If it were a technical limitation of the devs I would agree it is “embarrassing” but this was not outside of their ability.

I simply don’t blame the devs, I blame management. Management should have allocated the resources to fix this truly game breaking bug but they didn’t because money was still flowing. Is it any coincidence that ads played during the 6+ minutes you would wait to get into a game?


Yea, if it was a 6 month old bug or it was a hard to repro edge case, then sure it’s embarrassing. Oops! But years long, and highly visible/reproducible, without correcting? That’s veering into shameful.


Blame should always flow up, but that doesn’t mean there aren’t other causes. For all we know there has been a team of devs putting their kids through college “working” on this for years.


More likely, there were 99 other bugs that they had clearer metrics related to monetization that they prioritized (not that this bug didn't impact monetization - I bet it did, just not in a way they were measuring). Or that their QA engineers identified and flagged as high priority or major new features with their own bugs and regressions or whatever. These things get lost in the shuffle. More likely there were 1000s of bugs in the tracker, probably 10s of thousands.


Yes, I agree with your elaboration. Also, of the 100 bugs, they didn't get 99 of them, because there are surely some left. Maybe they got 90, and there are 9 still hiding out.


The bug was live and wasting paying-customers's time (and apparently spamming them with ads too?) for 6 years.


The game is played by millions of people, we don't need to wonder if there were a lot of more important bugs, this has been by far the most complained about issue for many years.

> This one escaped, because stuff happens.

No, they mostly just don't listen to their players when compared with other game companies.


I understand your frustration, but anyone who's run (or been on) a dev team knows you have a million things you could/should be doing, and many, many priorities. Often supporting new launches/features is the priority.

Sure, ideally there would be someone internal advocating for a fix like this on behalf of customers. But Shameful isn't the right word. It's a miss like any other.


The egg that's on Rockstar's face has more to do with how much money they likely left on the table by not diagnosing and fixing this issue early on. (And assuming they had a team overseeing this aspect of performance with adequate tools to observe and debug it should have been trivial.)

I personally feel like the magnitude of this -- especially compared to how simple the solution is -- makes it a miss that's really much more extreme than most others.

EDIT: Though I agree that "shameful" isn't the right word to use. Rockstar wronged themselves much more so than any other party.


Obviously you are right... but...

If the fix had been some super application specific super obscure thing, we would all understand.

Instead... the bug is easily describable to anyone who knows what JSON is and the fix is “don’t parse JSON the slow way”.

So I think some of the leniency afforded to the devs may be misplaced. Then again, hindsight and all that.


The cause doesn’t actually matter much, usually one doesn’t know the issue until they investigate. So often issues remain open for lack of interest, then someone finally does and it’s a missing comma. This doesn’t affect the fact that the issue has been ignored.

I’m sure GTA has unlimited bugs to look into, but a bug that turns people away before they even start should have probably been given higher priority.


> If the fix had been some super application specific super obscure thing, we would all understand.

Would we?

Even if I don't understand what it's doing, devs doing nothing about an O(n^2) algorithm that takes multiple minutes every load is clearly a big mistake.

Even if it's something that has to be that slow, cache the result.


Oh absolutely, I have a whole host of stuff on my todo list for projects I'm developing that has been there for months and will remain there for months longer.

That said, they're all things like "refactor the frobnicator code so it's more obvious what it does" and "make templating system log changes to server". Nothing on my list says "Look in to that major performance issue that literally thousands of customers complain about every single day". That stuff gets sorted immediately.


If so then they still acted in a gentle way that is rare afaik. If they were so careless they'd just lift the fix and ignore the author.


There probably is a story behind it. Like some developer looked into the bug in the beginning (when when the load time was not that bad because the json file was smaller), saw that the time was spend parsing json, and then, without thinking too much whether this makes sense, wrote in the bug ticket that, to make it faster, they would have to switch from json to something better/faster/different.

Since switching the file format is probably some work, maybe involving many differnt teams and network protocols, the bug was closed because it too much efford.

And then, whenever somebody complained, the old bug was referred which unfortunately never got a proper analysis.

Sad, but I saw this happening many times in the past in some projects. Since the old and wrong analysis will be in many powerpoint slides and heads, it is incredible hard to argue against or get even time to work on such a problem. Usually, you need someone who just takes some time form other projects without telling anyone and ninja fixes the problem to solve it.


Based on my own experiences, I'm absolutely certain many people over many years tried to get the load time fixed, and then were beaten down for their earnestness.

Just as I'm certain that this unsolicited 3rd party fix was used to settle some old scores.

Corporate politics is a blood sport.


you forgot to mention that ads are displayed during the loading screen


I think "embarrassing" could still be the correct word. If you were to ask the development teams responsible they might feel a little embarrassed over the solution being so small (yet technical).

And if they didn't care at all given what you say then why would they care to implement a fix for it all these years later?


This is akin to that guy buying a whole-page ad to complain about his ISP recently, resulting in enough shame to trigger action. It's a bad look for Rockstar considering the income that title generates and at most it was a few weeks worth of dev time. Very interesting to watch the public reaction to this in comparison though, seems to be a lot more sympathy for Rockstar than for ISPs.


They still made like a billion dollars in shark cards so apparently all the people buying shark cards didn't give a shit about the load time issue and that's the real problem.


It's a video game, so maybe give them some slack. We're not talking a SolarWinds breach here.


The game supports 8 platforms, has virtually miles of terrain, and now Rockstar is suppose to feel ashamed because they didn't fix a big performance issue that affected startup times?

The game probably has tens of millions of lines of code.

I'm glad they shipped. If they can enhance the startup time, great, bonus for them. Additional credit for thanking the third party that uncovered the perf issue.


I don't think you've been following this story. If you looked into the technical details [0], and surrounding context, you would understand why your comment isn't particularly insightful. For example, terrain has nothing to do with this bug and it isn't a complex fix that interacts with "millions of lines of code." A single run of a profiler would have caught this.

I also think it's relevant to note that GTA V has sold 140 million copies, 20 million last year alone, and Rockstar makes hundreds of millions of dollars a year on the title. For a multi billion dollar product to have a known, relatively simple, problem is shameful. I don't know what went wrong, but I don't think it was the competency of engineers or the complexity of the problem that allowed this to fester.

[0] https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times...


I think you missed previous news about this. This was an incredibly trivial issue that would show up in any use of a profiler on the slow startup screens. The fact they didn't is that they're either unable to run profilers on their code at Rockstar (which would be shameful indeed) or they never bothered to do so. Either case shows egg on face at Rockstar.


Strange that this was this downvoted. It is an issue that a single dev with a profiler should've been able to identify very easily. That they hadn't addressed it despite it being such an annoyance for so many users for so long is embarrassing.

Not to take away from the feat and initiative of doing so without access to source code.


+1 — it’s an item a single person without access to the source code was able to diagnose and fix.


I’m pretty shocked at all the defending of the game company here, honestly. Highly visible bug, affecting high percentage of users, relatively simple fix. I take pride in the software I wrote and would be personally very embarrassed if I caused this AND didn’t fix it for years. I guess companies don’t have feelings and can’t feel shame but I would personally feel shame as a supposed professional!

There are a lot of excuses for why this wasn’t fixed but no great reasons.


The idea that screwups are "shameful" is poison in any organization. I fight it in the companies I work for, I fight it here.

Bugs happen. Serious bugs happen. They will always happen. Shaming people for bugs undermines institutional robustness against bugs because it incentivizes people to hide mistakes. Furthermore, companies with blame-n-shame culture typically fault users when those users make inevitable "mistakes" statistically guaranteed by bad UX design.

In contrast, transparency is good, acknowledgment is good. Rockstar acknowledged fault and rewarded the reporter, which is difficult. That's a positive institutional arc I'd like to encourage, at Rockstar and in the wider industry.

If you stop at "embarrassing", I'm with you. But when you proceed to "shameful", that's quite different and we part ways.


Just to clarify my point: The bug was a screwup. We’ve all done screwups. My worst bug was an amateur hour royal screwup that resulted in DDOS. Nobody should be shamed for bugs, including serious ones that get to production.

Not fixing a serious, visible, known bug for six years is what’s shameful. There is nothing positive about neglect like that. It’s kind of admirable that they actually rewarded the guy but it shouldn’t have even got to that point.

Totally agree you should not shame for bugs. The shame is for ignoring this bug until one of your customers finally had enough and fixed it by patching the binary. If it was my company I would feel ashamed.


I haven't read all the comments on these articles, but I haven't seen anyone claim that the bug shouldn't have happened, that anyone should lose their job, that no one "good" makes those sort of mistakes, etc. But this was a bug that lasted long enough and irritated enough customers that one of them finally diagnosed and seemingly fixed it without ever looking at their source code. That's one hell of a process failure. And splitting hairs over whether we would feel professionaly "embarrassed" or "ashamed" in a similar situation is pretty silly.


Everybody makes mistakes. Bugs are inevitable.

Managerial problems that allow high-impact, low-fix-time, high-prevalence bugs to exist for 7 years are shameful and are in fact poison to an organization themselves.

Don't mistake the shaming for really being about the bug itself here. The problem isn't the bug, it's the organizational culture that allowed such a massive, easily fixed, widely encountered bug to persist for 7 years in a game that has made them $6 billion dollars over the last few years.


> now Rockstar is suppose to feel ashamed because they didn't fix a big performance issue that affected startup times?

Yes, they should be ashamed of this.


It's true that the game is probably absurdly complex, but how did they never profile loading times?


I'm genuinely surprised. Given how long they've left this untouched I thought they didn't care at all.


They probably just didn't have anyone working on it that had the knowledge (or incentive and/or permission) to fix it. AFAIK they've just been adding content to the online portion for the past 7~8 years, and you don't need engine programmers to do that.


"They had never gotten any programmer with the requisite expertise to take a look at the code in nearly a decade of opportunities" sounds more or less like a verbose phrasing of "They didn't care."


I mean someone probably looked at it briefly and said it was unfixable. Then management said don't waste anymore time on it.


Why would they care? The game made > $800 million. Players loved the game despite load time issues.


Haha, oh kiddo. It was $6 billion as of two and a half years ago. GTAV is the most profitable entertainment product of all time.


Even more than WOW?

I don't have the numbers but would have imagined that still takes the cake.


Because the could have made twice as much.


The fact that they didn't make this a priority to fix shows exactly how much this company actually cares about it's customers and it's reputation. It's unbelievable that despite the quite public outcry about the issue NOTHING was done until some 3rd party guy figured it out for them.

I'm quite surprised they are even going to patch the game because the level of "we dont give a shit" I thought they had about this issue certainly would have prevented them from doing anything now.


Imagine all of the quadratic poorly written code sitting on your code base at your workplace. It's probably not a handful. People need to stop pretending that their code is optimized and magically linear all the time. No one looked at this because they didn't know it about. There is no reasonable way for any person to fix things like this unless they are actually looking for it. How many people at Rockstar would be responsible for engine details like this and have the skill to find and fix it?

A game video company employees artists, sound effect engineers, and etc. They aren't full of programmers combing over the source code for quadratic footguns.


Noone is complaining that they did not look through all their code for quadratic footguns. There probably are bunch of places like that, but they don't matter and doing that would probably be just a waste of time. Premature optimization is the root of all evil and it makes all sense to dedicate all your optimization on the 1% of code that actually matters the most - but this was the 1% that obviously should have been the target of optimization and was not.

The issue is that slow loading times was a contentious issue with many, many customer complaints (IMHO it was the #1 complaint) and during many years not once did they spend even trivial effort on trying to look into that complaint, and that's shameful. As the tostercx's article shows, if they had ever assigned any engineer a couple hours to look into why loading is slow with a profiler, that issue would have been easily found; the fix as described was a bit more tricky because the author did not have access to the source code, but within a company that would have literally be done within an afternoon by a single non-exceptional person. But they never cared enough to have someone look into it.


You have customers complaining about everything all the time (and non-customers with a high profile "edgy" title like GTA), so filtering out the actually relevant topics might be hard (esp things like "bad loading time" when it might be the users computer).

The thing is that this loading time issue probably didn't exist at release as it was part of something that got expanded on over time and only got bad worse by the years.

Assuming somewhat linear growth of the data the loading time would have been at least 4x shorter only half the games lifetime ago (3 years) and 16x shorter 1.5 years after release. After that period they would probably not consider there to be issues in those places in the code so nobody really looked or realized anything was amiss there.


> Imagine all of the quadratic poorly written code sitting on your code base at your workplace.

Imagine now that 160 Million people use my codebase and have been suffering from this for 7 years.


The problem is not that they suddenly hit this performance drop and solved it in some reasonable timeframe, or it was some rare occurrence affecting .01% of users

The problem is that it’s existed for 6 years, affected anyone and everyone, and would have been caught and resolved by basically anyone who attempted to look into the issue with fairly low effort/experience-needed. That is, the problem is that no one bothered to look into it, despite good reason to do so.


> No one looked at this because they didn't know it about.

What are you talking about? Nobody knew about it? Is that a joke?

This long loading issue was posted EVERYWHERE and was the reason so many people gave up on GTA v. It’s simply impossible that they didn’t know the issue existed.

That they chose to ignore it is the reality.


Looking back, I realize it's wrong to assert opinions as facts with no basis at all. I definitely shouldn't have said that when I don't work at Rockstar or know anyone who does.

But I still think people are demonizing too much Rockstar and are trivializing the bug when it's far from obvious. In my opinion, there is a lot of hyperbole statements about this issue when it's more useful to think if your organization could or would catch bugs like this.


Ok, to be fair I am doing to same thing (asserting opinions without facts).

I don't work at Rockstar so it's true I have no idea how this bug was handled.

I just have a hard time believing they didn't know about the bug, and I also have a hard time believing they didn't understand that it was one of the chief complaints from their end users (a simple check of any forum or discussion group is all thats needed).

So, I an only assume short of an actual explanation, that they intentionally sidelined a fix for this bug. Why, is anyone's guess. If you had to ask me, management put all devs onto money making tasks like new items and store support, and fixing quality of life bugs even ones as egregious as this one, was a low priority.


The bug affected practically every user.


In a completely harmless way


Frankly it might've just gone unnoticed however strange that sounds. Not privy to any details but a probably scenario is probably something like this.

1: Some dev gets tasked with implementing the system, at the time there probably wasn't that many ingame purchases so that 10mb json maybe was a 500k file tops? So going by the O(N2) factor it would've buzzed by in a second (probably faster since being smaller there would've been more cache hits giving another factor or 2). Being at about a second or less they don't even consider optimizing it since it's not noticed among all other assets being loaded.

2: The content/ops teams adds more IAP over the years (it's a datafile so that devs don't have to implement everything), since they're adding big models,etc they don't notice too big changes and probably just chalk it up to bigger models,textures,etc.

3: Engine devs (those most often tasked or obsessive about optimizations) might be running a cut-down server for most testing (not noticing the huge loading time) and probably presume it's other assets if testing in a full env.

So in reality it's probably just an organizational oversight rather than "bad developers", now considering this public "shaming" they received they might actually place some people in an production optimization role for a while and optimize other parts if it's possible (the question is.. will they find it worthwhile to keep up in a longer period?).


That’s why you sometimes need a task force of as few people as possible.

“I want you to improve loading times whatever it takes”

Sometimes you’ve got to go hunting instead of fishing.


Seems most likely that they had just never dedicated the time to investigating the slowdown.


Tbh: Looks more like they haven't been aware of it.

This can result in severe consequences for the developers who worked on this particular part of the codebase.


No it won't. It's ridiculous to blame a few devs for this. C/C++ has unknown quirks like this.


meta comment and completely feeling based : sometimes it really seems like companies, politicians, bureaucracies, don't have a process for acknowledging mistakes and/or lack of knowledge (and/or are seemingly not allowed to).


Ah that's because at many companies if you make a mistake you are a mistake.

Which is BS tbh. Everyone makes mistakes.

I think with politicians and political appointees, it's a little different because that's very much a public trust. Nobody deserves to hold political office, they should instead be deserving of it by being ethical and having integrity.


> Everyone makes mistakes.

Amen. And avoiding a culture of blame (taking inspiration from the NTSB) makes it much more straightforward to put systemic measures in place which reduce the impact of those mistakes. People still make mistakes at the same rate, but more mistakes get trapped and more negative outcomes can be avoided.

The vitriol of this discussion shows just how difficult it is to establish such a culture. People really want to see mistakes as evidence of moral inferiority.


This is very insightful. If you make a mistake = shameful, you probably aren't going to meaningfully decrease the number of mistakes made, but you will ensure that no one wants to admit they've made one.

And I guarantee anyone who's written code long enough has written plenty of footguns of their own.


I'd go farther than even that.

I'd say that by talking about mistakes without blame, I think we actually avoid people making mistakes because folks can learn from other's mistakes, and see them as something they could just as easily do, and so internalize the lesson.


Apologizing just gives fuel for revenge. In this case I think because the harm done was so small, it was economical to apologize.

Edit: an anecdote from the professor of my college's startup program (a hundred-millionaire former CEO/founder)- "never say sorry."


> Apologizing just gives fuel for revenge. In this case I think because the harm done was so small, it was economical to apologize.

Yeah, it's always a tradeoff.

I worked at a non-tech fortune 500 for a few years and was able to see the inside of repair/warranty/disputes. These were very high ticket items and the brand was/is very highly regarded for after sales support.

In the employee handbook for repair support staff (the people communicating with clients about the status of their repair/claim) there was a passage similar to the below:

"Apologies MUST NOT acknowledge anything other than the customer's explicit feelings without written acknowledgment of fault from $MANAGER".

I asked a member of that team to decipher WTF that meant, and they showed me an example email. They were only allowed to apologise in the below way:

"Hi bob, we're sorry you feel our product is faulty".

I always found these weasel apologies infuriating in my personal life.


Yep that’s awful. The reason for that rule is probably legal. If your company is taken to court over damages resulting from a faulty product, a written acknowledgment of blame will probably make that worse for the company.

It’s still awful though. I would hesitate to do business with a company that doesn’t take any responsibility for their own products.


Right, so the company is saying "I will never acknowledge our mistakes because if I do we might have to fix them." or "I am trying to trick you into thinking I am apologizing even though I'm not".

That's more offensive and insulting and time-wasting than saying nothing.


More charitably, people have an instinctive tendency to bend over backwards to apologize for things they're not responsible for in the interest of cordial relations. This is good if you're casually trying to make friends, but bad if the other party might take you to court. Companies therefore very reasonably explicitly correct for this behaviour.


Yeah, I agree. Avoiding apologies is probably a good idea for the company. The part that stuck with me was the "acceptable" pseudo-apology.

Not apologised at all comes off better than a backhanded apology IMO.


You're right about it being a tricky attempt at faking an apology.

But the below isn't correct:

> Right, so the company is saying "I will never acknowledge our mistakes because if I do we might have to fix them."

I probably didn't word it perfectly in my original comment, but the company did acknowledge mistakes quite often after assessments of the claimed faulty/broken/whatever product (roughly 30% of faulty claims were approved and either repaired or reimbursed from memory). The rejected claims were often taken to court (like 15%) due to the value of products.

I was mostly commenting on the deception of only allowing pseudo-apologies.


> "never say sorry."

Well, this HN hate fest of a discussion certainly bears out that saying! The internet is a terrible place. You will get dragged mercilessly no matter what you do. The incentives are all towards sweeping mistakes under the rug.


There are at least some examples of startups choosing to apologize, and try to make things better in the future.

I think the real advice is “don’t say sorry unless you actually mean it.” People see right through that, because your actions and follow through won’t match your words. But people seem to appreciate sincere apologies that come with substantive corrective action.


You can easily just not say anything at all


> instead of ignoring

as-if that's the worst they could do. Suing the guy, pressing for charges with CFAA or DMCA anti-circumvention. There were far worse ways this could have gone.


It is a common misconception that the victim can "press charges" against someone for a criminal matter. This is entirely false.

Only the prosecuting attorney (someone employed by the government) gets to choose when and how someone is charged with a crime. The victim has no say whatsoever in the matter, and indeed the DA can charge someone even when the victim doesn't want them to, or decline to prosecute even when the victim requests it.


I did not say "press charges". I specifically said "press for charges" deliberately. In that context meaning to attempt to persuade someone (in this case a prosecutor/police/etc) to actually press charges.

Most of the time when people simply use "press charges" this is what they mean. I specifically used "press for charges" to try to avoid such pedantry. I guess I failed.


In many places, a victim has quite a lot of say whether a crime is prosecuted.


If shorter load times there translate to higher conversions in the same way this connection is observed by Amazon [0] this guy just made Rockstar a little fortune. Hiring the blogger outright and giving them a VP of Something Something Role would be more adequate.

[0]: https://www.gigaspaces.com/blog/amazon-found-every-100ms-of-...


This is going to be a great point on his resume.

How many people get to say something like, "Reduced GTA 5 load times by 70%"?!


That's pretty amazing by itself. But it was doing it without access to the source code that blew me away.


That really is the cherry on top.

Ya know on second thought, it's so unbelievable that he did this whole thing that I wouldn't be surprised if someone reading his resume thought he was bullshitting.


Althought it is impressive, it isn't that uncommon in the reverse engineering world and quite believable imo.


And that's why you have source in your resume.


Gamers playing this game have all kinds of tricks to try to bypass the load screen itself because it takes a long time to go through it but the game by design forces you to look at the load screen multiple times just to do one task. One way that I remember was to play in a Solo session that way you don't have to load every time you come out of multiplayer. They even make a GTA online software tool that blocks GTA on the firewall at very specific intervals to force an online session in order to speed up load times.


There should be such tricks for Codemasters' games specifically and PS1/PSP/Vita games in general. All those intro screens and warnings are absolute bullshit, just publishers spitting in gamers' faces.


Unfortunately he will flunk the interview after failing to invert a binary tree..


I know it's tongue in cheek, but shouldn't you use a more difficult example? Inverting a binary tree is trivial. (Example solution: https://leetcode.com/problems/invert-binary-tree/discuss/111...)


It's a reference to the engineer that created homebrew but failed a google interview where they asked him that question: https://twitter.com/mxcl/status/608682016205344768?s=20


Not worth working there anyways.


It's truly amazing that he did all this while decompiling the source code, but for engineers working on a large system it's not that uncommon to fix large impact performance bugs.


Just today I improved performance of an enterprise system by over 15000% by commenting out a "sleep(1);" line I used to debug a race condition!

Might not put that one on the resume.


One person. Only one person gets to say that.


"...without working for Rockstar"


Is the resume where all cool accomplishments go after they've faded from the public eye? Your comment struck me as somehow extremely (and unintentionally) fatalistic... that this cool hack will somehow end up churned through the capitalistic society into a bullet point to earn a wage. Actually kind of depressing.


Well, what else should the fate of this achievement be? Very few achievements are immortalized.

I don't think it's got anything to do with capitalism, we just tend to forget things quickly.


I’m sure ZA/UM is hiring.


One.


Sans source code to boot!


The scale of this fix is mind boggling.

Millions of people play this game every day. This lone man, driven by his love of solving puzzles, is single handedly unlocking potentially hundreds of millions of hours of “lost time” across the globe.


I like that Steve Jobs was calculating people "killed"(lifetimes wasted) by slow start of OS to motivate engineers. We need more people like him.


Not to mention the huge amount of cumulative energy consumption on those wasted cycles.


More like, opened up more time for them to play! :)


t0st's original post has an update that says he got $10k from R*'s bug bounty program.

[1]: https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times...


Reminds me of a Windows installer I working on many years ago, that seemed to take a long time using CPU but otherwise not doing any real work. Turned out the SDK we were using tried to update the progress bar every kilobyte or something like that, and our near-gigabyte sized packages filled the message loop with a bunch of pointless updates (30.001% -> 30.002% but both displayed as 30%).

Once the vendor fixed the issue it shaved a few minutes off the install time that had previously just been a UI glitch (processing the large backlog of progress bar update messages).


> Turned out the SDK we were using tried to update the progress bar every kilobyte or something like that, and our near-gigabyte sized packages filled the message loop with a bunch of pointless updates (30.001% -> 30.002% but both displayed as 30%).

Had a similar situation with an Electron app I worked on. App’s window was getting very stuttery at some point during a long-running operation done by a third-party library, even though it happened in a background worker thread and wasn’t supposed to block the UI (which simply showed the progress of the operation).

Turned out, that third-party library called my progress notifier function (in which I used Electron’s IPC to notify the GUI) on increments that were fairly spaced out most of the time, but really frequent during some operations. Mere overhead from IPC calls and GUI updates added up to a point where everything lagged, even though the operation itself wasn’t that intensive. I wrapped my progress notifier in a throttler, which I frankly should’ve done much earlier, and that solved it.


This must be why when you turn off the progress bar for Invoke-WebResource in PowerShell your download speeds become reasonable, as otherwise it updates the terminal every byte and takes a million years to download stuff that takes a second in Firefox or via wget.



I want to see more of this from giant companies. This gives me hope that there are real people amongst the mindless drones in these hulking corporations.


You mean real marketing people, who know that talk is cheap and the small bounty might mitigate some of the negative PR they received because of this?


Don't put words in my mouth. I meant exactly what I said.


I know what you meant, but with those words I hinted, that the decision from Rockstar was likely more a move out of PR motivation, not moral standards, that you assumed.


I wonder how much money the load time slowdown has cost Rockstar -- given their revenue from "Shark Cards", which are bought by players who want the fun of playing with all the expensive in-game items without spending time grinding for them.

What's the correlation between being averse to spending time on less-fun aspects of the game (to the point that you'd pay Rockstar extra money to get only the more-fun time)... and being averse to waiting for long load times, when you just want to play a quick session of more-fun?

Given Rockstar's investment in a steady stream of new big-ticket in-game items, and their huge revenue from that, I wouldn't be surprised if the slow load times cost them many millions.


Red Dead redemption has the same load time issue I bought that game for five bucks. Which was rockstar's cry for help in a dying online game.I tried it out for about 15 minutes realized it suffered the same issue turned it off and never turned it back on


Discussion about the findings two weeks ago:

https://news.ycombinator.com/item?id=26296339


Tell me that is not cool.. being able to say you had that much of an impact on million of players across the globe. Not to mention, bragging rights and cool achievement you can put on your resume ( that anyone should be able to understand ).


Not to mention the $10,000 bug bounty...


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: