Hacker News new | past | comments | ask | show | jobs | submit login
I cut GTA Online loading times (2021) (nee.lv)
476 points by ddtaylor on June 9, 2022 | hide | past | favorite | 213 comments



There’s an implicit story here which I find quite sad that not a single developer at Rockstar in the prior eight years had the autonomy or bandwidth to discover and fix this themselves.


I’m starting to slowly realize how terrible it is working in environments where people simply aren’t curious. It’s so hard for me as an individual contributor to decouple this way of thinking. Curiosity is how we find better ways of doing things or finding out hidden flows that can be better. Incurious people are a contagion, but I want to believe you can make people curious. It can’t be innate.

Every company talks about wanting to have driven curious employees to solve complex issues or challenges but you just get stomped with a boot to your face if you do anything outside your lane or simply ask why.

It’s also worse because I’m starting to realize how lucky I was to be raised as a curious individual from my parents who never went to college but at least were curious themselves about different things (pottery, soldering, gardening).

The quote “Curiosity is insubordination in its purest form” suddenly has a different context when working for a company.


I'd be careful about painting the developers at Rockstar as "not curious," that feels like an incredibly unfair conclusion to draw from this. Those developers have other pressures, and deadlines to hit. There are a million reasons why this bug wasn't solved by the team there. When a user complains about a bug in software you wrote, is it because you just "weren't curious" enough? Or is the situation more complicated?

Lots of incredibly talented people worked on GTA Online, it's worth trying to think through why a team of smart developers might not have prioritized this, rather than assuming something negative about them.


I wasn’t trying to blame engineers that wasn’t my intention, but someone needs to own establishing the environment that actively encourages or doesn’t encourage curiosity.

Talented people are found everywhere, but hopefully someone with authority over them doesn’t slowly poison the drive out of them.


Why should developers necessarily be trying to cut loading times? Perhaps GTA is phenomenally successful because they have good BAs and POs who are able to identify and convince engineers to work on what matters to the bottom line.


Screw this. Developers should advocate for their craft and for quality, and business metrics should hold an adversarial position to balance things out. We definitely should not be accepting the narrative that developers just do what they are told and don't push back on what we know is good for product - we are in a unique position of power, we should recognise this and use it for the betterment of the industry as a whole. The alternative is we let corporate greed slide our standards of quality into the gutter thereby making the job at hand miserable.


Very much agree with this.

That said given the quality of what they have put out, I'd surmise the issue here is morelikely they don't feel the same pain on load times?

Bandwidth too good, rigs too powerful, dev envs overpowered, etc. to enable developer velocity can often also mask real world problems.

Hell, even hot reload can mean you don't see load as often and are able to easily rationalize as "not that big a deal" - I wouldn't necessarily take it as a sign they are not deeply invested in their craft.


I very much agree with you as well actually.

One of the things that I've instituted at my work is our methodology for acquiring test phones for our product (WhatsApp for hospitals, basically). We go to a supermarket and buy the most eye-catching box of the budget phones available in the phone aisle.

We've squashed so many bugs and performance problems this way. It's cheaper, it instils empathy in product decisions, and our customers love the user experience as a result.

Similarly, one of my first big decisions after joining was to wean ourselves from our AWS addiction using Kubernetes. This also allowed us to design a dev env that fits on a modern machine comfortably, but can scale to entire Hospital Trusts in production. Once again: focusing on portability now means we can sell our product on-prem and hybrid, because we designed our developer UX this way.


I don't know, this seems like it was a pure CPU problem on good hardware. But anyway a developer who has 10 hour days and backlog stretching beyond the horizon will not have idle time to go try things out. It needs to be someone's explicit job to go hunt down things like this.

And given it was data-size dependent, I wouldn't be surprised this was not an issue at the start while someone who's job it was to look for this kind of issue was looking. And the issue appeared after they got retasked to something else.

In fact maybe this is the only kind of issue that you end up with after all is said and done. Just like in WW2 they reinforced the parts of the fuselage on returning fighter craft without bullet holes, because if a bullet hit there, the craft would not return.


> Developers should advocate for their craft and for quality, and business metrics should hold an adversarial position to balance things out.

What makes you think this isn't the case? Business metrics holding an adversarial position over other interests obviously doesn't preclude business metrics defeating other interests.


In my experience, business interests always override everything else. Often times to the detriment of the business, and often as soon as mere 3 to 6 months in the future.


Sounds like an extremely false dichotomy. There is also the solution that you set up your own company, or convince your manager your time is best spent on this.

I’ve worked around too many startups ruled by developers. They’re being run into the ground because those developers think technical NFRs as defined by them are the be all and end all. They’re not. Those same developers will happily spend months fiddling with the system without making any actual improvements and call it “paying down technical debt”.

Give me a PM (or good engineer) that understands the bigger picture every time.


> There is also the solution that you set up your own company, or convince your manager your time is best spent on this.

This is the definition of a flippant comment. Why should society require we become the lord of our own domain or a political chameleon just so that we can advocate for decent standards? Should people who live on flood plains "just move" when the tide comes in? Ridiculous.

Understanding the bigger picture includes understanding the developer's point of view.

> I’ve worked around too many startups ruled by developers. They’re being run into the ground because those developers think technical NFRs as defined by them are the be all and end all. They’re not. Those same developers will happily spend months fiddling with the system without making any actual improvements and call it “paying down technical debt”.

Sounds like those super-clever managers didn't understand what the hell was going on to me. Perhaps they should've held their nose and tried to learn about their product so they can understand and communicate that bigger picture they speak of instead of abdicating their adversarial role and leaving dev teams rudderless.


What makes you think they didn’t understand the developers’ point of view? Maybe they understood it, but didn’t agree with it.


> Give me a PM (or good engineer) that understands the bigger picture every time.

I was referring to this sentence: you're now telling us that the ideal good engineer understands the developer's point of view but doesn't agree with it. It sounds like you just want to be surrounded by yes men to me. The perfect recipe for a failed startup.


Maybe, just maybe, the developers are sometimes wrong about where they should spend their time.


Even if your parent comment presented false dichotomy, you responding with the other extreme isn't convincing.

As a senior programmer I respect business priorities and I don't go fixing technical debt for months.

That being said, we the people suck at moderation. See, the problem I faced was exactly the opposite of what you encountered: I've been too accommodating and allowed myself to be micro-managed to the point of a manager telling me how should I do my job; that's when it finally clicked with me on what a toxic workplace have I landed back then.

So neither of both extremes is good. I agree business priorities have to be respected; absolutely. At the same time we shouldn't allow the businessmen to tell us we can't ever fix technical debt.


Exactly. Managers are there to facilitate people and communication, business leaders are there to facilitate capital and strategic vision, engineers actually make the thing and master the detail in the process. Good businesses learn to listen: not just to their customers, but also to their domain experts.


This is why I try to hold developers accountable for actions their software performs or allows, like auto-banning accounts or not supporting billing limits.


Loading times were the #1 complaint from users about the game. I personally wouldn't play a game where 5 minute loading times were the norm, 10+ minute times are common, and 15+ minute times are not unheard of.

It's the #1 complaint. The thing that makes this egregious isn't so much that they didn't fix the #1 complain, the thing that makes it egregious is that it's a simple enough flaw that a mediocre developer could have just run WPR/WPA against it and see that 90% of those 5+ minutes was spent in strlen. Maybe it's not their area, maybe they don't know how to fix it, but they will recognize that it's a problem with a likely easy fix.

So even if we completely ignore the fact that it's the #1 complaint, it's still egregious that it's been allowed to go on for this long.


You've clearly never played it nor read any of the community discussions about the game. The #1 complaint is (or was) loading times. It's why I've personally stopped playing years ago.


I play GTA, am annoyed by those loading screens, and have found those forums while looking for solutions.

Companies are successful not because they delight every customer consistently, but because they do just enough to make the customer spend money with them.

After all, I will still buy the next version.


I mean that's fair, but I'm talking about wanting to be curious and encouraging environments where curious individuals can be lauded instead of dismissed. Not everyone on a team HAS to be curious but for those that aren't they shouldn't be smacked upside the head with corporate bureaucracy.


You are assuming curiosity was the problem instead of a lot of other possible factors like bandwidth.


I wasn't blaming this problem solely existing due to being incurious but more of lamenting how corporate environments don't encourage people to be curious.


That may have been your intention, but your statement of "environments where people aren't curious" comes across as putting the blame on the people themselves. As has been stated elsewhere, often times engineering is lacking in bandwidth to do anything but meet the deadlines.

I don't disagree that the "environment" itself could do more to further making improvements outside of the assigned work - initiatives such as 20% time, cafe days, etc. exist in other companies. But I don't put the blame on the curiosity of the people themselves.


I mean at the end of the day we’re products of our environments.

If SWEs can only work on features and do nothing else that is still a sign of poor project management. Just accepting that fate is worse for your growth I’d wager.

But idk, I never worked in an environment that encouraged curious people either. I suppose the closest for me was working at Comcast but that was only one manager with a small team of 4 people and no real deadlines.

I feel like two separate thoughts came from replies to my comment. One advocating for engineers to push back and care about their crafts, the other absolving the choices that were clearly made by people as some mechanized process that could never be changed.

The only thing I’m wondering is what the replies would be like 10 years ago on HN.


Game was out for almost EIGHT YEARS by the time this bug was fixed. There can be no reasonable explanation for them not fixing that. 5 minute load every time you try to go into multiplayer. The amount of revenue lost to that one single bug is staggering. This bug was a reason why I never played GTA online, for example.


Performance issues are one of those things that are easily waved off with broad explanations like "well there's a lot of assets to download", "their internet connection may be slow" etc. it may not have been clear to anyone at that stage that it was a "bug".


The underlying organizational problem then becomes the fact that performance isn't priority number 1. I would suspect that if the perf was taken seriously, some time spent on profiling would immediately reveal this issue.


Performance is not priority number 1. Number 1 is to ship the game on schedule so you can stop crunching for the first time in months, number 2 is to make an interesting game that people pay for and justifies your not getting laid off. The games industry is not a particularly safe place to be in from what I understand.


It is (probably) taken seriously, I am sure they were pushing things to limit to render just one more car or have a one more AI etc. My assumption the loading times just somehow slipped off and no one thought it was a solvable problem or it is just loading performance is not taken as seriously


> My assumption the loading times just somehow slipped off and no one thought it was a solvable problem or it is just loading performance is not taken as seriously

That does sounds like awfully like confirming that indeed no-one was curious, because they were too busy chasing KPIs or whatever other metrics management were using.


If was the number one complaint about the game. R* is one of the biggest studios around. It would have cost them nothing to just hire a developer whose only task is to hound down that bug and fix it.


It could've been waved off if there wasn't single player part of that same game that loaded few times faster.


That no one on the team maintaining the game looked into it likely means they don’t play the live game themselves. Developers and managers on up the chain.

Understandable given the heavy crunch it sounds like Rock Star works in. If you’re a developer spending 60+ hours a week working on it, you’re not going to play that in the few personal hours left over.


Your argument is a valid reason why this bug wasn't fixed when the game was first released. But no crunch lasts 8 years. And complaint about loading times were everywhere for all this time.


There are people that are paid to play the game (testers) or collect feedback from the people that plays it. Hard to imagine that they were oblivious to the problem


>The amount of revenue lost to that one single bug is staggering.

The players who are still there are used to it so the value is likely near zero. It's the sort of thing that's 100% worth fixing before launch or soon after so you don't lose players in the first place. But the fix loses value every day it goes unfixed because you're already lost those load time-sensitive players. Keep in mind that years after release it still makes Rockstar/Take-Two millions per day (https://www.thegamer.com/gta-5-rockstar-2-5-million-per-day/) so I guess a lot of players just don't care.

For the record: I'm surprised anyone is willing to tolerate that shit.


Lots of people cite long load times as why they stopped playing GTA5. They are certainly making bank, but they also certainly left money on the table because those players who left could have still been there and spending money.


Them making a lot of money doesn't prove that they couldn't have made more money. Everyone knows better performance means better conversion which means more $$$.


It is definitely non-zero, I think since the launch of game the number of revenue lost to this single bug is in tens, possibly hundreds of millions of dollars. Considering that game made somewhere around 10 billions USD, 1-2% of total revenue lost to loading time seems plausible, given how horrible the problem was.


GTA Online player count statistics indicate 2.5 million GTA Online players every month. A reduction in total waiting time for all the players of 18.5 years every month.


That's assuming each player only logs once in a month, which is definitely not the case.


I was trying to make the fact nobody decided to fix it for years, less outrageous that it is...


I agree with the sentiment. However, it is a tad surprising when you consider the productivity hit, even just internally speaking, that ~5min of cpu time in strlen() on every start of the app incurs.

How much time did perfectly qualified engineers, who could have attached a profiler at any moment, spend sitting at a loading screen spending most of its time in unnecessary strlen() calls? I understand we see this through 20/20 hindsight, serendipity can work against you, etc., but that's a crazy irony.


A toxic culture resistant to change can obtain these results no matter how brilliant/skilled the people.


It's true, but also, you can get this result with a competent team and good culture, curious people, etc. You could just have bad luck and nobody happened to check it.


You might say that assuming this was due to inherent incuriousity amongst the developers is quite incurious.


it was a feature - they served ads during loading screens


+1 to the other replies here, I think it's very poor form to throw the engineers under the bus for not being curious. The most likely explanation is they were under insane time and resource constraints, especially in the games industry. They wrote something that worked, then were forced to move on to the other 5000 tasks in their backlog that had to get done before release. When you're already being told to crunch 80+ hour weeks for months or years at a time, you run out of gas.

I guess it's possible that the engineers who were talented enough to create this extremely impressive game in the first place somehow threw up their hands and said "guess it can never take less than 7 minutes to load, best we can do". But it seems much more likely that they were well aware, but never allowed by poor management to go back and fix it. The game was released and printing money, why bother. They're on to the next title.


The problem with this way of thinking is that it basically absolves everyone of blame. Bad stuff happens and nobody's at fault. You point to the managers, because the engineers were so busy, but the managers are going to claim they were busy too. Again, from the manager's perspective - the managers aren't even in a position to understand the problem or the reasons for it.


Having been on both sides of this (as a developer and later a senior manager with product/feature responsibility), I'd say it can be both. A really good manager/executive has enough knowledge and experience to ask the right questions and parse the answers even when they are technical. Sadly, many managers aren't that good (at least yet).

On the other side, a really good developer has the knowledge and experience to frame the problem in an understandable business context when communicating the issue and options to management. Some of the best work experiences I've ever had were in orgs with such experienced developers and managers. Working together we were often able to solve apparently intractable problems with creative options which would never have been conceived absent the clarity of thought and communication on both sides.


This thinking implies nothing is worth doing unless it has a clearly articulable direct business value whereas a lot of technical debt simply doesn't. When the only people affected by a problem are people who have already spent their money or people who have no autonomy it's unlikely the decision makers will decide to investigate and fix it. If you're lucky you can convince decision makers that there is an impact to hiring or efficiency but in my experience these arguments are extremely challenging to make in the face of other stuff that decision makers feel directly impact business, and for good reason. I'd say this is a big reason autonomy is crucial when it comes to cultivating a curious team because without autonomy you are limited to things that can achieve consensus in an environment where people inherently have different interests to attend to.

I feel like the "20% time" paradigm where one day out of the week is left up to the developer to schedule (within reason) is the right way to balance the conflicts of interest and still leave room for people to be curious.


"the managers aren't even in a position to understand the problem or the reasons for it." Which is again the fault of management. It is the responsibility of management to facilitate proper management.


As a shareholder, I’d be rather happy with the management of GTA - it’s phenomenally profitable. That sounds like proper management to me.


You're pretty quick to act like there is a lot of blame to be allocated, but GTA 5 is arguably the most successful game in history. With games that truly have technical problems so bad they ruin the game, people don't complain - they just don't play it.

Maybe a long load time just wasn't that important for the success of GTA.


It wasn’t my intention to blame engineers but I agree with the other reply to you, someone needs to be responsible for creating these environments. It’s not like we’re dealing with fundamental forces of nature here, this is all self afflicted.


Don't blame the people, blame the environment.

People might start off curious but eventually they keep running into political roadblocks and other nonsense, which slowly beats the interest out of them.

If those are the kinds of people that continue surviving then those are the kinds of people which your environment cultivates.


That’s totally fair, I honestly meant to blame the environment and not people. It just sucks to see the pattern of curiosity get stamped out of people due to environmental changes that aren’t obvious.


Discussion from today: "Do you know if feature X works? A customer is asking about it" "No, it was disabled because of a bug a while back" "Hmm, found the ticket, it's from 4 years ago!" "Yes, basically it's a bug from a new version of a 3rd party software that made it unusable. At that point it was basically decided that we will not try to find a workaround, we just tell the customer that it's a 3rd party issue."

The funny thing is, all the people who made that decision left the company/project and now we have the option to change it. Can't wait to see what happens next, if the new leadership changes that or not (my money is on "not").

The other funny thing is, I can think of at least two workarounds that only need a day of work and a couple for testing it.


Sadly at Rockstar it's likely that employees are curious, but are under so much pressure for deadlines, and working such long hours, that they don't have time to follow their curiosity.


When problems are this obvious, it's surprising that someone doesn't have a look, even if they're busy. Maybe I'm asking too much/projecting here, but I just don't understand how people put up with fairly obvious semi-breakage as much as they do.

I have this at my current job, I seem to be the only one (not really, but seems like it sometimes) interested in finding out why things are slow/broken and investing the time to fix it. If you're "busy" all the time, but put up with things that slow you down, you're hamstringing yourself by not fixing the brokenness. When only a small number of people take the time to fix things (typically other's issues), that becomes very demotivating.

That would be like being on an all-day ski tour, and feeling too rushed to bother getting rid of the slush on your skis. Yes, you feel rushed because you're moving slower that usual. Taking 4 minutes to kick the slush off will make you much faster on the overall trip. (Sharpening the saw analogy, etc).

Sometimes "busy" is a state of mind, more than it is a reflection of reality. I know each situation is unique, but I've seen this enough times to understand that it's often a form of dysfunction rather than a reasonable reaction to the work situation. This is like when people used to say "we're too busy to write tests". Well, maybe you need to get better at communicating the trade-offs to the broader business, and be a bit more assertive about non-functional aspects that are still important to address.


From reading a lot of comments on HN, I think you might be in a semi-privileged position if you have enough time left over (and/or are allowed by management) to find slow/broken code and then are allowed to even commit code with doesn't have a JIRA ticket (created by someone else) attached :)

Dysfunctional... possibly true but isn't that (unfortunately) more or less the norm in IT? Not to say it shouldn't be fixed but at the end of the day you have 'x' time to ship 'y' features in order get paid 'z' dollars so that the company can continue to survive... combined with "management" often under-estimating with respect to time, requirements and (un)foreseen hurdles.


> Maybe I'm asking too much/projecting here, but I just don't understand how people put up with fairly obvious semi-breakage as much as they do.

> I have this at my current job, I seem to be the only one (not really, but seems like it sometimes) interested in finding out why things are slow/broken and investing the time to fix it.

You will quickly shed yourself of this philosophy if you become a business owner. You can test those waters by releasing a non-trivial open source project that gains a non-trivial amount of users.

There will always be at least 100 things that are broken in some way but not impacting the core business. You'll have to pick only 1 to work on. And, if you're unlucky, some frustrated customer may write a blog post about issue #65. An issue that you're fully aware of but haven't prioritized.

Guess what you'll do as a business owner? You'll prioritize that issue since it's now very visible and maybe even throw $10,000 at the customer who took time to do their research; as is what happened here.

> This is like when people used to say "we're too busy to write tests". Well, maybe you need to get better at communicating the trade-offs to the broader business, and be a bit more assertive about non-functional aspects that are still important to address.

Or perhaps the need for tests was communicated and the business made the decision to not prioritize it.

You may not like that decision.

Mind that the decision to "not write tests" is never explicitly stated so simply. No one wants to look bad. Instead that decision is expressed through minimizing its level of priority compared to other things.

Still don't like the decision? Well you have three options:

1) Ignore the stated goals of the company and do what you think is right instead. In which case you're acting against the interests of the company that's paying you.

2) Sacrifice your own free time in order to provide your idea of value to the company (that may actually be valuable! but that the company doesn't value enough to prioritize)

3) Do what the company asks with plausible deniability. Also known as kicking the can. Also known as not my problem.

Sometimes doing #1 or #2 could greatly benefit you. Perhaps that thing you're doing (eg: writing tests) will somehow measurably improve the business in the future WITH PROOF so that you'll get get recognized. Or perhaps there's management shake-up and the new leadership perfectly aligns with your philosophy, etc, etc.

But that's rare.

More often than not your attempt at fixing the company's problems behind the scenes will go unnoticed and unrewarded. And if you do get rewarded, often times it's a pittance compared to the amount of time you invested.

My personal philosophy is I simply do not work for companies that don't align with what I value. For example, to me, it's exhausting and unrewarding to work at a company that doesn't appreciate a GREAT user experience.


Why are they working so hard on an 8 year old game? Aren't they just basically producing content packs now? Doesn't even seem like the playerbase cares when there are gamebreaking bugs ongoing for months.


Rockstar doesn't publish new games very often, so if they have deadlines they're internally-set ones.


Curiosity has to come from the top and fall all the way down. As others and you have noted in other comments, its the environment for the workers that sets up whether or not they will even be able to pursue curious things, and that's on the managers to create that environent. However, the managers themselves have bossess too, so that has to trickle up, to their bosses bosses all the way up the chain.

In short, every job needs to be able to have autonomy to enact on the ground solutions for what their immediate team needs. You shouldn't be penalized for going off the reservation and coming back with a creative solution. The problem is corporate structure for the past few decades has been about rooting out this autonomy and anything opaque about job responsibilities. There is a lot that's been done that would need undoing to change those environments, and imo you are better off finding another workplace than bringing about the monumental change required. A mature corporation is like a boulder tumbling down a mountainside; you can hardly change its course much less stop it short of where its heading.


> I want to believe you can make people curious

In my experience it's the opposite. Most people are born innately curious. It takes intense systemic conditioning to suppress their natural curiosity. For many of us that conditioning begins in the traditional education system and is completed in the workplace.


>It takes intense systemic conditioning to suppress their natural curiosity.

Yep, the public school system in America excels at squelching curiosity and any desire to learn.


It's not a particularly American phenomenon.

I can remember reading "Deschooling Society" by Ivan Illich whilst in the UK's comprehensive school system.

On retrospect, I don't think I realised the importance of schools facilitating socialisation, rather than just learning.


Just to be clear, you are saying, matter of factly, that the whole point of the American education system, is to suppress free thought.

You better tell Congress, the last thing they want is for America to fall into totalitarianism.


John Taylor Gatto was an award-winning public schoolteacher who more-or-less agreed with this sentiment.

https://en.wikipedia.org/wiki/John_Taylor_Gatto


How are you incapable of understand sarcasm? In fact you're arguing from Appeal to Authority now. I just, I don't understand. The weird thing is that your argument is extremely convincing at first glance even though the entirety of your argument is in a link to wikipedia?


Sharing knowledge and understanding doesn't have to be an argument.


Just what do you expect my response is going to be to that? I certainly don't understand anything more now. You bring up someone obscure and state he is representative of what I could conclude is all teachers urging for reform of our education system? Maybe after reading the previous arguments by COMPLETELY SEPERATE PEOPLE I have been mislead.

Not to be personal or anything, but I went to public school.


I found this on the ground, did you drop it? </s>


It is not that developers arent curious. They probably are in their cages set by bad project managers who dont even play own game and dont notice it takes forever to load.

The project managers could put load times as a priority but they genuinely dont see it as a problem or dont care at all. It happens all the time. There is a lot of unoptimized software.

Perhaps the issue comes from the top: the project managers are supposed to only do things that increase revenue and all other things are not rewarded at all? Perhaps executives only care if they hit quartely results? (Short term gain traded for long term loss - the game is made for whales who are so invested that they will play even when it is technically bad - so milk them now with new content for whales and ignore everththing else, who cares that new players will be discouraged by bad load times).

In my opinion if nobody plays own game / eats own dogfood, they dont even see its problems. Blizzard is at this stage now - some of their games barely work. You buy Starcraft 1 through the launcher... and it doesnt work(I think it works now, but it took them 1 month to notice it and another to fix it). Heroes of the storm - downloaded a 137mb patch every run - it took them half a year to notice it and do something with it. It is like nobody cares about the brand at all. Game company where your games dont work.

IMHO a problem with people - that comes from the top. They want only story points for things that bring money. The project managers either dont care, or cannot deal with other problems at all. I bet they dont care. This happens in many companies - there were few people who tried, they got burned for that: terminated or quit -> so now nobody cares. "The game is good enough to still sell some skins to the whales, so why bother".

Look at the results: they got an external comment what to fix. Did they launch their own internal project to optimize the game? Did they check for other things that make it load slow? It looka that someone else did a partial fix that is "good enough" so everything is back to usual and nobody cares about load times.


I'd never heard that Nabakov quote, but it's terrifically cynical and, I hope, false but, in my experience, not entirely untrue.

Designers' unnecessary desire to delight users staves off the more Kafkaesque user experiences, and we all live in the gap of cops' unnecessary (occasional) sympathy for (some) nominal outlaws (we're all nominal outlaws).[0]

When you mentioned curiosity I thought of a firm where colleagues scoffed at me as I rummaged through conference room drawers, saying, "you're not going to find anything interesting in there." I'm curious about cupboards as well as log traces -- an almost ubiquitously "eyes-glaze-over" boring artifact of modernity that, alongside popularly mind-numbing "code", I am equipped to find interesting.

Is it any wonder, then, in this workplace where such innocent, cheap, curiosity were scoffed at, I was not too long after threatened with the extraordinarily archaic (in our at-will state, at least) legal notion of "insubordination"?

It's a shame, because insofar as "all ambiguity is resolved by actions of practitioners at the sharp end of the system,"[1] the liberated curiosity of individual contributors is often the only "crack in everything" that's "how the light gets in."[2]

0. "With two lines of a man's handwriting, an accusation could be made against the most innocent." (Françoise Bertaut de Motteville)

1. https://how.complexsystems.fail/

2. "Anthem" - Leonard Cohen


In gamedev where crunch is the default and people are routinely working for 10, 12 or more hours a day, sometimes including weekends, sometimes for months on end - it's not very kind to assume it's curiosity that they lack.


Yes but death marches are typically a sign of bad project management IME. As for lack of curiosity, the environment obviously plays a huge role in what is encouraged.

I feel like history is repeating itself here (at least in regards to death marches):

https://en.wikipedia.org/wiki/Erin_Hoffman#%22EA_Spouse%22_b...


>Every company talks about wanting to have driven curious employees to solve complex issues or challenges but you just get stomped with a boot to your face if you do anything outside your lane or simply ask why.

That's because "curiosity" translates to "more work" at #BigCo. And people work at #BigCo so that they can comfortably do as little work as necessary to maintain their lifestyles. This is a hard rule for any organization. So if you want what you're describing, it will only be found at small startups where everyone has skin in the game.


My brother is the polar opposite of me when it comes to curiosity.

I ALWAYS ask questions like "How did this come to be, How much did this cost, who designed this and why" etc.

My brother gets annoyed that A) I ask and question everything, and B) Laments he doesnt know as much as me.

However, this is spurred by my ADHD... and I at times wish to be able to have a more precise control of my focus.


I strongly believe that R&D teams should have some time reserved to "do what you want" (limited to product improvement, obviously)

Perhaps this might not be appropriate in the gaming industry... and who knows if Rockstar don't already have this.

However, the GTA loading time is horrendous (and I had a spectrum sinclair when I was a kid) - surely there must have been a work item, ticket, bug reports, or whatever, for this.


I work in a company where that can happen. In fact we had just a case today. For the last 20 years, our ever growing C# monster application used across the world to make millions, literally, had random crashes. Comes with volume spike, usage spike, we never quite knew: everything would display a red cross, the gdi handlers would reach 10k and bam.

For 19 year we just looked at it in defeat. For the last year, bit on and off. Starting last monday it crashed every morning for a user sitting next to me, a big nasty german boss (Im French).

I sat down, read his logs, saw it failed on ImageList creation again for some fucking reason, looked how many of these things we used and what they did. 2 hours later this 20 yo mystery bug dozens of devs had noped out of was fixed. We fully get it now. Wasnt even a matter of priority, it took 2 hours of looking at it. A lunch break basically.

I dont have a sort of morale of the story or whatnot, but well, Rockstar would have figured it out eventually. Just need a nasty fucker next to you whining abt it everyday.


2 hours lunch are you guys hiring?


Yes, desperately even, but if you join just for the lunch break (where we fix bugs as you saw lol, I haven't eaten during a lunch break for the last few years), it's a bit of a red flag :p

But seriously, look at tier-2 investment banks: we're nicer and less stressed than the Goldmans of the world but still make more money than most. And problems are interesting I find. But you need to like handling money and not "changing the world" in any shape or form.


There's a attitude, arguably backed by sales, that load times don't really matter.

All of the Source games (HL2, Portal, Alyx, etc...) have arguably bad load times. They all sold great! GTA sold great even with it's previous load times.

It's just not a priority for most teams. In Rockstar's case it happened to be easy to fix but in most cases it's not so easy because the engineering team didn't want to deal with it.

AFAIK, Source, and maybe Unreal/Unity are built in such a way that by default, if you need to start the level over you have to re-load it entirely. They have mutable state spread all over the place and have no way to mark what's important (needs to be saved, what's not (can be zeroed), and/or they haven't bother to put all mutable state in a single places so either it can be restored from memory or loaded by just loading a tiny bit.

There's zero reason HL:Alyx should take 15-30 seconds to restart a level you're already in when clearly less than 1k of state has changed since you started the level. But, fixing that would require a major re-design of the Source engine. Slow load times don't affect sales. So almost no one does it. Easy just to let each gameplay programmer do whatever they want and to restart just reload the level.

I put up with it for some games (the ones mentioned above for example). But I've also quit a few recently. Games where you die a lot and to add to the frustration it takes 1-2 minutes to load again only to get killed in 5-20 seconds. That frustration means trial and erroring to figure out what to do gets a punch in the face and so I stop playing what might have been a good game otherwise.


I have a solid 2-3 years worth of work assigned to me in my issue tracker, with new issues rolling in every day.

All I can do is prioritize and hope nobody gets too unhappy that their pet features is never going to happen.


where I work at, it is completely discouraged to do anything out of what was asked you to do. so I don't judge the workers/slaves. It's not allowed to be creative/careful in most places if it's not harming the bosse's wallets.


In order for a worker to bother with this

1) It has to be alright to make miscellaneous changes like this, and

2) It has to contribute to getting promoted.

The second point is IMO why random QoL changes like this are rare in large corporations.


I don't know that #2 is universally true. I am at a point in my career now where I will be able to comfortably retire without ever being promoted again (i.,e. CoL raises only). Judging by the salary ranges I see on offer for jobs for people with 5+ years of experience, this can't be that rare.

When I see a pain point that looks soluble, I take a crack at it. Sometimes it works, sometimes it doesn't. Sometimes I get recognized for it, sometimes I don't. One, relatively minor (to me) thing, ended up being a big business win and my manager made sure I got a fat bonus for it. Other things have seemed more like a win for me that nobody but 2 of my peers ended up caring about.

#1 is true to the nth power though. I've seen places where if you do something even one millimeter outside of your silo, it will come up as a negative in your performance review. As in, if you write a 5 line perl script that makes you significantly more productive, do not share it as you will be asked time and time again why you wasted your time on something outside of your job description.

I think something missing for the context of TFA is that the gaming industry tends to be much more deadline focused than other industries. In theory you might be free to "make miscellaneous changes like this" but only if things aren't already behind schedule, and things are always behind schedule.


> As in, if you write a 5 line perl script that makes you significantly more productive, do not share it as you will be asked time and time again why you wasted your time on something outside of your job description.

This sounds like something I'd see in a Dilbert comic. Do you mind sharing what industry you work in? Perhaps you saw this in old-fashioned industries like banking or insurance? Have you experienced this in tech?


This specific story was a friend working for a large defense contractor. I've never worked directly for a defense contractor, but know enough people that have that I should note that culture varies greatly from site to site; different sites of very large companies might as well be different companies (and in many cases were different companies before being purchased).

That sort of thing also varies with perceived team performance. If your team is perceived by upper mid-management (or higher) as being "good" then individuals on the team have more freedom. If your team is perceived as being "not good" then every deviation from standard practices is clearly the cause of the poor team performance.


I've had this situation fluctuate at my workplace. When misc changes are supported, and those improvements are actually valued by management, I've seen the system improve dramatically.

I've also seen a single-minded focus on pushing cards across a board, which seems to typically correlate with higher levels of a dogmatic SCRUM process.

Pushing cards is great, but not when it prevents the various little things that have to happen to cultivate quality software. Especially not when there's a gatekeeper prioritizing these things into oblivion.


#2: Just do your assigned task well, that's it.


This was discussed a bit in the original post [1] but I believe this comes down to the problem being sufficiently deep in the stack that there was an assumption that this wasn't something that could be improved on. I'm sure the loading code looked good.

It took someone who didn't know what that code looked like being curious about what was actually happening.

1: https://news.ycombinator.com/item?id=26296339


Shows that dogfooding is one of the best thing you can do for your product. I remember reading a story about Valve when they were working on Dota 2, I think Gabe said that every day after work they would play for hours and couldn't stop.


If I recall correctly the Dev team didn’t notice this as they were using a test JSON which was not very complicated. Only when the game was in production did the complex JSON cause the long loading times


To be fair, the person that found the issue was persistent and did some serious digging to figure out what was going on.

I imagine that even if you were on the GTA team, you'd most likely think the long loading time was unavoidable due to some complexity you were not aware of.

Great work to solve it from outside of the company.


Serious digging was needed because he does not have the source. Any devs could easily profile the game during loading and that sscanf or strlen would be at top

Maybe they saw sscanf and assumed it is normal that it takes time, but more likely there were so much stuff to do and somehow no one checked


You would think that the more time people spend playing the game, the more money the game makes. I would think online loading metrics would be a similar problem for whomever is responsible for input latency.


I'm not surprised really. Given how big a product like this is, how many people would actually have worked on this section of code itself.


The most likely explanation is they never used this launcher themselves. That applies to QA department as well.


nor the incentive


Another take:

Dev 1: Hey, I think we can improve the load time by fixing X, Y, and Z.

Dev 2: that's neat, create a ticket so we can keep track of that.

Product 1: hey, what's this ticket about?

Dev 1: oh, that's to speed up the load time. It's a quick fix.

Product 1: was that in the requirements?

Dev 1: No, but it will be a huge improvement.

Product 1: ok, I'll mark it as Tech debt, and put it in the back log.

~~ Many Month Later ~~

Dev 3: hey, did you know we can improve load time by updating X, Y, and Z?

Dev 2: Oh yeah, we even have a ticket for that. Let's let Product know it's still an issue and maybe they'll prioritize it.

Product 2: Was this part of the requirement?

~~ years later ~~

Product 3: hey, there's this old ticket here. Do you think it's still relevant?

Dev 35: I'm not familiar with that, but it's too old, most likely not relevant any more. We can delete it.

....

Dev N: Hey, I think we can improve the load time by fixing X, Y, and Z....


That's why you stop asking permission to make quick fixes and just submit the PR. Obviously there is a balance here but if you can't "steal" a few hours in a week from the product driven roadmap to do so then there is something wrong in the engineering culture.


Absolutely. If something is oddly slow, I run the profiler to see what's going on. If the profiler shows me something boneheaded like this that is a big win to fix, I fix it. Obviously I make sure the tests still pass, but then I check it in. If someone is going to give me shit for improving the product, that is not somewhere I want to work.


The boy scout "leave it better than you found it" mantra has always served me well. Hell, sometimes that small fix can lead you down a rabbit hole that uncovers fixes to larger problems that have been plaguing an app for an eternity.


What of those odd stories where somewhere else in the code another dev realized the segment of code was doing strange things and create a work around that now breaks because of the cowboy anything goes attitude changes? Usually with comments talking about it "being very weird to do this, but it works"


If the tests dont fail thats not my problem.


Do any orgs explicitly ask the engineers themselves decide how to allocate a certain percentage of story points in a given sprint?

Product people are great for selecting which features to build out and which bugs to address, but are poorly equipped to decide how to prioritize things like: performance optimizations, bugfixes that haven't (yet) been reported by the end users, code refactoring, paying down tech debt, modernizing tooling, misc quality of life improvements for the engineering team, etc.


My last org did exactly that. The team lead that runs sprint planning was always an engineer. Product management is still there for direction and context, but the decision maker was always an engineer. I don't think this is particularly special way to do it, but I might be wrong.


My first job worked like that. We explicitly had a 25% stream of technical improvements. It worked pretty well.


Same. When I was a team lead I always pushed for 30% of all story points being for maintenance / tech debt that product couldn't touch.


Ask for forgiveness not for permission. Whole heartedly agree.


>That's why you stop asking permission to make quick fixes and just submit the PR.

If the company pays me to close tickets and Jira stories, why should I care about improving its products?

If the company pays me to actually improve its products, then sure, I would do that.


Because the existence of the problem bothers you implicitly. Or because it’s usually a good way to a promo


Usually for perf issues like this, you and your colleagues also suffer. It is not just the time you lose waiting, but you also drop out of flow/get distracted and your productivity suffers more. The end user gains are just a side benefit.


The solution is for ICs to stick their necks out, go over the product manager's head, and ignore the org chart? Not get a new product manager who cares about the product?


The product manager shouldnt be the boss of an engineer. Thats what engineering managers are for. Product / Eng managers work together to set goals of what they want done, it shouldnt include how they want it done, and exclude potential side fixes. Thats how you kill the morale of an engineering team.


I agree with Tom: a well functioning eng org allows engineers to fix critical bugs without going through a bureaucracy. This empowers the engineers and makes them a product owner as well, though you need good engineers who care.

If you force every little bug to go through the product manager, you take time away from the PM to do big picture things by forcing them to track every little thing that's broken, and create a lot of friction for even the littlest bug to be squashed. This fosters an eng culture of "not my problem," which leads to bugs slipping through the cracks for years.


I tried playing GTA online once and actually gave up due to the loading times (on a top end gaming rig)

I can't believe I'm alone, I wonder how much this cost rockstar


Think about how many human lifetimes worth of extra minutes were wasted from everyone waiting for this x number of players X years this has been a problem. Even if you can’t implement bubble sort on a whiteboard in your sleep, 5+ minute load times should have been setting off alarm bells in many engineers heads. I’m sure this did actually cost R* real money. Bounce rates for websites increase 32% for 1s vs 3s load times, according to Google: https://www.thinkwithgoogle.com/marketing-strategies/app-and... Obviously we don’t have public data on it but I’m quite sure that some at least some people got frustrated with waiting and just stopped playing.


If you think about it, those lives actually were saved. People either didn't play the video game, or they spent a moment in the loading screen in quiet contemplation.


To play devil's advocate we should balance those out with the impatient gamers who acted out aggressively (e.g. the 100th time proceeded to kick a PC, smaller family member, throwing a mouse across the room, etc) or consumed more unhealthy snacks/drinks :(

I personally was often frustrated waiting several minutes, thankfully I don't have any violent tendencies, snacks on the other hand... :/


We are in the era where its impossible to be bored. In the elevator? Phone out on instagram. Loading screen? Same thing. Instant dopamine the minute you need a drip. The playerbase clearly doesn't care. To be honest, these loading screen times are as bad as they've always been for console games (especially considering it was only a generation ago that they ran off a disc).


From complaints I saw online, users emphatically cared. It was a common complaint, and some users would report that the load times would deter them from playing the game.



Not much, the GTA5 online scene is alive and well. Considering they've sold this game over the course of three generations makes me believe they've made a mega flipton of money off this game.


> they've made a mega flipton of money off this game.

It can be simultaneously true that they made a lot off the game && with lower loading times they could have made even more. In-game microtransaction ARPU scales with playtime. >4 min per session spent waiting to load instead of playing == lost rev.


I know so many people that have stopped playing due to the loading times. Just because a scene is active and alive, doesn't mean it couldn't be even more active and alive with such a minor fix that could be implemented within a day.

I am pretty sure they lost >100.000.000$ by that. Just shows how shitty the management over there is that they never tried fixing that problem.


By the time you get to the loading screen they've already profited. The fact that they continue to sell the same game over and over means individuals have likely purchased the game multiple times meaning they don't care about loading times, they just like the game.

It's easy to say their management is shitty from the comfort of your computer, but the reality is Rockstar produces great games, and they're making a ton of money because there games are good. I find most multiplayer games have a start up cost, glad they shortened it, but I don't think it's stopped anyone from playing that really wants to play.

https://www.thegamer.com/gta-5-rockstar-2-5-million-per-day/ https://www.tweaktown.com/news/80912/grand-theft-auto-made-o...


My statement is mostly based on the assumption that a large part of their revenue is based on microtransactions, for which satisfied customers are key. Therefore if I stop playing due to the loading times, they effectively loose money since I certainly won't buy any in-game money with my real money. But you are of course right that I have already bought the game.

Regarding the management: If it takes >5 minutes on top high-end gaming rigs to load a game and if your loading times have become a meme in the internet, I think a somewhat reasonable product manager should ask their developers to at least invest 5 minutes in figuring out why it takes so long (that's probably about the time needed to figure it out given the tools the developers have) and if it can be decreased in a reasonable amount of time. I was just amazed by how simple the problem actually was for an issue that has led me and many others to stop playing. (And I actually really wanted to play, but I just didn't want to look 15 minutes at loading screens for having 5 minutes of actual playtime.)


The action is, as always, on the margins. Those people who kind of wanted to play but got frustrated at the load times, you want them. In some senses you care more about them, the people you can gain or lose with changes, than your more committed users who are a little bit locked in.

GTA V has sold, and continues to sell, amazingly well. But Online, where this slowdown was exclusive to, is the real moneymaking engine. Any little marginal push to keep people out of Online, from feeling they could just jump in whenever, was significant.

The real baffler to me is just thinking about all the time spent by people internal to Rockstar sitting on that same screen.


My exact same experience. As to what it cost, in the old world paradigm I'd say "nothing, we already bought the game", but I guess these days there's probably a decent argument for a cohort of people who are too impatient to wait to get into the game and who pay extravagant sums of real-world money to get in-game items immediately.


There's other costs: how many times per day to R* devs & testers have to go through the same loading? How many man-hours wasted? How many compute resources wasted? How many upgrades were justified by "I need better hardware to get my job done in a timely manner"?


yes, GTA online's business model is making you buy in-game currency and paid DLC

which is kinda hard if the game takes 30 minutes to load


Same. Asking around why players tolerate it, the anecdotal conclusion I’ve got is that what makes these frequent and extremely slow loads tolerable is playing with friends on audio chat.


The same thing happened to me. GTAO earns billions/year, crazy that "fix the insane loading times" didn't bubble up to their top-priority.


Probably nothing. Loading times in rockstar games on consoles have always been this bad. GTA4 was just as bad too.


You're probably wrong. I remember e-commerces trying to quantify how many millions of dollars they would lose when clicking would take too much time on their websites.

EDIT: things like https://www.fastcompany.com/1825005/how-one-second-could-cos...

> Amazon’s calculated that a page load slowdown of just one second could cost it $1.6 billion in sales each year. Google has calculated that by slowing its search results by just four tenths of a second they could lose 8 million searches per day


That's ecommerce. This is a video game and a unique one at that, so there are other factors at play. I don't think many people go "game took forever to load, guess I'll play for less time then." There could even be a bullwhip effect, a long load time might make you want to stay playing for a longer period of time.


the load times were soooooo bad. Insultingly bad. It doesnt seem like you understand the true extent to how horrible rockstar managed this issue. I definitely quit playing for exactly this reason, even at the expense of spending less time with a good friend who i played with. Rockstar is a garbage disrespectful company for allowing this to go on for so long.


> I don't think many people go "game took forever to load, guess I'll play for less time then."

Ive quit some online games that take too long to load. Matchmaking times that are too long make me quit games.


From personal experience I can't disagree more with you. I tend to play video games that have less loading time and tutorials because I just don't have the time to play and when I want to play I don't want to sit and wait for what will be mostly a 10-20min session.


Previous submission (as linked in tfa) https://news.ycombinator.com/item?id=26296339 (Feb 2021, 699 comments, 3883 points)


I cannot fathom how dysfunctional Rockstar must be that this slipped through the cracks for so long given the fact that GTA Online is Rockstar’s cash cow.

6 minutes is insanity and must have turned off so many people from playing.. this person deserved far more than $10k


Because the user experience isn't important.

Enough still play it and buy a few micro tractions per month regardless of the load times Dev time is better spent reskinning a hat or something.


You are probably being sarcastic, or cynical about current corporate business world where nobody cares.

But user experience is important. With faster lpad timea they probably could get more players. More players probably means more money, since there are more chances to convert someone from a player to a paying customer.

Perhaps if someone can endure a 6 minute load time they are more likely to buy something in the game (a fan of the series?), but we could say that those who like fast load times also use their wallets fast. But it is pure speculation - we dont have the data. Also obviously Rockstar doesnt have this data - they couldnt A/B test if load time converts to more sales - their game only had bad load times.


Certainly there has to be some portion of the potential audience who would have become microtransaction purchasers who didn't because they were turned off by the long load times.


Clickbait title ;)

The problem wasn't strlen(), that just showed up at the bottom of the call stack when profiling.

IIRC the actual problem was an exceptionally dumb JSON parser combined with parsing a pretty big JSON file (which probably also grew much bigger than the original developer working on that feature anticipated).

Edit: the title has been edited in the meantime, originally it was something about strlen() being responsible for the slow loading time.


The JSON parser was not "exceptionally dumb": it just used sscanf(s, "%d", ...) for parsing numbers. But most implementations of C's standard library (yes, including BSD and Linux's) implement sscanf() by converting the source string into a dummy FILE* object (which involves calling strlen() to set the stream's size) and calling fscanf() on it. Apparently most people who use sscanf() are unaware of this "lovely" implementation quirk and I can't really blame them.


Is C's std impl slow because of calling strlen() or due to converting into a dummy FILE* object or due to calling fscanf?


All of the above, I guess.

It wants to reuse code in fscanf(), but for that it needs a FILE * interface. But to create that, it needs to implement EOF-handling, and to do that it needs to provide a size, so it has to call strlen().

I had no idea. :|


strlen() was a major component of the slowdown. Yes, the JSON file was large and strlen() was indirectly caused by sscanf() being used. But it's not accurate to say it's a clickbait title when a bulk of the slowdown was caused by constantly recalculating the length of the C string. The article shows in detail how he created a monkey-patched strlen() that cached the length to avoid recalculating it over and over which was attributed to the speedup.

The speedup gains were not caused by reducing the size of the JSON file or doing any exceptionally better JSON parsing. The speedups were a direct result of simply not calling strlen() over and over in an O(N^2) fashion.


All true, but replacing the strlen() with a caching version is just fixing the symptoms (of calling strlen too often... but the only realistic solution without the ability to recompile the game - so kudos to the author to come up with this hack).

I bet the "official" patch by Rockstar simply dropped in a less embarassing JSON parser library :)


You don't think that it's a bigger WTF that sscanf implementation calls strlen when parsing a number (when it iirc even ignores any non-digit characters for the actual number parsing so no reason it couldn't terminate that part w/o the extra call)?


sscanf is essentially being used as a tokenizer for the JSON grammar, which means something like this:

    {"foo": "bar"}
Is potentially turning into 5 different tokens:

    {
    "foo"
    :
    "bar"
    }
After each token the "state" of sscanf is being discarded and each invocation of sscanf is independent. The API simply isn't descriptive enough to know that a string is immutable and hasn't been changed and there are certainly many valid use cases where one might be re-using the same char[] buffer with different contents.


Even with the cache, calculating strlen() just once was already a waste.

The JSON to be parsed was read in from somewhere, either a file or received over the network. When that happened the machine knew how long it was.

One of the few places where C++ is sometimes better about things than Rust is providing information gathered which is ancillary to the direct purpose, in case you wanted it, so that you needn't then ask for the same information a moment later. In Rust this information may have a difficult to ascertain lifespan, and so it's dangerous, but in C++ they figure the out-of-date nature of the report is your problem. Sometimes to your detriment of course, but often you could really have used that and must now go measure it yourself because the thing you just called knew it but didn't tell you.

Here though it wouldn't matter, Rust's array types, and both the owned mutable String and reference &str types know how long they are, so you are never flailing about counting even once. You also would just serde_json this data in Rust because serde is practically part of the Rust Standard Library, whereas there's nothing quite so ubiquitous in C or C++.


How much "state" would sscanf really need? From memory (and with an extra peek at the man page) each directive is in sequence of the "format" specifier is sequential (it's not nearly as flexible as a Regexp) so you really only need:

A: a processing state (int/enum). B: a current fmt str pointer. C: va_list struct to keep track of destinations. D: a source string pointer.

Now if what I saw mentioned in another comment is true that there is some insane conversion to a FILE* stream to do this, then yes I can see why it happens but it's not really NEEDED to implement the function as specified.


This article was written before Rockstar released a patch and the 70% speedup is from his strlen() caching.


Yes I'm aware, that article basically triggered the official patch. Would be interesting to know if the Rockstar patch provided a further speedup though.


The author provided an update at the bottom. His hack cut the tone from 6m to 1m 30s. The official rockstar patch took it down to 1m 53s. It’s MUCH better but still off the mark by a bit.


It was 1:50 vs 1:53 - probably just noise.


It's funny though. Several years ago a coworker did some profiling on one of our server software components and found that it was spending up to 10% of the CPU time in strlen()... for very similar reasons.


Had something similar happen with low-level components in a PHP application I inherited repeatedly calling trim() and/or str_pad() on values that had already been processed. Refactoring those out led to a 10-15% performance boost in our test suites.


I've seen PHP applications use 30% of their CPU in the “realpath cache”, which at the time was a super-slow (O(n²) in this case) home-grown hash table for caching the result of realpath(), which was called all the time because someone decided include_once() needed to run its paths through realpath() in case someone included a file multiple time through symlinks. And I guess they couldn't trust inode numbers or Windows didn't have them or something…

I hope it's gone. I really do hope it's gone :-)


The title is still not right as the poster is not the 'I' as in the title.


I assume there is an upcoming: Show HN: Passive karma farming to 10k

Two submissions an hour, 24 hours a day...


Wow. I wanted to call you out for just being an asshole, but seriously - is this account just a bot? And who cares about karma on HN?


Both things can be true.

There's clearly a bot here but ddtaylor has commented on this thread, so there's also a person in there somewhere.


He's just reposting stuff from older years, mostly on the same exact day/month.

Should this be allowed? It's not breaking any "rules" but then 10k people could setup a bot to do the same thing and repost stuff twice an hour..


I used to be a video game developer.

I guess it's down to priorities and passing the buck.

Our priorities were always on profiling the running game to make sure we got every ounce of juice out of the hardware and the game ran smoothly on the lowest systems. I can't think of one time we would have had the spare time to look at loading times. In those days we were being pushed into 80+ hour weeks just to make the game work at all, never mind finding time for "luxuries" such as this.

And of course, on a big team, a problem like this would always be someone else's problem.


You are saying a game had a 6 minute load screen and everybody would throw their hands up and say “but look at the dynamic lighting once it loads… it’ll all be worthwhile!”

If that’s not group think gone bad, I don’t know what is.


Not to mention, unless they have a really clever system to do hot code reloading, EVERY SINGLE DEVELOPER AND TESTER would have been plagued by these load times.

I wonder if the watercooler rooms at Rockstar are exceptionally furnished as a result.


Sometimes testers aren't even working with the full data set. And testers might have some seriously shit-hot hardware that is max spec too. When I was developing everyone would go buy the absolute best PCs out there at the start of every new game to make sure they lasted through dev.


Max spec wouldn’t make a difference. 7 years later, still slow.


The customer has put up with 6 minute load times, but a low framerate kills sales quickly (unless you're Star Citizen).


And kills reviews. Most reviewers won't freak about the load time, but the framerate they will.


Alright, this comment thread has some potential R* devs. (Jokes jokes)


I'm not saying there is a R* cabal, because of course there is no cabal, but if I told you what the *back* of this sign meant, I would have to kill you:

https://www.reddit.com/r/gaming/comments/3ylmm4/a_staggering...


I've noticed similar stupid problems playing Portal 2 with a friend. Downloading the maps takes a really long time, but I've got a decent internet connection and so does my friend. It appears that when his machine is loading, my machine has no network activity whatsoever, and then once he's loaded the maps, then my machine will start loading them and his machine has no network activity.

The game has a number of other weird issues like this. After you've played a map, it asks you to rate a thumbs up or thumbs down, but the hit areas don't match where the button is drawn, for example. Sometimes the mouse doesn't cause my character to turn. I can quit and restart the game and it's the same. But if I open a different game, quit it and then restart Portal, it works fine. WTF?

I realize this is an old game so they aren't updating it (I have to boot into a 32-bit OS to even play it!) but it's a little depressing that such egregious issues went unnoticed for so long that we're now probably past the point of fixing them.


I had to do a double take when reading your comment before realizing that you were probably referring to the Mac version of Portal 2. If that is the case then it's not unexpected, for OSX routinely break compatibility between major versions unless the developer is able to keep up with the changes.

This is not to say things are that things are that much better on the windows side. While most 32bit binary run fine in WoW64, driver issues become more common as the games get older.

Another major issue is 3rd party services bundled with the games that are no longer available. I can name more than a dozen of games published around 2010 that are no longer available for purchase because the original build depends on the deprecated Games for Windows Live service and would refuse to launch without it installed.

Microsoft abandoned GFWL around 2018 and turned off the online installer at a later date so you would have to download an offline installer from some dodgy non-official website. Thankfully it still works with current versions of windows as long as you still have the physical copy of the game, and the backend of GFWL is partially working with achievements still being synchronized with Xbox Live accounts. But who knows how long that's going to last?


I feel like these problems are all around and often present in games. There's one that particularly infuriates me in Hearthstone, a Blizzard developed TCG, where when a game ends it causes the processor to furiously spin up and causes all kinds of anomalous behaviour like webcam cutting out, mic distortion and other generally laggy behaviour.

I'm aware that there's probably a lot going on at the game's end but I'm still utterly convinced there is a coding bug causing this amount of processor spiking but I lack the skills of debugging Unity, which is the game's engine, to determine what actually is going on. I guess that's the issue, the ability of conducting the research the kind in the OP article is extraordinarily rare even among game devs so these problems persist (Hearthstone is just turning 8 years old now) without anyone with the deep knowledge to rectify it.


First of all, I find it an impressive skill to be able to do this on a binary blob.

Secondly (optimistic version), it really shows that they (management) didn't care, that nobody ever made a ticket to fix loading times, since this would be trivial do find with a normal profiler on a debug build.

Secondly (pessimistic version), nobody at rockstar had enough skill and/or professional pride/sense of ownership to find this out themselves.


Seconding the other comments here - I'm a developer and I'd have to agree it's about priorities. Every game I've worked on has had load time targets but those targets were set based on what the loading time of the game was at the time the target was introduced, so the goal would be to not make them regress.

The other thing to note is that with this particular problem, the issue is parsing a json blob on a live environment. I can count on one hand the number of times I connected a development client to a live environment _ever_, it's just not the done thing. Developers likely have local environments that have a much smaller json blob to parse (and that's likely where the load time benchmarks were run, if they exist).


Wow, wow, wow! This was such a nice read. Definitely buying the guy a beer (there is a button at the bottom).


How can not a single person have hooked up a debugger and just paused it during the 5 minute load time? Especially given the difference between story and online mode? After eight years?


proprietary software combined with a small circle of people that play GTA5 and have the technical ability to disassemble code

also: corporate life, original devs rotated out. new team that has been maintaining is just adding features, new micro transactions, or building new heists


Although a bit different rather sad seeing Take Two/Rockstar attempt to censor other fan projects through DMCA or lawsuits.


A really cool example of good debugging especially without the source.

Also a good reminder that the slow thing may not be what you think and if you aren’t profiling you don’t really know what’s going on.

And an even better reminder that typical “metrics” trying to assess developer value can be pretty meaningless. This is the kind of code change that would drastically improve everything but show up as a blip on some of the pathetic measurements out there, e.g. hardly any lines of code or whatever.


Perhaps my favorite thing I've ever read on HN :) I love to see it crop up again


Please add RSS to your blog!


Added a feed plugin since people keep asking. Never used RSS so hope it's working correctly.


A heart warming story of a hacker turned game dev! Nice job.

They offer you a job too?


I wonder if this code with poor time complexity was written by the same people who are always complaining about leetcode being useless.


Is the source for Luke Stackwalker available anywhere? I found an old Github project, but it's last update was 7 years ago.


Cheaters and long load times are the two reasons I stopped playing.

I couldn't be bothered to start again after this update came out. Ah well


Did rockstar ever release a patch for this ?


I'm not sure if you made it to the end of TFA, but yes. And the author got a 10k bug bounty


They took his fix, applied it to the game in an update and paid him a bug bounty of $10,000.

"Just got awarded $10k through their H1 in-game bounty as an exception"

https://support.rockstargames.com/articles/360061161574/GTAV...


Anecdotally; I still wait 5-6 minutes for online to load, and 3-4 for single player.


Digital Foundry measured a 20.2-second boot load time for GTA V single player on the PS5 and 28.9-second boot load time on a PC. Not even the PS4 takes 3-4 minutes.

https://www.youtube.com/watch?v=m5QHInJShQw


Good for them. Computing performance is relative between machines.

I have no obligation to confirm for you relative truth. Gathering the evidence would be a waste of time and CPU cycles for what is ultimately an ePeen size competition.

Fact: my computers hardware and software configuration loads GTA5 in 5-6 minutes. I experienced it first hand two days ago. Sorry all of physical reality does not align with your experience, not sorry.

Don’t believe me if you don’t want to.


A user released one first, then rockstar did subsequently: https://www.pcgamer.com/rockstar-thanks-gta-online-player-wh...



This is why you need a language (on a framework) with high introspection, with smart insights. Excuse my opinionatedism, but this is what .NET Core gives you. It even streams JSON serialization/deserialization without much memory requirements, iirc.


He did basic troubleshooting with task manager, on an obfuscated executable, with a disassembler and a memory dump, without knowing anything about the code base, and fixed it with in-memory patching. So while I agree introspection can be a godsend, this specific case is not a tooling problem.

Something this blatant is what happens if nobody takes a look. I can imagine they got tons of complaints, but 'it is slow' is nebulous enough to consider it basically unsolvable/normal. Maybe it never got a real bug report, maybe the report never made it past the service desk, maybe management thought it would cause a wild goose chase without resolving anything. But no developer looked into this, as just hitting a breakpoint in the 4 minute pause would insta-fix this.


> This is why you need a language (on a framework) with high introspection, with smart insights.

That would be nice if it were true.

What happens in reality is you end up paying the performance cost in other areas specifically due to the framework. Frameworks tend to cater to very broad use cases and will generally be less performant than custom code written for one specific use case. Frameworks also tend to be extremely interdependent meaning that to rip out one part (for perf) you need to rip out another huge part.

That is why, to this day, if one's business NEEDS something to be performant (no do-overs! no second chances!) then industry veterans will opt for custom systems. Yes they'll have more edge case bugs. Yes they won't be performant in areas you don't care about. But you will still sell 165 million copies of your game because it performs well on a bunch of different platforms with varying specs.

This particular case is more of an organizational problem. No one was responsible for load times, it wasn't a priority to improve, and users weren't complaining enough about it to matter.


Nah.

We just need to stop this idea that JSON(or similar formats) is an acceptable data storage mechanism. It's fine when interacting with low bandwidth APIs across different systems, where the need for debugging is greater than performance.

But on a shipped game? Just bake this JSON into a better, binary serialization format. No need to parse data that seldom changes over and over again.

It doesn't matter if you can stream json, if you don't need to call strlen, what have you. Just don't parse unnecessary stuff.


I'd guess that json is the least of their problems. Even with the fix, load time is still an unreasonable 2 minutes. This suggests they aren't doing any of the load performance things game devs have figured out over the decades, like streaming loaders and stacked memory pools.


This is GTA Online, so while it's a "shipped game" that JSON changes, Rockstar decides to offer the "Pfister Astron Custom" for $2.5M and they just change the JSON. I don't know whether it's fetched from their servers on startup, or whether it's on disk and just updated periodically, but either way choosing JSON is not as crazy as you seem to think.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: