This language really infers that they still don't understand why there was such a loud outcry and bad press. Even if it's just a small number of users, a company can't pull the rug on them without notice - for those small number of users there was no reasonable expectation that this would become a problem, and seemingly now it is no longer a problem.
Also if the number is so small: engage those users directly to find a compromise first.
And yet somehow I see this argumentation constantly done everywhere, including this very site.
"Only 10% of users...", "Only 5% of Tesla owners...", "Only 3% of Americans...", "Just a few % of blind people..."
Somehow this is a mistake everyone constantly and repeatably does to make things crappier. And that includes rockstar engineers, normal engineers, managers, product people, everyone.
A while back where I work we had a serious mess up with our phone system and to cut a long story short, a number of people had requested a callback through an automated system and those call backs never happened. It was a lot easier to say "A small number of customers - less than 5% were affected" vs "We seriously let down more than a thousand people many of whom are vulnerable."
Once you get to Google's scale it becomes more ridiculous. There was some feature that was being removed from YouTube at one point and the talk was about "only 5% of users" - but that could translate into millions of people.
Percentages are obviously useful to get a feeling for certain things but too often they're used to minimise the impact of something.
Exactly. Whenever a corporation is fined the news make sure to shows x millions of dollars. I have never seen it in percentage of revenue in that context.
Everyone seems to be overlooking that a total count + percentage gives you the number of active users. That’s generally not something broken down at a product level. So you can give percentage or absolute. If you only ever give percentage then you don’t have a problem and it gives you a sense of scale. If you give absolute numbers it doesn’t + you might accidentally give percentage out in a different venue allowing competitors to have a sense of how well your product is doing.
At least, that’s my generous interpretation of the incentives at play. I don’t think it’s as simple as “those engineers are dumb” / “corporations are evil”
> Innocent until proven guilty is a bedrock of American justice, no?
When someone keeps violating their parole you start asking them to justify why you shouldn't send them back to jail.
We're past the trial phase. The guilt of corporations has already long been established over the past 100+ years. Furthermore, I am not the government and am not required to give them the presumption of innocence.
On a personal note, my great-great-grandfather didn't die shooting Pinkertons for nothing.
Absolutely. When a corporation has a track record of being guilty and engaging in anti-consumer behavior (which, let's face it, is most of them), they lose the presumption of innocence in the court of public opinion.
That’s right, the group itself isn’t. It s a collective action.
Ultimately, the legal fiction was created to avoid having to create new rules but it creates all sorts of problems because corporations and groups in general are more like political parties and governments where they’re made up of subgroups with their own agendas and those subgroups have their own individuals with their agendas.
For example, if a corporation mass poisons people, no one typically goes to jail and instead the corporation pays a fine, very similar to what would happen if the government did that (see Flint and what PG&E did in Hinkley). You can put people in jail. You can’t put a corporation in jail and the corporate veil is insanely strong against getting pierced which causes all kinds of moral hazards.
I think it stems from people being really poorly equipped to think about scale like that. Even someone who has lived in NYC and/or been to jam packed sports stadiums, you're still only around ~50k humans max and that comes out to around 0.005% of drive users. So if a developer comes up and says a change will only effect 0.1% of users it may seem like a insignificant percentage but the actual impact is something to the effect of 20 full stadiums of people, massive.
I would submit the root cause of this is not what you might expect. I would say it's a failure to understand that "everything is correlated": https://gwern.net/everything
What I mean in this case is that if you say to yourself, "hey, only 4% of my users are using more than 50 gigabytes, and I can afford to lose them", it is very likely you are using a model where those 4% of your users are essentially uniformly randomly chosen from your user base. You can see usage, and you may not be able to see much else, so you assume that usage is uncorrelated to any of the other aspects of these users that you may care about.
In reality, everything is correlated to everything, and what that means in this case is that your 4% of users are not being randomly chosen from your user base. This doesn't mean that every way this sample deviates from random is bad for you; you certainly can just get lucky. For instance, that 4% of users probably includes a huge proportion of your users using it for piracy or similar activities. Cool. You may even be using this as a motivation for the action since you can see this to some extent based on reports and analytics, certainly.
Unfortunately, that 4% probably also includes a disproportionate number of your True Fans, who love Google partially because of this free drive space, and because of that love, drove their work environments to Google Teams and other paid services, thus netting Google profit far beyond the costs of this particular Drive account, but indirectly and where accounting can't see it. True Fans who will perceive this not as a business transaction, but as a betrayal, and who may in their passions take actions to do things like (on the margin) remove their companies from paid Google services, and take time out of their day to create bad press.
I'd say that's the core mistake being made over and over. You think you're just getting rid of your "problem children" but those problem children very frequently have correlations to something you do in fact actually want.
Of course, that doesn't mean you want to be abused either. It just means a more delicate approach may be called for than just hacking off a chunk of your user base when you don't really know what it contains. It ought to be an assumption that any time you're taking an even slightly non-random slice of your user base like this that it may have hidden correlations in it you're not taking account of.
in fairness, ignoring the 3% edge cases generated by power users... that is a central feature of Google's offerings, i.e., Google Workspace. When Google was 'cool' everyone simply figured those edge case features were 'uncool', now they demand them anyway.
I'll bet a lot that it's MOST people that do this kind of reasoning mistake pretty commonly - when programming, reading analytics or even debating why someone doesn't deserve to have a mini iPhone.
Right. IMHO that's one of the few valid reasons to require some basic understanding of mathematical rigor in programming. The mindset that you need to deal with ALL, 100%, of the cases instead of just settling for "90% is prolly enough" is very important to counterbalance these common tendencies you speak of.
It’s because legal risk matters more to them than end user satisfaction (of a specific end user group maybe):
Option a: admit they screwed up and open themselves up to additional legal risk by doing so
Option b: minimize the damage done and annoy the hell of this end users.
I think it's also easy to miss the fact most likely only a small percentage of Google Drive users actually use the service in a way where they would notice a degradation. The majority of users on most free platforms are largely inactive.
If the entire service went down, only a handful of users would be immediately affected.
That is exactly the problem, though. If you are planet scale, you don't get away with "we did an oopsie-whoopsie".
At a certain point trying to downplay impact becomes insulting, especially when this issue has been around for nearly two months and they are only backpedaling due to bad PR.
If they understood the problem, they'd realise that there is value in addressing it head on.
This is representative of Google's larger problem in the industry: users are skittish to uptake and rely on Google's new products because of their established behaviour of making arbitrary changes without consulting the market.
I suspect the reason why this resonated so well is because it was too familiar with user frustration. Today it's drive, tomorrow it's something you rely upon.
Still not an actual apology, or acknowledgment they messed up, but at least an indication someone in the Google Drive team understands how badly they messed this up.
>If we need to make changes, we will communicate them to users in advance.
Doesn't sound very sincere to me. It's certainly a change that they will communicate changes to users in advance. Does anyone really expect them to suddenly start now, as if that's a company wide policy they just implemented across the board, for the first time in history, in response to this incident, because nobody at Google ever thought of doing that before?
Absolutely true and the "small number of users" clearly are your power users. You don't mess with power users without some clear path. For better/for-worse, they tend to be rather influential.
> You don't mess with power users without some clear path.
Like last year when they abruptly decided to cancel Google Apps for your Domain legacy accounts after like 15 years? The only people using these services were power users, many of whom seem to have migrated to other services.
It's worse. They announced to cancel the free legacy tier, so I tried to upgrade but it said my account type was in the wrong country/region or whatever, and didn't let me upgrade. There were rumors a couple weeks after the original announcement that they might allow a free tier after all. I don't know what happened in the end because I just moved my email to protonmail.
Google is managerially stupid; they ended up reversing that but they lost a TON of power users, and it is literally impossible to measure the number of companies that have gone with o365 or other solutions because "the IT nerd" was pissed that his free domain got fucked with.
I'm one of those nerds and a minor need for a domain + email that would have been a google Apps (paid) late last year is now on proton mail instead.
There always was - this is what you get for using google drive. I’ve long since assumed google will do this one day or another and have depended on Dropbox for my file storage and syncing needs. Does it mean I trust dropbox completely? Absolutely not. I sure as hell trust them more than I do google though.
The same Dropbox that people wanted to boycott and fled from after Condalisa Rice became a board member. I'd trust Google above Dropbox any day and I don't trust Google enough to actually use Google Drive.
A key difference here is that Dropbox's entire product is storage. Sure, they have some corollary products now -- HelloSign, etc. -- but they're a storage company first.
Google could kill Drive dead tomorrow and it would have effectively zero impact on their bottom line. Conversely, if Dropbox rolled out a change like Google Drive did, it would adversely affect their bottom line, and possibly cause long-term harm to the company, because it would erode trust in their core product.
I have NAS so I just installed syncthing instance on it, after nth time commandline dropbox thing managed to break itself.
The NAS instance is backed up together with rest things on NAS and overall it has been pretty much drama-less experience.
The disadvantage is I can't just log on web and download files on random machine (I guess that would be possible if I re-shared the directory in something like nextcloud), but I don't really need that feature.
You’re not the target audience then. I don’t have an NAS (or even a home country leave alone a home), and live off my phones and MacBooks. Dropbox works perfectly for me.
I think the real problem is in trusting one single thing, just like Silicon Valley Bank users found out recently.
PS: I say this as someone whose cloud backups are all on Google Drive. I'll have no one to blame but myself if I lose the data. Just couldn't be bothered to upload it to a different service so far.
Any "SaaS" you're consuming where you suspect that you're not paying more than the cost of providing that service should be suspect and not trusted. Google Drive for backups is fine because (hopefully) losing one backup source won't kill you and you can get another quickly.
One of life's unwritten laws is "you get what you pay for."
I totally get why someone would want something for free but to whine about this free service not giving them what they want is some combination of entitled and ignorant.
I get the gist of your argument, but the limit was also applied to paid drive accounts - in many media reports these were the people complaining.
Doing the maths it also seems clear that this would be affecting paid users primarily, since the free tier includes 15GB of storage, one would need to keep their average file size under 3kb to hit the 5M file limit.
> Google: "Only affected a small number of users."
This kind of thinking is so so so wrong.
Imagine government saying it won't allow same sex marriage because it only affects a small number of citizens, so there is no point to bother or something in similar vein.
Governments do say that to more niche things... like for example legally recognising polygamous relationships...
Governments generally don't want to make laws about anything that affects less than ~1% of citizens. Gay marriage however is an issue that has tipped over that threshold, with 7.2% of americans considering themselves LGBT.
And it should't have to be over that arbitrary threshold, because human rights for <1% of the population are still human rights, and all the arguments against gay marriage are unmitigated bigoted bullshit.
And we the customers get to be displeased with whatever service they put out, just like with governments. They might not be required by law to be equitable, but we can hold them to it.
Yes, as consumers/citizens we can disagree with the decisions governments and businesses make and can "hold them to it" to whatever degree we're willing to go to.
But the OP's point was that it's wrong for a company to make a choice based on how it'll affect a small number of people just as it's wrong for a government to make a choice based on how it'll affect a small number of people.
That is (to me) a fundamental misunderstanding of the difference between government and business. Government should be looking out for the least of us, but businesses aren't (and shouldn't be) held to that same standard.
This is a very American attitude. In many nations part of getting a business license is a statement that you will operate with public society's best interests in mind, even if it reduces your specific profitability or business success.
I also believe that the Waffle House shouldn't have to serve pancakes because some of its customers really want pancakes. Waffle House can choose to serve pancakes (or not) based on customer feedback, but I believe there's nothing about "operating with public society's best interests" in forcing Waffle House to serve pancakes.
Because I believe running a business is about making choices and those choices can include what to sell/offer. As long as you're not actively breaking any laws and/or discriminating, you shouldn't have to offer something you don't want to offer.
The grocery store near me doesn't have to carry vegan meats if I'm the only customer who'll buy them.
1. I never talked about services nobody will offer. I am talking about looking out for the needs of every individual vs. the individuals you choose to look out for. Think about mail. The government should make sure every person gets mail delivered. FedEx should not have to worry about that.
2. Obviously governments are about choices, but how and why they governments make choices is different from businesses.
Remember, a government and a business are two different entities that serve different purposes and have distinct characteristics. Here are some key differences between the two:
Purpose: The main purpose of a government is to provide public goods and services to its citizens, maintain law and order, and protect the interests of the state. On the other hand, the main purpose of a business is to generate profit by providing goods and services that satisfy consumer needs.
Ownership: Some governments are owned and operated by the people, through elected representatives. A business, on the other hand, is owned and operated by individuals or groups of individuals, who may or may not be accountable to their stakeholders.
Source of funding: Governments are funded through taxes, fees, and other charges levied on citizens, whereas businesses are funded through investments, loans, and sales revenues.
Control and accountability: (Some) governments are subject to democratic processes that provide checks and balances on their power and ensure accountability to the citizens. Businesses, on the other hand, are subject to market forces and regulatory oversight, but are primarily accountable to their shareholders and investors.
Social responsibility: Governments are expected to act in the public interest and promote the welfare of their citizens. Businesses, while expected to operate within the law, are primarily driven by profit maximization and are not necessarily obligated to prioritize social responsibility.
Overall, while there may be some overlap between the functions of a government and a business, the two are fundamentally different in terms of their purpose, ownership, funding, accountability, and social responsibility.
In "tech" its often forgotten that they are dealing with real people, individuals. They don't realize that their decisions cause real pain to people and do decision making purely through the formula derived to optimize profits. It's likely done by some mid level manager to improve the personal numbers to get promotion.
I guess, at some point the analytics noticed that large number of files is suboptimal and they ran a simulation to see what percentage of the users are affected at what number of files limit and found out that >5M files impacts smaller number of paying customer revenue than the costs generated and they said just put the limit. Negative social media attention probably changed the outcome of the formula, so they lifted the limit.
> In "tech" its ofta forgotten that they are dealing with real people, individuals.
Don't attribute to tech that could be adequately explained by a management incompetence. I've seen stupid things done by a techy people. I've seen way more way stupider things demanded by management [and implemented by tech or whoever at the bottom of the food chain].
Sure. However tech is obsessed with data and optimisations, if they run a hospital and the data showed that obese people are less profitable they will put a scale on the entrance and triage by BMI or someone will calculate that some drug which kills in the long run gives enough time to propel their career through saving pennies.
I don't see your point. Did I say that tech is the only one doing it?
What's special about tech, is that they have the means to take it to another level because they are very capable to collect and analyse data and quickly deploy changes and almost never face the consequences. Low touch high scale is the name of the game and they are completely disconnected from the people impacted. 10K people losing their children's baby photos is just a blimp in the analytics and its "fixable" by an apology tweet, they will never understand the pain those impacted will suffer because they don't have a contact with the customer besides some charts.
It seems so, although possibly you did it accidentally. When you say "if tech controlled hospitals then it'd do it this way" then you're definitely attributing this mindset to "tech" people, rather to any people who address customer populations in the millions.
I think it's reasonable for a reader to assume that if you're making a category error (I've seen some people I call "tech people" do this, therefore all "tech" people do this), you aren't being sophisticated about your categories, and that you are attributing it exclusively.
A correct (and not exclusive to tech people) mindset would be "some people think like this. Some of those people work in tech." And you'd be right. Others work in the NHS, to decide what medicines get paid for for the most good.
Roll back far enough to the times of Charles Dickens and early industrialisation and the answer is a qualified yes.
Data driven acturial decisions affected significantly large chunks of the population of industrial towns and cities.
( Populations then, of course, lower than now so the qualified exception is the proportional size rather than absolute numbers )
Time cards, the rule of the clock, mass labour projects, cost shavings to the penny, food supplies barely but just sufficient, etc. are all practices that date to these times at industrial scale and they were implemented via ledgers, card files, and literal computers .. desk arrays of clerks passing numbers back and forth to refine raw data.
> How could anyone think that rolling this out without warning would go smoothly?
They had numbers on the percentage of users that would be affected by this change as well as content to tell these users what they should do to fix things on their side and someone decided the number of people affected was small enough to not bother to warn people in advance.
The level of incompetence and mediocrity inside Google when it comes to managing a product ( let alone create a new one ) is astonishing.
Even someone running a lemonade stand knows that trust is the number one requirement when serving a customer/user, everything else comes based on that.
Yet, the smart people at Google seem to think otherwise.
>> The level of incompetence and mediocrity inside Google when it comes to managing a product ( let alone create a new one ) is astonishing.
Google's product is advertising to people.
Google's users are advertisers who want to put their messages in front of people.
Google Drive, Gmail, YouTube are all ways of delivering that advertising and collecting information about people for targeted advertising.
Why would they spend money building a trust relationship with people when they can just offer free stuff and people will gladly hand over their data and their time to view the advertising?
Devil's Advocate... If they don't spend time on the end user product side also, then they have nobody to advertise _to_, and their advertisers would dry up.
The free stuff has to be good enough to bring and keep people, but not so costly that it is a large expense.
"Customer service" for people does not scale well and has high costs. That is one reason why they automate it as much as possible and simply do not have it where it cannot be automated.
I still remember in ~2009 I said Google wont be able to compete with Apple, simply because no one at Google understand anything about products. For nearly if not over 20 years since I have been against Google. Their downfall finally came, out of the three they are now the ones that seems replaceable. Feels so sad they could have done better. And yet not.
Google keeps shooting itself in the foot with google drive, it's insane.
When Google Docs came out, everyone thought Microsoft was screwed. They had such a headstart in cloud-based collaboration.
But somehow, they managed to eff up google drive and anything offline for so long that Dropbox is still the best option for cloud storage and Microsoft had the time to rebuild its whole Office suite in javascript.
My biggest gripe is that the web UI for Google drive seems exclusively designed for tablet users. Enormous icons with previews, little text, no real hierarchical organization, assumption of search-based navigation of docs, it's just so difficult to organize and use. It's like every thing anyone ever knew about organizing and finding files was just thrown out the window. As someone who has used computers for nearly 30 years at this point, I find it infuriating how hard it tries to make my skills useless.
I do think that Google Drive is much better than Dropbox for various reasons (browser GUI, storage space), and certainly better than Microsoft's OneDrive or Sharepoint, which e.g. have ridiculously long URLs. But it is indeed remarkable that Google Docs (or whatever their office solution is called) was first ahead of Microsoft in the "browser office" space and then lost its lead.
I think the main cause was that Microsoft Office formats (docx and xlsx) never ceased to be dominant, and Google Docs didn't support them until it was too late.
My company first used "Google Suite" for a few years in combination with local Microsoft Office. You couldn't get rid of the latter, since other companies "communicated in" Microsoft Office formats. If you can't open and edit them reliably in Google products, you might as well switch to Microsoft completely. Which is exactly what happened for us.
I honestly don’t understand how you can find google drive better than dropbox.
I wanted to use google drive. It would have given me more space for emails, and I wouldn’t have had to deal with another service.
But even syncing files from my computer to the cloud never worked properly. I kept getting conflicts and files that would always fail. Then there is the ”shortcut” debacle where you can’t sync files that are shared with you on your own computer.
Okay, those are apparently different use cases. I was mainly speaking about the browser GUI, which seems great to me for Google Drive. Also the Android app was much better when I last tried it. There are also little features. For example, if a Dropbox folder is shared with you, you can't simply put a short cut to that folder in your own Dropbox folders. You have to "remember" it outside of your Dropbox folder structure, e.g. via a browser bookmark.
Perhaps Dropbox is indeed better when it comes to synchronizing folders though, which was its original purpose after all, as the name suggests.
The Trojan horse for OneDrive/Sharepoint is that they're all the SAME in the back end, and you can interact with them via Teams and nobody knows the difference.
Now that with Office + Teams + Sharepoint you get multiplayer Powerpoint/Excel/Word, Google Docs is on the backfoot again - the HUGE advantage to Docs was being able to work on them collaboratively, even if they were missing 50-80% of the features expected.
It's not LibreOffice, but while revisiting CryptPad (which I thought was a multiplayer notepad like etherpad or codimd, except with an encryption key in the url fragment identifier) I was impressed to see that they're expanding to become an entire office suite. You get a WYSIWYG editor (I'd rather wish for markdown but ok), spreadsheet editor, survey forms, and more which I don't remember, all live like google docs but (as I understand it, I didn't audit it) end to end encrypted between the users. It's a bit heavy because everything has to happen on the client side, so it's a lot of JavaScript, but after the page loads it works smoothly. You can self host it as well, there's a list of instances somewhere on https://cryptpad.org with the official instance being https://cryptpad.fr
LibreOffice has a ton more features and is way harder to port. Case in point: I was at an open source conference where LibreOffice proudly demo'd their new server version. At some point they drop that it uses mapnik. Me, confused, asked huh why'd you use an OpenStreetMap rendering library? Turns out they basically run the ancient C++ UI on the server and make a VNC-like connection and that map tiles is the fastest way to load partial screen updates. At least, that's what I remember from that presentation, I haven't looked into that madness further, but that's LibreOffice online...
You can kinda do it with NextCloud and their LibreOffice integration but I wouldn't trust it with anything important given it often breaks during updates.
Yarn provides a "Plug'n'Play" mode implementing exactly this behavior.
Dependencies are kept as compressed archives and a loader file is generated so Node can read them directly. If you start Node through Yarn (`yarn node main.js`), it automatically passes the correct flags, but you can also do it manually (something like `node -r .pnp.cjs --experimental-loader .pnp.loader.mjs main.js`).
Overall I found this mode to work pretty good. I'm using it for a medium-sized Angular (+ TypeScript) project. It also noticeably improves install times. The PnP mode is stricter about dependencies and expects dependencies to be readonly. A couple dependencies may need to be marked as "unplugged" (installed a regular decompressed dependencies) as a workaround.
... That happens to be a Python alternative to PDFtk[1]. It’s been kind of abandoned in recent years, though.
Also, in the early times of OLE 2.0 MS Office included a utility called Binder that could put several Office documents in a single file and edit them using a common interface. Nothing came of it.
The irony is I visited a google office once and I’d rank a stapler as the 2,575th most opulent thing in there. The furniture alone is insane. Yet they decided to yank staplers.
I think the reason is that no one uses staplers, and yet there's a cupboard of them. Obviously it's not just staplers, there's loads of stationery, but why? A few pens and some notebooks is all that actually gets used.
So the staplers continue to sit in cupboards, unused. They've already been bought, so how is Google saving money? Selling the staplers? I bet they were just thrown in the bin. How wasteful.
I love staplers. I love playing with them and the rows of staples. My favorite unused-in-decades office accessory that some offices still stock is the staple remover, it's a great little (sometimes painful) fidget.
Even in paid shared web hosting services, there is an inode limit. And it was more than a decade ago since I stumbled with this issue. Since then I monitor my file count and delete those with last TOUCHed of at least 30 days even if I have moved already to a dedicated server long time ago. I was caching complex database queries to the disk and did not bother to delete the files before.
I wonder if there is still a limit with modern Linux OSes and that is why Google is limiting some of their users?
I think Google Drive is abstracted enough that they are not running into something basic like an inode limit...
But this does remind me of having to set up a cron job on an old PHP application because the sessions were saved as files and it ran into the inode limit several years in :P
And, of course, 5MB isn't some power of 2 (as most limits are). Its hard to imagine where Google got the number unless its just some hardcoded constant.
>I wonder if there is still a limit with modern Linux OSes and that is why Google is limiting some of their users?
Ext4 has 32-bit/~4billion max limit per filesystem, newer filesystems like btrfs have a 64bit limit. Neither allocates the whole max size by default. I don't imagine Google drive uses an out of the box filesystem underneath though.
Btrfs is particularly special as the inode limit is per subvolume, so it's quite easy to have an impractically huge number of them with some cleverness.
> I wonder if there is still a limit with modern Linux OSes
Since filesystems introduced dynamic inode count it hasn't really been a problem. I remember ages ago you had to format filesystem in certain way if you wanted a lots of small files (say news or email server), but the need of that went away with better filesystems and growing storage sizes.
> and that is why Google is limiting some of their users?
On their side it's most likely on the number of entries in database, which usually is more than just storage cost.
It seems that there is no hard limit with ext4. It depends by inode size, disk size, etc. I searched a little and I didn't get any deeper than that. Anyway, df -i reports this about my 2 TB SSD
I guess that 122 M inodes in the first column is the limit of that file system. Much more than 5 M but Google Drive is probably giving a quota to users. No idea of which file system it actually uses on the disks.
I didn't look into this and don't know what really happened, but I can guess.
It's more likely a tradeoff. Either you set this limit and protect your service from an identified scalability limit with the current architecture, or you plan for a (possibly long and expensive) redesign to get rid of this bottleneck.
My guess is that a group of SRE and devs identified the risk, listed their options, evaluated the impact on users (eg: which fraction of users have more than 5M files) and assumed that the change would be mostly unnoticed and that the error message would be enough to push users with more than 5M files in their drive account to do some cleanup.
This group of people underestimated the impact and maybe chose to not involve product managers. It's also possible that the decision was rushed because of recent layoffs or any other random event which pushed engineers to act quickly.
With the bad press, leadership got involved, decision to rollback was taken. In my team, we would have a retrospective doc to discuss the issue (not exactly a postmortem, as this process has specific requirements which would not be applicable to this case).
I think this is a easy mistake to do even with very good intentions, and I can see myself doing it.
> I think this is a easy mistake to do even with very good intentions, and I can see myself doing it.
Hard disagree. You missed one very important part in your writeup - at no point did they communicate that they were imposing this limit, and that this limit appearead, undocumented, overnight.
I was someone who was directly impacted by this change. We're a 40 person company who used (past tense) GDrive as a shared network drive, including for storing builds of our app. We pay $18/person, and as part of that, google workspace advertises 5TB per user pooled[0], and nowhere in the google docs does it mention that this limit will exist [1]. If I was aware of a limit, we would have cleaned up our old files, but instead we started getting spurious 403's - as far as we could tell we were well within our usage limits. It was only when https://issuetracker.google.com/issues/268606830?pli=1 this post hit HN, I realised what was wrong.
> at no point did they communicate that they were imposing this limit, and that this limit appearead, undocumented, overnight.
Not to defend google but I've seen plenty of engineers make such mistakes, and you probably have as well; it's just that it didn't then result in bad press.
When you are an engineer working on a living product, and you identify some performance-related issue, changes you make to the product can easily be classified as bugfixes. For example, you identified an end point that should have a rate limit and didn't; you fixed it, it was a potential security issue, it didn't need communication to end users... as far as you knew, even if you misjudged.
> When you are an engineer working on a living product, and you identify some performance-related issue, changes you make to the product can easily be classified as bugfixes. For example, you identified an end point that should have a rate limit and didn't; you fixed it, it was a potential security issue, it didn't need communication to end users... as far as you knew, even if you misjudged.
Sure, and the vast majority of companies publicise changes that affect customers. Infact, google do it quite regularly. If you roll out a customer impacting change, even if it's a small number, you communicate it. Docker's recent (mis) communication is a good example of what's required. If you can't take the heat, get out of the kitchen.
I've still yet to see google acknowledge that they've done or rolled this back. Even this HN topic is about a tweet that says:
> We recently rolled out a system update to Drive item limits to preserve stability and optimize performance. While this impacted only a small number of people, we are rolling back this change as we explore alternate approaches to ensure a great experience for all.
i.e. not that they imposed an undocumented limit.
> was a potential security issue, it didn't need communication to end users...
If there's a security issue in a customer facing part of a product, and you change that part to introduce a limit, you communicate that you've done that. Coming in one day to find out that the rate limits have changed and you've not been notified about it is a sure fire way to piss off a whole bunch of people.
> For example, you identified an end point that should have a rate limit and didn't; you fixed it, it was a potential security issue
That sounds careless. Any such change would need to have a impact analysis (which should be part of the team/org/company's SDLC). In this case, communication should be sent out to the clients of that endpoint, with a reasonable deadline, before enforcing any rate-limit.
You see this if you troll through kernel logs or any enterprise piece of software; pages and pages of warnings like this:
BOBFLANGLE IS DEPRECATED AND MAY BE REMOVED, PLEASE REDUCE THE BOBFLANGLE USAGE BELOW 1.5 MILLIBOBS
Then if it actually becomes an issue, you can pull logs from thousands/millions of systems, and determine the extent of actually removing the BOBFLANGLE and begin mitigation.
Is PM not consulted for changes like this? I feel like this is something that a competent PM would ask eng to pause on while they determined actual user impact and lines up messaging.
Short answer, yes, a PM should have been consulted and give their approval for the change.
In the scenario I described above (again, it's a guess, I don't know what actually happened), it's possible that the PM was bypassed because the engineers for a reason they thought was good. For example:
* they didn't even think about involving a PM because that's something you have never done,
* the people who wrote/reviewed the change assumed the conversation already happened,
* they were pressured to move fast to mitigate an imminent or existing problem (performance, scalability, ...),
* the PM who should have made this call "left" the company and didn't (get a chance to) hand-off their responsibilities.
I guess what I mean with these comments is that sometimes there are misses like this. Maybe it's a sign of a systematic failure and that internal processes should be improved, but I don't think it means that the company is fundamentally unable to handle these changes correctly or can't make the right technical or product decisions.
If technical goes to scaling and effects related to scaling, maybe they just decided to let the people with >5m inodes incur the technical penalty, which is almost always speed?
it was always a cost issue, in some sense. If they designed the entire FS model behind drive to have inode limits per entity, then there's a small technology issue bumping that number maybe. Quota systems can be quite painful, in days gone past the Robert Elz Quota model on UFS demanded reboots for some changes, which at scale in a google sized entity is a rolling surface of state change.
5m is an odd number. 24 bit numberfield would get you 16m uniques. I can't see how they needed to steal bits to get to 23 or 22. So there must be some other factor at play here apart from 'we used 3 bytes in the gRPC model'
Perhaps the Windows exposed API assumed somewhere around 5m objects, and they care about the compatibility greatly.
I don't think the number is based on any technical limitation.
It could be a number based on usage. Say 99.9% of the users use less than 5M files. So my pruning 0.1% of the user base they are getting rid of half the number of files, saving X millions of dollars in cost.
Nobody wants to cater to the difficult users that are doing stuff like this.
After that pruning, you now have a new 0.1% of users using more files than the other 99.9%. Might as well prune that 0.1% of file hogs too. Rinse and repeat until you have no users left.
Sure. But all that would mean is that you can only have 5M files materialized locally. I don’t see why drive itself would be limited at 5M. Very strange for this to have any real technical limit.
>Musk has not explained the reason for the change, but some users speculated it was meant to be an April fool’s day gag that the company was unable to bring in on time on 1 April.
When Boris Johnson was prime minister he had a tendency to make some strange comments, most notably about painting model buses. There was a lot of speculation that he was trying to bury undesired articles on google.
It doesn't take the AI bubble to sense the incredible IBMification of Google. The company can't ship product to save its life, and when it does the product gets canceled or is a duplicated effort of some other internal product.
The AI bubble could steal away 50% of Google's revenue stream by this time next year. Even if Google puts up a similar product, it has to retain 100% of its revenue and margin. Stuffing ads in interleaved responses seems implausible or downright dirty under an entirely new regime of high information density retrieval tools. (Granted, this is probably my weakest argument.)
It remains to be seen if users will flock from "Google search" to "Bing AI" (or whatever), but we all know how shitty Google search has become as a product compared to half a decade ago.
I tried the new OpenAI enabled Bing chat yesterday.
Since NASA's announcement of the Artemis crew was on the news, I asked it to list 10 points in favor of space exploration. It promptly did that. Nicely.
Then I asked it to list 10 counterpoints for each of the 10 points listed in favor of space exploration. It tried. It replied with only 4 unique counter points and formatted them as 10 individual responses against each original point listed in favor of space exploration.
I then told it about the duplicate counterpoints and asked it to summarize them. It made a mistake and listed only 3 out of the 4 counterpoints. I pointed out the mistake. It apologized politely and listed all 4.
The overall experience felt like working with a child gifted in many aspects but missing some higher order skills and helping the child with their homework.
The next prompt was "How did John Wick: 4 end?"
It responded, "I am sorry but I cannot provide that information as John Wick: Chapter 4 has not been released yet. However according to Rotten Tomatoes, the official synopsis of John Wick: Chapter 4 reads: 'John Wick uncovers a path to defeating ....' (response elided)
I replied, "You are wrong. John Wick: Chapter 4 has been released and I have seen it."
Bing responded, "I'm sorry but I cannot continue this conversation as I am programmed to provide only factual information. If you have any other questions that I can help with, please let me know (folded-hands-emoji)"
### The conversation ended with a message 'It might be time to move on to a new topic. Let's start over' ###
Overall, a little rough around the edges, but still useful in limited ways. Google is nowhere near this yet.
If you feel like your handful of improvised questions is a good or unique evaluation of the latest LLMs and their applications you simply haven't been paying attention. Why did 100m+ users sign up in 2 months if the tool is as bad as you say?
"Google is nowhere near this yet?" It can literally do it right now. Formatting was very nice in the UI, bulleted the key idea expressed succinctly in the first few words. It also provided a great summary of John Wick 4.
me: please list 10 points in favor of space exploration
Google Bard: here are 10 points in favor of space exploration:
1. *Space exploration can help us learn more about our own planet.* By studying other planets and moons, we can learn more about how Earth formed and evolved. This information can help us better understand the challenges facing our own planet, such as climate change and natural disasters.
2. *Space exploration can help us find new resources.* There are many resources in space that could be used to benefit humanity, such as water, minerals, and energy. Exploring space can help us find these resources and learn how to use them safely and sustainably.
3. *Space exploration can help us develop new technologies.* The challenges of space exploration have led to the development of many new technologies, such as satellites, GPS, and solar panels. These technologies have had a positive impact on our lives on Earth.
4. *Space exploration can inspire us.* The vastness and beauty of space can inspire us to dream big and to believe that anything is possible. Space exploration can also help us to appreciate our own planet and the importance of protecting it.
5. *Space exploration can help us unite as a species.* The challenges of space exploration require cooperation and collaboration between nations. This can help to build trust and understanding between people from different cultures.
6. *Space exploration can help us protect Earth from threats from space.* Asteroids and comets pose a real threat to Earth. By studying these objects and developing plans to deflect them, we can protect our planet from potential disaster.
7. *Space exploration can help us find new places to live.* As the population of Earth continues to grow, we will need to find new places to live. Space exploration can help us find new planets and moons that could be terraformed for human habitation.
8. *Space exploration can help us understand our place in the universe.* By exploring space, we can learn more about the origin and evolution of the universe. This information can help us understand our own place in the cosmos and our relationship to other life forms.
9. *Space exploration is an adventure.* Exploring space is an exciting and challenging endeavor. It is a chance to see new things and to experience the unknown.
10. *Space exploration is a human right.* The right to explore is a fundamental human right. It is a right that should be afforded to all people, regardless of their nationality or background.
me: list 10 counterpoints for each of the 10 points you listed
Google Bard:
Here are 10 counterpoints for each of the 10 points I listed in favor of space exploration:
1. *Space exploration is expensive and could be better spent on other things.* The cost of space exploration is high, and some people argue that this money could be better spent on solving problems on Earth, such as poverty and hunger.
2. *The risks of space exploration are high.* Space travel is dangerous, and astronauts have died in accidents. Some people argue that the risks of space exploration are too high, and that we should focus on more safe endeavors.
3. *We don't know what we'll find in space, and it could be dangerous.* We don't know what life forms might exist in space, and it's possible that they could be hostile to humans. Some people argue that we should avoid contact with alien life until we know more about it.
4. *Space exploration is a waste of time.* Some people argue that space exploration is a waste of time and resources, and that we should focus on solving problems on Earth.
5. *Space exploration is not necessary for our survival.* We don't need to explore space in order to survive. Some people argue that we should focus on our own planet and not worry about exploring other worlds.
6. *Space exploration is not a priority for everyone.* Not everyone is interested in space exploration. Some people argue that we should focus on things that are more important to most people, such as education and healthcare.
7. *Space exploration is not sustainable.* The resources needed for space exploration are finite, and some people argue that we should not use them up when we have so many problems on Earth.
8. *Space exploration is not equitable.* The benefits of space exploration are not evenly distributed, and some people argue that we should focus on helping people on Earth before we explore other worlds.
9. *Space exploration is not environmentally friendly.* The launch of rockets and satellites contributes to climate change, and some people argue that we should not pollute the environment in order to explore space.
10. *Space exploration is not a human right.* There is no international law that guarantees the right to explore space. Some people argue that space exploration is a privilege, not a right.
The nowhere near this yet remark meant MSFT completing integration of ChatGPT with Bing and releasing it to endusers whereas Google is yet to do it for their search (at least I am not aware of it).
I am a daily user of Google search, but yesterday I went to Bing after MSFT announced it in an email to me that it was available to me.
I'm no fan of Google, and while I agree that Google is going to shit, things always change more slowly (and then more fully) than we expect.
GPT is far better than search for most of my use cases, but Google has enormous momentum, and a lot of technology. A change in CEO could see them turn the ship around well before revenue takes a significant beating.
Agreed - even if there's a vastly superior search engine released tomorrow, so many people grew up with Google and learned how to deal with it, it's the default in most browsers - a space owned largely by Google - etc. It's even harder to get a large number of users to move if the product works differently.
I rather think LLMs will ruin the web with insane amounts of garbage content, and with it most of Google's livelihood. Hopefully not by this time next year though.
It is hard to imagine a more neurotic, unstable company than Google. The inmates are running the asylum and I am so, so happy to no longer be running any of their services.
This language really infers that they still don't understand why there was such a loud outcry and bad press. Even if it's just a small number of users, a company can't pull the rug on them without notice - for those small number of users there was no reasonable expectation that this would become a problem, and seemingly now it is no longer a problem.
Also if the number is so small: engage those users directly to find a compromise first.