WhatsApp, a Facebook owned app, will not give you read access to your own chat database - it’s encrypted. But they have an arrangement with Google where it’s will back up unencrypted to your Google account (though, Google won’t let you retrieve it ... only the WhatsApp app can)
Everything about this duopoly, their cooperation and lack of competition stinks to high heaven.
Wiping the WhatsApp chat history on the old phone before giving it to a new owner is also much more important. It often contains highly embarrassing or even incriminating details.
In ecology, it's rare to species compete to extinction or exclusion from a niche. It's much more common to observe segmentation. It's a lower energy state for birds to mate at different times, eat leaves from different parts of the same tree, etc., dividing the resources of their niche among themselves, than to force each other out of the niche completely.
The situation is analogous. I'm sure they'd destroy each other if they could, but it's obviously not happening right now. It's much easier to say, you can have your current users, we'll have ours, and we'll focus our energy on capturing emerging markets. Which is why you see them racing to provide free internet to developing countries and such.
Don't ask me what's leading WhatsApp's legal department to believe that this feature doesn't comply with German privacy laws. It's quite puzzling. (I would've rather expected a feature like this to be mandated by EU laws.)
As of now, there's no non-hacky way of exporting chats for German users. (Hacky ways include: dubious third-party software, copying each message individually, taking screenshots.)
The net effect is that if you're switching you have to redo 2fa for all accounts where you haven't separately backed up your TOTP codes.
You can store data on Google Drive that isn't accessible from Google Drive and only that app can read or write it.
But why would WhatsApp store it unencrypted on Google Drive, and encrypted (without giving you any access to the key) on your own SD card or local backup? It makes no sense.
Except ... Google and Facebook have agreed that WhatsApp stores the database unencrypted, and in return Google doesn't count it against the user's quota. Everybody wins! Facebook get to have free backup for users, Google gets to see all user messages and metadata. And the user gets rid of this pesky privacy thing!
 https://faq.whatsapp.com/android/chats/about-google-drive-ba... - "WhatsApp backups no longer count against your Google Drive storage quota." and "Media and messages you back up aren't protected by WhatsApp end-to-end encryption while in Google Drive."
The design is pretty clearly to support the bog standard backup use case of "something happened to my device, I got a new one, I want my old data".
It would be nice if they added an option to also encrypt backups, although I think it would be non trivial to design where the key material would live. (phone enclave? Multiple devices? adding/removing devices? etc) Not to mention just having the option might cause more harm than good if people turn it on and regret it.
I think their design philosophy here is likely "let's err on the side of smooth UX and let the 'end-to-end encryption is srs bzns' loonies go to signal"
They already support encrypted backups. In fact, they don't support unencrypted locally.
> The design is pretty clearly to support the bog standard backup use case of "something happened to my device, I got a new one, I want my old data".
They had, in fact, perfected the local encrypted seamless "something went wrong" workflow - they keys are delivered from the server (stored or generated, I don't know) once you've proved you can receive texts on that phone number. Without that, the local backup is useless.
And IIRC, they ALSO did the same with Google Drive. But then, one day, it stopped counting against your quota and the encryption disappeared.
> I think their design philosophy here is likely "let's err on the side of smooth UX
The UI was as smooth as it can be. It still is, for local backup.
I try not to attribute to malice that which can be explained by stupidity, deadlines or missing budgets, but in this case, there was a decision to remove encryption, that required negotiating with a rival. That decision had costs, and the process had costs. There is a benefit associated or this decision would not have been taken. I don't know exactly what it is.
On one hand - even after end to end encryption, we continue to see WhatsApp messages quoted in prosecution including in SEC actions and prosecutors are clearly getting access to messages even pre-arrest. The ability of prosecutors to compell unlocking after arrest in US is questionable. So how are they getting access to the messages if there's truly end to end encryption? Also there are any number of Governments in the important non-US markets that would simply ban WhatsApp if they didn't have access against a warrant (eg. See BlackBerry cases from some years ago)
On the other hand, no security researcher seems to have found or at least reported a back door in the WhatsApp apps that supports MITM undetectable to physical key verification which is the obvious weakness in an end to end encryption system based on centralised key repository system.
On the gripping hand, all high profile cases trying to compell tech companies to provide evidence have been against Apple. I've not heard of one against WhatsApp.
Ergo - the most like scenario is that this feature is deliberately built keeping messages un-encrypted so that when law enforcement shows up with a warrant or a muzzled warrant, they can just hand over the archive without weakening their app. People are convinced to opt in because it's the only way to change devices while keeping your messages. To give people comfort that unencrypted messages ate safe, it is sold as as "it's in your own Google drive"
But, you know -- they gave the same kind of info (and much, much more) to firms like Cambridge Analytica for free for years. So perhaps they didn't even get more in return.
Facebook's purchase of WhatsApp was OKd by EU regulators subject to some constraints, e.g. that they don't use WhatsApp data to enhance FB graph and/or gain other unfair advantages. FB has since admitted to using WhatsApp metadata in violation of that constraint (and didn't even get a slap on the wrist) - and perhaps giving the data to Google is also part of playing dumb. I don't know, but I'm sure it has everything to do with benefit for FB and GOOG, and almost surely at the expense of user privacy.
The Google Drive backup is most likely still encrypted, just not "end to end" because the key needs to be stored on a Facebook server.
That makes a lot more sense to me.
some serious cognitive dissonace is going on - either that or it's the effects of the news-cycle making us blind to it. Maybe we should think twice before applauding FAANG engineers in future posts because Technology isn't neutral.
note: a company can't be evil. it's just a group of people who can be evil and who hide behind the facade of the "legal person" statute. I'm happy to celebrate individual breackthroughs but let's not forget that without (metaphorically) nailing the culprits to the cross who are responsible for ICE contracts or privacy invasive tech there isn't even a point in complaining about all the horrid stuff these people (working for these companies) do.
It's like Kierkegaard's example of the mob who collectively intimidates but every member argues they have contributed virtually nothing, and so any sort of responsibility is simply dissolved.
Anyone who is responsible needs to be aware of the holistic system that they're part of and that they are enabling rather than just their own little world. That's no excuse.
Consider this menu of possible ways to be associated with a human rights violation:
- Do it yourself.
- Be a part of the chain of command that ordered it.
- Be commanded by the chain of command that ordered it.
- Be part of the same organization as the one that contains the chain of command that ordered it.
- Be a member of the society that contains the organization that contains the chain of command that ordered someone to do it.
- Be a member of another society, that could have tried to stop it, but didn't.
Where do we switch from participant to accomplice to bystander to "completely uninvolved?"
First they came for the socialists, and I did not speak out because I was not a socialist.
edit: In case it isn't clear: You just chose sides right there in your comment.
yes - and I've done incredibly well during it. I also know change doesn't come for free. the sad thing is the bigger the companies the smaller the spines of the individuals working there.
We all put thoughts into our own well being. When sufficiently comfortable or successful at that we start caring for people we interact with on a daily basis. When that too is a reasonable success we start doing stuff for people we interact with less frequently. At some point you can chose to go look for people who need our help, do something for the town, the city, the country, humanity etc There is tremendous joy to be had in this process and it builds character. I admire people like that.
People who don't do it are missing out. They in stead keep telling themselves they can't do anything for others knowing they are lying. The endless rationalization of the irrational takes a huge amount of time until they graduate themselves in the DIY course in psychopathy.
In their ego trip they end up feeling sorry for themselves. In their self inflicted blindness to what misfortune looks like the self pity knows no boundaries.
Also: no one loves me! I wonder why that is? They keep trying to spend money to buy happiness but all they get is cheap thrills. Some end up in a pathetic cycle where all they can do is expand their wealth, ignoring even their partner and their kids. They don't have time for that, they have important other things to do. I pity people like that. If I can think of some way to help them with their condition I will do the responsible thing.
But they also produced some really cool tech. Unix being the obvious one to bring up on this site.
That era is romanticized to hell (including by me), and Ritchie, Thompson, and the like are folk heroes. Despite being engineers at the subsidiary of a giant monopolistic entity.
Ultimately, for people who think the tech is cool, it doesn't matter where it comes from. In 50 years people will be romanticizing the era of the FAANG companies (more than they already do) the same way we romanticize bell labs.
That said, with FB/Google these days the bad weighs more than the good.
just because we made the mistake to let them become this big without giving them criticism doesn't justify why we should continue treat them with gloves. quite the opposite. they shouldn't feel comfortable at all in their position - they should feel like they're being held accountable (and act accordingly whether with rebelling internally or by leaving depending on the size of their spine)
On android? On iOS you can get a tarball of all your chats.
As much as I'm happy to pile on Google, Facebook and WhatsApp, this appears to be untrue.
The statement that "Facebook gives an unencrypted copy of your chat database to Google but will not give it to you" is true; I'm referring to the SQLite database. You can back it up locally, but it's encrypted and Facebook will not willingly give you the key under any circumstance (though it is possible to trick them by impersonating the app). However, the Google Drive backup is unencrypted. But Google won't let you download it directly either (although, again, you can trick them by impersonating the app).
Recently I was trying to switch my wife away from the free Keeper app to the one I use and backup to our own storage, they make it impossible to get your passwords without paying them or rooting your phone. I hand copied them one by one from her old to new phone. Honestly, screw any company that does this.
if that's the case then facebook can't "give you" the key, because they don't have it, it's supposed to be on your phone and only there..
Is there something i'm missing here ?
They make a copies of that plaintext database for backup purposes:
One on google drive (not mandatory, but it IIRC by default), which is still plaintext but not accessible to yourself.
One on your local SD card (if you ask for it), which is encrypted with a key that's on your phone but was sent down from FB servers. If you switch your phone, and try to restore that database, it will contact FB servers to retrieve the key.
> if that's the case then facebook can't "give you" the key, because they don't have it, it's supposed to be on your phone and only there..
Regardless of the backup encryption, e2e doesn't mean that they don't have the key; it just means that it leaves one phone encrypted and enters the other encrypted without being decrypted along the way -- unlike, e.g. regular email, which is usually encrypted in transit by TLS, but gets decrypted and re-encrypted at every stop along the way.
But you can export all chats one by one.
When you make a local backup, it is encrypted, and Facebook doesn't think you deserve a copy of the key or an unencrypted copy.
Or do you demand that any file under your control that contains chat should be accessible, rather than just one of them?
You can't download it without impersonating the WhstsApp app.
Do you know if the icloud backup is encrypted or not?
Haven’t seen anything like this before.
Convert all active whatsapp contacts to signal? 80%? 50%?
Same for Facebook messenger!
It needs to be uncomfortable enough - if they want to communicate with us and are minimally tech savvy ("click link to install app") they have to switch to signal.
I'm starting to send out messages over my morning coffee.
Let's do this together:
1. There are tons of replies in this thread. People have very strong feelings about signal.
2. Some people feel VERY STRONGLY that the pin numbers are stupid, and it has a terrible UX.
3. Some people feel VERY STRONGLY that the phone numbers are the simplest, most straightforward thing to use, and they would hate Signal if they didn't support it.
4. Some people feel VERY STRONGLY that they don't want their phone number used as an identifier, and it's a failure of Signal that they use them.
And it seems like many of these replies assume that their own position is the only possible one to take (or at least the only intelligent one!).
Imagine working on Signal and trying to create a product that pleases all these people, as well as even less technically minded folks, while having none of the usual analytics at your disposal to measure actual user interaction!
My hat's off to Signal. I admire their work but I don't envy their position.
At this point, none of us even use phones, my family pretty much use Zoom and at work everything is Google Meet, my phone function is basically a dead function at this point.
You can make the argument that I don't truly “owns” the domain name off of which my e-mail address hangs, either, but it's a much better situation than phone number ownership. There, at least, the constraints of ownership (rental) are clearly defined, as is my ability to move it between registrars.
My email represents me, and I fully own it with minimal upkeep costs.
Yes I think they suck, but do they suck more or less than en email? Not clear.
I'm a core contributor at Status, and the team I'm on is currently developing a desktop client to complement the mobile client. We have alpha builds for Linux and macOS, and soon Windows.
Status uses the Waku protocol.
One thing: I don't think it's unusual for a user to have little/no interest in the wallet or dapp browser. In that case they simply use it as a chat app that's focused on privacy and security while ignoring the other features.
The uptake? Like ~30% of the people I'd like to contact bother with it. The rest, it's even worse - we get stuck with SMS. And a lot of conversations I just miss, period, because they're groups of people in one of those other ecosystems (and missing those is a price I'm personally OK with paying, but not everybody will be).
The situation is just really bad. Signal is a good baby step forward, but it's not really the answer for most people, because they're sticky to wherever their network graph of contacts already is, and they're not going to switch just because the one weird nerd in their life isn't available there.
are you OK with THAT? You support children harassment by cops?
Throw some biased media page calling big companies bad. Not a hard thing to do today or write an opinion piece on popular news journal site and reference it. Medium works too.
We also need to cancel people for supporting these platforms.
And after it started insisting on creating a PIN each time I start Signal, I finally deleted it from my phone.
You can forget about security if you can't get a seamless user experience.
Plus, it's just a terrible UX for a SMS client. I really wanted to like it, but it's just too obnoxious to use. I've certainly not recommended it, I've had better luck switching relatives to Linux mint (seriously, I am not exaggerating). Creating a group MMS is annoying, and there's a nag screen if you try to call someone from a message within the app. I understand that most of these problems are due to the secure nature of the app, but trying to use it as your full-time SMS client is just too much of a pain. I'll likely still keep it installed, as the 1 thing it does do is make it really easy to transfer files/send messages between devices (send messages to yourself, then download them using the client's for other OS's).
Personally, I don't feel the UX is "terrible", I can use it to do most of the things that I could do in WhatsApp. My friends and I also tested out the video call feature and found the quality to be the same, if not better.
That has never happened to me.
The result is uncomfortable, since Signal never knows the other side removed it, so messages land in thin air. Several times did I have awkward moments when someone was angry "I thought you'd let me know if it was cancelled" - "I did - I sent you a text about that two days ago! On Signal!"
They can start by removing the atrocious phone number verification first.
Being forced to give them one of my 'most unique identifiers' (my phone number) was an absolute 'no'.
There are some good-looking and user-friendly clients
1. Shady addictive game requires full access to drive.
2. Shady game downloads all your chats with all contacts
3. FB is on fire since they allowed access to people's chats with others, even those who didn't download the game.
4. People get mad on FB/Google for not protecting their data.
Claiming that this is the reason is sort of like rearranging the chairs on the deck of the titanic when you can see the iceberg.
Much more likely it is to stop import/export to a competing app. The fact they will let you keep a local encrypted copy, but not an unencrypted one under any circumstances, is telling.
If we are to take the position that all users are tech literate and should be fully in charge of how their data is used, then even Cambridge analytica wasn't bad. People explicitly gave it access to data that was accessible to them. However, the uproar proves otherwise. People don't like it when they click yes yes and some random app has access to their social media data or messages. This is why Gmail also started requiring independent audit for all gmail apps.
SMS had the same problems forever, but isn't owned by a company, thus you can't blame xyz company if random apps access your SMS. No company got ever investigated by multiple nation states over that.
You still haven't addressed my main question though, how will a random app scraping your WhatsApp be any less of a scandal than Cambridge Analytica?
At least this time you can disable the SDK (at least one app developer has a kill switch for the SDK) unlike the last issue that happened during code initialisation you couldn’t skip.
"A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable." -- Leslie Lamport, 1987 (https://lamport.azurewebsites.net/pubs/distributed-system.tx...)
No, I don’t think there’s any “to be fair” in that, unless those SDKs have been officially EOL’d. Even then it shouldn't take down the whole app, Facebook should just break.
That this only happens with some SDKs certainly makes the problem less severe than last time, but only relatively.
There's no such thing as QA at Facebook. Really. Developers are supposedly responsible for all testing of their own code, but all of the pressure is to "move fast" so that goes about as well as you'd expect.
If you have a team run by idiots, then you won't have QA.
If you have a team run by adults, then there is plenty of QA. I mean its total shite that this wasn't caught. I'd expect much better, especially as this is the second time in n months. But then perhaps working for a company run by a dick who can't listen is getting to people.
Engineer in test. QA engineering lead. (https://www.facebook.com/careers/jobs/?q=QA) You must have leet scuba skills with searching skills like that.
AR/VR, Oculus, portal and workplace all have dedicated QA engineering teams. I'm sure there are more.
> Given your characterization of MZ
if it was only zuck. Zuck has two faults: he's not able to articulate any single thought inside of 15 minutes and he consistently throws out integrity products that would have stopped bad PR.
Its not just me, you should look at the micropulse. confidence in leadership has taken a good battering recently.
The whole juneteenth buisness, we had an entire _day_ of people telling us the problem. Senior management then sat next to them and said in reverent tones: "thank you so much [[dabs a tear]] for telling us your lived experience" posts a selfie and then does fuck all else.
Or, having revved up the company to generate hundreds of ideas to tackle abuse on FB platform, do shit all with them, not even acknowledge they exist.
Nothing to do with scuba, and if you did any operational work you'd know that it's in the category of tools that are often flaky due to lack of QA. It's also worth noting that the link you provide only shows 41 listings, and most of those merely mention "familiarity with QA" in some boilerplate for what's obviously a developer position. Leet reading skills you have there. Percentage-wise, there are fewer true QA positions there than we have E-10s. For a company that hires 10K a year, that's indistinguishable from zero.
> AR/VR, Oculus, portal and workplace
In other words some small percentage of the company, none of it near me. All of the systems that the profitable parts of Facebook rely on - to provision systems, monitor systems, store data, move data around, analyze data, etc. - rely entirely on developer testing. So here's an amended statement to make you happy.
"99.9% of Facebook has no QA, and the rest has really bad QA."
just over 10% of job openings have something to do with testing or QA. Of course QA is similar to software engineering, the whole point is to automate testing.
> [AR/VR, Oculus, portal and workplace..][..is] some small percentage of the company
yeah, go look at the headcount.
> none of it near me.
Now we are talking.
> rely entirely on developer testing
again, that wasn't the distinction I was making. It doesn't matter who's testing, so long as it gets done properly. Considering PE is a blended role. Its not a hard push to think that testing might be part of the role of a modern developer. Mind you, so is writing documentation, but thats not a priority either.
The first two times it took months to fix. It’s happening again currently, too.
Clear there are GAPING holes in their test suites.
Make a lot of money selling lifetime subscriptions, then have nowhere to go for cash next year: look at all that money! I'm very successful now!
If all you're measuring and optimizing for is "valuation" or "revenue" you're not automatically going to get a good product or happy customers...
Workaround for developers: Go to Facebook Developer page > Analytics and disable automatic Events. Apparently this doesn't solve the issue for everyone.
Issue in Facebook's tracker: https://developers.facebook.com/status/issues/17391881029111...
I couldn’t help but notice neither app worked even after a reboot, yet google maps did. I was thinking there was a carplay bug and then thought: “oh boy here we go, what are the odds, let’s check hackernews” and bam, of course it’s the Facebook sdk again
I’m thankful that googles iOS-chrome and google maps doesn’t use the Facebook sdk for something...
I want a central document with clear steps on how to mitigate. Currently the best option for documentation seems to be wading through comment threads, which is very frustrating.
Or if you must, use their oauth flow and API but _don't_ include code of theirs you don't control directly in your binary. It's just asking for trouble.
We had nearly 100,000 users logged into our app via Facebook login so we were basically strongarmed into complying. We somehow pulled it off before the deadline for both Android and iOS but felt really dirty afterwards.
>It turns out that by just including the SDK with your app, Facebook runs hidden code on launch. (FBSDKApplicationDelegate.m)
1. It is required for a "login with Facebook" feature, apparently. I'd presume this goes for other feautures that integrate with facebook, such as "invite from your friends" or "share on facebook(messenger)".
2. It gives access to metrics, when added to your own, greatly enrich your own data collection.
3. It is required when working with Facebook ads - though I would not know how this ties in with mobile apps: on web it is clearer (where you are required to host "the pixel" on your website in order to have ads on facebook link to that website.)
4. It is feature rich, so as a pure SDK, it offers neat utilities; though for each of these, there are better alternatives found in separate libs.
I don't remember apps listing the SDKs they use. As a humble user, how do I voluntarily choose whether to download an app or not based on their usage of Facebook's garbage?
Edit: You can't have it both ways. You can't remove a bunch of power and information from the user in the name of "safety" and "UX" and then claim that users are making a bunch of choices they can't possibly be informed about "voluntarily"
There are some intricacies on iOS surrounding null values and serialization, and as a developer it is important to understand that you may encounter NSNull. As a standard practice in my company, we type check every remote value before calling anything on it. Seems like that would have prevented this issue.
It makes me wonder what all those engineers at FB are actually doing... ? Every time I tried to integrate or look into the FB SDK for simple functionality it was a total clusterfuck.
A bit of snark but also maybe stop trying to use a data collection engine to leverage anything for yourself?
Source: have had that issue before
Also maybe it’s time for them to start rewriting things in Swift, where this would be much less likely to happen
If Apple wanted to make money , it should make it harder for advertising supported apps to make money and force apps to either not exist or get people to pay for them.
I would be okay with that.
What has this industry become? How are we so goddamn inept at writing software? This is an industry where we can automate testing - not many other industries have this capability - and we still don't have a simple test regime of "check if the app opens" for any of those apps. Somehow, at the most valuable software companies of the world, nobody has set up a system that makes sure that you can't introduce a change that renders these apps unusable?
WHAT THE FUCK.
I have always been afraid of software engineering becoming a trade similar to medicine or law or (regular) engineering where significant barriers will be put up before you can enter the industry, but this sort of shit makes me feel like it could be for the better. I will have to go and get a degree, but that's the price I have to pay I guess.
Please don't use uppercase for emphasis. If you want to emphasize a word or phrase, put asterisks around it and it will get italicized.
A couple of days later I caught up with the people allegedly responsible for this stuff in the bigger review meeting. "Shouldn't you open a SEV because you are shipping stuff that can't possibly work and aren't catching it pre-release?" The response was a lot of hand-waving and stupid manager-speak that boiled down to "no, go away".
I basically wrote them off at that point. If they could act that way and get away with it by way of implicit acceptance from upper management, that's just the way it was going to be.
I see the quality is as high as always.
Facebook's own apps did not break since they are on the latest version of FBSDK.
What Facebook needs to do is to test whether their changes don't break other people's apps.
2. Client-side SDKs can be designed in a way that even if the server-side changes (which it shouldn't), it does not cause the whole application to crash. This can of course be hard depending at which level the SDK operates and how integrated it is into the app.
3. Are there no contracts between Google/Spotify/etc and Facebook? If I were FB and there were contracts on the line, I would make sure that there is no way something like this can slip through (automated testing).
I'd argue this is at the very core of why you test. I mean this problem is as simple as (n+1).m vs n.(m+1) in terms of versioning vs revisioning.
If that is not being tested then I'd say Facebook isn't actually testing at all - at least not in a practically relevant sense ...
... despite the shit-load of YouTube videos from FB devs lecturing about CI/CD and what not.
Other than remove the SDK, of course.
Fool me once, and all that.
Bingo! App developer voluntarily pulled in a dependency which they didn't fully understand. When you bring in a dependency, it's not still someone else's code/problem, it becomes your problem.
If hacker news goes down while you are writing this comment, it is your responsibility to ensure you can still post your comment despite the service breakage.
Many of the apps crashing are using Facebook for a portion of their app, such as one login option.
Letting that take down the whole app is insufficiently defensive coding, whether in the app, sdk, or glue between them.
Absolutely, but the problem is that in order for you to use these features, you're supposed to use the official Facebook SDK. The additional problem, is that Facebook has no interest in defensive coding, because if you're not interacting with Facebook in an app, there's no benefit to Facebook for you using that app at all.
(Am not a mobile dev, but it wasn't exactly hard to find that, either).
2. Throw exceptions
Don't tell me "No amount of testing" just because you aren't responsible for your own dependencies.
Removed the SDK myself from a top 100 App Store app.
Recently tested an enterprise marketing app with an SDK ... and we tested if the endpoint wasn't available. Nothing happened -- it worked just fine. Failed quietly and gracefully -- the user would never notice.
"We" know how to do this.
Facebook hasn't done anything to fix this problem since it happened a few weeks ago.
But it's Facebook -- I hope they keep it up. Can't wait for them to AOL themselves.
But if you're Facebook, and you're pushing your SDK as a great solution and it's embedded in high profile apps and you are letting this happen a couple of times, I get to make fun of you. (a dollop of sarcasm, to be sure..)
I can almost guarantee that the programmer who wrote this had a degree and passed a very rigorous testing process to get their job at Facebook. Doesn't mean anything when the company tells them to cut corners to iterate faster.
Stop blaming engineers for the mistakes of management.
The app makers cannot easily prevent the crash since it is happening in Facebook provided code. Perhaps the only thing they could do is to code in kill switches. Another possibility would be to modify the Facebook code. That introduces maintainability issues.
The crash happened because of a server side change that triggered roughly the Objective-C equivalent of a null pointer exception.
Objective-C is an old unsafe language.
Client-side libraries have a poor history in the industry. They are often written by server engineers who aren’t expert on the platform.
For instance, if it was really a static load method that called the server, then this was a poor design decision.
Furthermore, weaker server engineers are notorious for breaking backward compatibility. They don’t design nor build in tests to ensure backward compatibility. The backward compatibility must include backward data compatibility.
The above issues are indicative of organizational issues.
Facebook is a little bit of an outlier. It was notorious for badly written client libraries stretching back to the Facebook platform days. The joke was that their libraries may have been written by interns even though so many companies were dependent on them.
The Facebook libraries and the ilk are used for sign on through Facebook and ad attribution. The trade off an app maker needs to make it a whether to run ads on Facebook and whether to support sign in.
Facebook is not the only big company with these issues. Google’s equivalent also crashed albeit at a much lower rate due to a much more convoluted issue.
Apple introduced the Swift programming language to address the deficiencies in Objective-C. That these companies are running into these issues makes me wonder if they never transitioned to Swift. They are certainly way behind the latest version of the libraries.
To summarize, it is a cascade of issues some technical and some organizational rather than a single coding error. These issues span companies.
> Objective-C is an old unsafe language.
This was a safe crash, not an unsafe one. Swift can and would crash the same way.
Looks like the server passed empty value in JSON which got converted to NSNull in Objective-C. It's a common mistake to assume null is returned instead of NSNull.
The Swift equivalent of the JSON library should return nil which you then need to explicitly check for non nilness before using unless you use the unsafe exclamation pint.
Welcome to Apple Swift version 5.3 (swiftlang-1184.108.40.206 clang-1220.127.116.11).
Type :help for assistance.
1> import Foundation
2> // IIRC this used to throw an exception.
3> // Now a respondsToSelector: check is silently inserted and we get a surprise nil.
4> (Int?.none as AnyObject).count + "call Swift's bluff".count
Fatal error: Unexpectedly found nil while unwrapping an Optional value: file repl.swift, line 4
2020-07-10 18:03:43.815959-0700 repl_swift[73029:930570] Fatal error: Unexpectedly found nil while unwrapping an Optional value: file repl.swift, line 4
Execution interrupted. Enter code to recover and continue.
Enter LLDB commands to investigate (type :help for assistance.)
I think we are in general agreement?
This one was a good one because the app cache was poisoned so the only solution was to ask customers to reinstall the app (after Google deployed their fix, which took a while) or upload an update that nuked the SDK cache on startup.
This would still happen because A) the programmers that made this mistake probably have the knowledge to pass any certification exam you can throw at them, B) most of the time we are not just building something that has exploded 1000 times before, we are creating new stuff.
The engineering process was the issue here, and sadly the only way to prevent this is not to make personal barriers of entry higher but to slow down the process with more QA, testing or whatever you can throw at it. In other words, make software more expensive. And keep in mind that this would only give better numbers but it would not solve the issue either (see Boeing 737 MAX).
Very unlikely. You probably don't realize how tough some professional exams are in some fields. We are talking dedicated months of study here.
> B) most of the time we are not just building something that has exploded 1000 times before, we are creating new stuff
Most software out there is just CRUD stuff, not really new stuff...
Wow who would have thought no standards, free access to learning, fake-ass meritocracy, and exorbitant profits and market control would have led to such a thing
Pick a standard, whether it's a degree program, a cert, or apprenticeships. Hurry up! Just give us a solid target to aim for.
There's a good chance that the one who wrote this bug had a degree.
Surgeries in medicine still get botched despite all of their degrees and barriers. Degrees don't produce good code, good developers produce good code.
> I will have to go and get a degree
Was not aware that having a degree made you immune to writing bugs.
This will be a fart in the wind and lessons will be learned.
The industry seems to have very low standards. Let's let these mistakes happen over and over and let's just patch them as we go. No long-term thinking.
Someone should collect all these horrible incidents and start a university course on just plainly what NOT to do.
Don't sit in an ivory tower and throw stones in glass houses. Can you guarantee that your own SDLC house is in order? Because it might be, right up until a sneaky edge case nips you in the butt, and then suddenly it isn't.
Why should we just sweep this under the rug and call it a day? There is something very important that can be learned from this. Apparently, FB themselves have not learned, because this is not the first time this has happened. Why was this allowed to happen more than once? Did Apple, Google, Spotify, Facebook etc not take any actions?
"Shit happens" is one way to look at it. Another way is to question whether we as software engineers are really doing our job properly.
> FB themselves have not learned, because this is not the first time this has happened.
They have tens of thousands of engineers likely split into hundreds of different teams/microservices focussing on different parts of their software stack. A ton of them are new to it and nobody knows every part of the stack so shit can happen.
What is the biggest engineering organization size you have worked for and what was your uptime?
The linked issue makes it very clear that the "tiny outage" affects literally every app that has the Facebook SDK merely linked, for a frankly dumb reason (static initialization) that they should have learned from the last time it happened.
It's not a SaaS. "Uptime" for a linked library is stupid.
They're using static initialization and it bit them in the ass again. They should stop using it because it is prone to things like this which result in apps not launching at all. I'd like to see them commit to stop using this method to initialize themselves, and take responsibility for the crashes. Then, IMO, we'd be good.
In a scenario like this, I'd hope they'd want to make this bug's recurrence impossible.
Here is our Android config in case anyone's interested:
Like they work in many other companies.
In any case, this is not about being bright but about being diligent.