In case you only read the headline, it was confirmed to only happen if the device's region was set to China. From the statement:
> To accomplish this task, Safari receives a list of websites known to be malicious from Google, and for devices with their region code set to mainland China, it receives a list from Tencent.
From Apple's statement, Tencent's API functions exactly the same as Google's k-anonymity model, which you can read more about the api[0] and how it works[1] (also via this JAMIA paper[2]).
It worth noting, for some features like hiding Taiwan flag, the logic is to check both system setting and hardware sales region, it will be on if either is true.
On the softer side, shutter sound can’t be muted on the camera app for JP sold iPhones, I can’t find any offical doc (not customer discussion) mentioning it.
edit: there was some doc, just with a vague “shutter sound might be disabled depending on your region” mention.
it's a ridiculous law . You want to take a picture of your sleeping baby without waking her. Sucks to be you. Want to take a picture of your cat or dog without them changing their pose in response to the sound. You can't. Want to take a picture at a wedding without distrubing the mood? Or bad for you. Want to take a picture at a fancy quiet romantic dinner without annoying all the other guests? Can't do it.
In the meantime any pervert (the claimed reason for the law) can use any non-phone camera or for that matter any other camera app than the built in one
Even if there is, it will only be at the beginning. So you can circumvent the intent of the law (e.g. filming the police) by starting to film yourself, pretending to stop and then point the camera at the subject.
I think you are mistaken. There is no formal law AFAIK but it was just an agreement made by companies in Japan. You are free to order a phone outside of Japan. This was a huge issue in Japan when phones started getting cameras and so everyone made a handshake agreement.
In Japan, the difference between formal law and cultural law with respect to their enforcement is often not a distinction worth making (as John Stuart Mill pointed out in On Liberty⁎). Just ask women how interested the police are in dealing with rape or domestic abuse.
Only yesterday I was reading a Reddit thread where someone new to Japan asked whether to inform the police of the apparently violent arguments his neighbours were having and the responses varied from "don't bother, the police won't care" to "make a noise complaint, at least the police will be interested". That's the least scary anecdote I have.
As far as I remember Jphone, the company first heavily advertising camera phones, forced the shutter on its own to preemptively clear itself from any potential issue.
Every phone carrier from there had the choice to forgo the mandatory shutter sound but risk backlash if anything were to happen, as Jphone set a precedent. And they choose the safety move.
When the iPhone came, it would have just been a marketing quagmire to go with a mutable sound.
iOS has the issue that you can not select default apps, unlike Android. So yes you can use a 3rd party app but you can't make it the app that takes pictures when the phone is locked nor can you make it the app that gets launched from the accessible from anywhere control panel. In other words, their is friction using a 3rd party app.
But of course that's also part of the issue. If 3rd party apps don't have to make a sound then what was the point of the law? The perverts can just use a 3rd party app and 99.999% of users have to deal with a stupid for no actual protection.
And someone always finds a way to get a device which can make photos silently. For example by importing it, modding it, using older devices, etc. It is not solving anything. It is simply a stupid law against users.
Ya know, any non-pervert can use those non-phone cameras for all of the use cases you mentioned too... Is taking pictures of your fancy dinner or jumpy pets really so hard to do with a superior and low cost dedicated camera? The results will be superior for a fraction of the price!
Here's the thing you refuse to acknowledge (or perhaps didn't think through before writing) -- when someone is holding a P+S camera or bigger, everyone knows they're taking a picture. But when someone is holding a smartphone, you assume they are not taking a picture because 99.9% of the time phones are not in camera mode.
The point wasn't to ensure that all cameras made noise, it was to ensure that a phone couldn't be used as a secret camera in public.
Seriously, I struggle to parse the logic here: Do you think that creepers are taking actual full size cameras into subways for creepshots? Or are you thinking they're using "Spy Camera" type devices? Because the people the law was meant to deter generally aren't that level of creeper, and their creepshots are a crime of convenience not a premeditated act of sexual assault.
The argument is that the shutter sound inconveniences regular users while doing nothing to prevent the misuse it claims to prevent. Which part did you miss?
I missed the part where every creeper alive circumvents the policy without inconvenience but every other user alive can't.
I require consistency, so either EVERYONE is inconvenienced, or EVERYONE can circumvent it without issue.
The part I missed is the justification for inconsistent and illogical arguments where magically your ability to be inconvenienced is based only on your desire to be a creep.
Refusing human rights is a touchy subject, especially when the violators of human rights pay so well. The corrupting influence of money is becoming more and more widespread. Pretty soon, perhaps the only way to avoid human slavery will be direct payouts to the powerful, as if that isn't slavery already. Perhaps we're already there.
On a recent story (perhaps the most recent popular one on this topic), a former Microsoft employee said they maintained an internal database to keep track of what was necessary for each country. I’ll see if I can dig it up.
No, you'd need to purchase the device outside of China and set the locale to something else.
In the safe browsing example, you'd just lose it completely because it would be replaced by Google's safe browsing list (also used by Chrome and Firefox), which you couldn't connect to from within China. So the end result is the same as turning off safe browsing.
You definitely haven't been in China. Even when they blocked some VPNs, Apple devices are a million times better in China because it has the official app store.
You cannot begin to understand what it means to be forced to use one of those non-Google Play stores. Even if you set your device to English, Firefox will be in Chinese and you get a weird Skype version as well.
I mean it is a pretty interesting experience living Google-free on Android, and you could resort to APKPure to install some stuff like Whatsapp. But honestly, unless you get a Xiaomi phone or equivalent that you can switch to an international ROM, the whole experience is pretty annoying. Spam calls and text blockers on Android are the main highlight though.
If players that big are in the game then no, F-Droid is a dangerous consolidation of power. They really shouldn't sign apps themselves and only sign over reproducible builds.
I really never digged into this, but isn't they already provide an option to do this? [0] And there is even an option to setup verification server [1]. I get it's not perfect, but they have limited resources and someone have to work on such feature full time.
It started as a free speech Twitter alternative aimed at republicans. But along with republicans and free speech activists they've attracted right-leaning extremists that were banned from Twitter for a plethora of good reasons. As a result of those users being left unmoderated, Gab gained a certain reputation.
Gab's apps weren't accepted to either app store, they were booted by several hosting companies, had to switch domain registrar and their stripe account was suspended. Switch to a Mastodon fork is a recent development.
Gab was a platform advocating "free speech" and as is the meme with those types of platforms these days, it was filled by people denying the holocaust and claiming all black people operate better as slaves. The list is not exhaustive, but you get the idea.
To my mind, no good-faith claim that such things do not need to be ostracised exists.
Them being on Twitter is bad, but being on their own platform is arguably worse.
I fully agree with this. It will end up being a self-fulfilling prophecy for those arguing against true freedom of speech. When you deplatform a viewpoint entirely, then the people with that viewpoint are going to coalesce onto a much smaller platform, which will end up being an echo chamber for the viewpoints that the rest of the world desires to quash. Then, it will end up becoming an argument to prevent such echo chambers from even forming in the first place, which would effectively make it impossible for smaller players to get an inroad, as they'll just be preemptively accused of being inherently alt-right.
These people don't understand that, when you centralize control over avenues of speech like that, then one day that centralized control can do a 180 and start banning any type of speech, and not just "harmful" or "problematic" speech.
It's literally the same type of tactic that China uses to suppress dissent, just on a different scale (and with a different culture behind it). It's so transparent, yet the prevailing "public narrative" is still highly supportive of deplatforming.
This argument is compelling on the surface, were it not that Europe has had laws against this kind of speech for a fairly long time now and it has made nothing worse. The place where it really is worse is the US, where even people who are not fascists seem to imagine that it is important to give them a "platform".
F-Droid will be blocked by the Great Firewall as soon as it becomes a popular way of circumventing censorship. On top of that, there is a real possibility that most Android phones sold in China are backdoored by the government in some way.
F-Droid allows many ways to update without using its main server - from alternative servers to transfer via NFC/wifi. Perhaps traffic shaping can detect typical F-Droid traffic, but a simple IP block won't do.
Of course there's free software activism in China and there are apps by Chinese developers available via F-Droid. But that doesn't change anything for users who aren't free software activists. They either use the app store preinstalled on their device, or download APKs from their search engine (which often boils down to the same thing).
F-Droid would need coporate backing or SEO to get popular in China, not free software activism.
This seems like a situation where sufficiently effective free software advocacy will be persecuted. Will be interesting to see how it pans out, if at all.
Sideloading would not fix built-in malware that cannot be removed. You would have to flash the device with a ROM like LineageOS, at which point you would have a FOSS (excluding drivers/firmware) system which you can (mostly) trust. F-Droid is a repository / app store of good free Android software apps that you can also mostly trust, which you could then use to add functionality.
There's no zh_US (or zh_UK, etc.), so any mainlander living overseas and still use Chinese as the preferred language on their iPhones will get their IP address sent to Tencent. There's not much difference here.
Average users aren't aware of its existence. Also for a lot of mainlanders living in the US they won't prefer to set their region to US because that means imperial units instead of metrics units.
Well, they're handing over the URL prefix hashes, which Tencent uses to look up the real URL list that it returns to Safari. Conveniently, Tencent can also log this hash/list identifier and use whatever algorithms it likes to build a user profile.
I doubt Apple is actually "handing over IP addresses" - rather it's a feature of connecting directly to Tencent via the IP internet. Though they could conceivably provide some sort of proxying, caching, and batching if they wanted to improve privacy.
Similarly Apple isn't "handing over the timestamps" - Tencent can just look at the clock.
Yeah... this kinda seems like an overly sensationalized piece that's just much ado about nothing. From what the article claims vs. what Apple responded with, it sounds like they're interpreting Apple's statement in the most negative manner possible. What's probably actually happening, if Apple is being honest about how similar it is to Google's use of the prefixes, is that the prefixes are sent to Google/Tencent and a bulk list is sent back. Safari checks the list against the actual hash completely locally and blocks whatever matches the list.
So, for example, if someone wanted to visit freehongkong.com with their language set to mainland Chinese, Safari would hash the URL which would give something like "ba7816bf 8f01cfea 414140de 5dae2223 b00361a3 96177a9c b410ff61 f20015ad" and then Google/Tencent would send back every hash and the URL that matches only "ba7816bf" which could be in the order of thousands of URL, or even potentially millions. Safari would then check the URLs that are returned to see which one is the actual matching URL and it blocks the content in the browser without sending any more information to Google/Tencent.
Unless Tencent keeps an active profile and uses machine-learning to create models for every single user of its systems and attempts to tie those models to IP addresses that are assumed to be static, this still maintains complete privacy for the end user. I fail to see how this method is in any way equivalent to what the article is claiming.
I think it's reasonable to be a little concerned. The IP address does not have to be the identifier, it's just another datapoint for the model. For example, how many people in your residential DHCP pool visit Github and Hacker News every day? Whichever IP has those hashes in its pool is probably "you", so eventually a model can be built that watches you move from IP to IP. (Eventually there could be enough data that it doesn't even need to use the "same DHCP pool" as a heuristic, and it can watch you move between home and work or even on your cross-country trips.)
I think it would be difficult to get a strong signal as to what sort of browsing you're doing from this information, but I also think that if life and death ride on what websites you visit, you should turn this off.
Something I should have added to the comment during the editing window:
You also have to be careful about targeted malware, which can give up far more information than these hashes. So I am not really sure where the balance is. It is more concerning when Tencent is running the service instead of Google. A well-targeted ad is better than being disappeared.
Wouldn't tencent, having a list of politically undesirable domains, already know their hashes? Now the browsers that visit them phone tencent to let them know. This is useful when the actual http(s) transfer is under vpn wrapping.
They would definitely have a list of the domains but a hash isn't just generated from the domain root. They would have to have a hash generated for every single URL and then they would have to collect every single instance where a hash came up as a "hit" for that hash. They'd then have to model these out to determine which hashes were being hit that were related from a specific IP address which, in my opinion, would be very, very computationally expensive.
A few permutations of each would do it. http://foo.com, with /index.html, /favicon.ico, etc. You don't have to match all of them to mark a user (who in China is 1:1 to IP, IIRC) as visitor of undesirable site.
That doesn't make sense. http://foo.com/index.html would hash to something completely different from http://foo.com/favicon.ico. On top of that, the prefix, which is only the first 32 or 48 bits, would match for thousands of other URLs whose total hash would be completely different. Additionally, hash collisions are a thing so 2 completely different inputs could potentially return the same full hash, much less just the prefix.
You would need a very, very, very large database to keep track of all the permutations necessary to determine a domain name.
I think OP means that the collection of visited prefixes can be indicative. If foo.example.com, bar.example.com, and baz.example.com are "undesirable" sites, and you have requested hash prefixes matching those for foo.example.com/index, foo.example.com/favicon, bar.example.com/index, bar.example.com/favico, baz.example.com/index, and baz.example.com/favicon, you might be able to start making educated guesses that the user is visiting such sites. It really depends how much of an overlap for "acceptable" browsing falling into these buckets there is, or if you can weight certain prefixes as likely to be indicative of visiting "undesirable" sites. You might not able to say this user visited this specific page at this specific time, but you might be able to fingerprint a user as visiting a particular category of sites.
Furthermore, I'm unclear how exactly Safe Browsing is implemented, but if it checks every resource on a page, the set of prefixes for all resources on a given page may leak more data than one query on its own, so you can look for requests for that set of queries in a short time frame to identify a particular page.
While that might be true, this feels like it would be a long way off. The prefixes could potentially match millions of sites and the results are only then matched on the client side. The server would never know which prefixes are the “actual” hit and which are obfuscated prefixes for completely different domains.
Simple: the service provider visits these sites itself, gets hashes of all the resources for the homepage and relevant links (or just spiders them). With the set of hashes, look at prefixes. Users who request one prefix in that set aren't suspicious. Users with many prefixes in that set are.
This form of surveillance would be a game of bingo: for a given "bad" domain you have multiple individual URLs that a browser would hit. Put a few of them on the trigger watchlist, grouped by domain, and when a browser hits some threshold number of the same group within seconds the algorithm shouts "bingo" because chances are this won't be unrelated collisions.
Note also that mass surveillance isn't partucularly concerned about false positives.
The safe browsing hashing algorithm takes that into account. You are supposed to canonicalize and simplify the URL. For example https://example.com/some/path?k=w#hash would have the hash removed, query parameter optionally removed, then path components optionally removed, and finally sub domains removed up to the public suffix plus one, and produce multiple hashes (up to 30 of them).
Or the hashes of all the resources loaded by those pages. That collection of prefixes is going to start getting unique quickly.
Edit: example:
P(baduser | hashA )= ~0
P(baduser | hashA and hashB and ... hashF) = 0.75
One prefix doesn't do it, but a bunch together in a smallish time period can fingerprint a site quite well. This is analagous to browser fingerprinting.
If I understand correctly, the full URL is hashed (or at least the first n bits of the full URL). It really isn't computationally expensive to work out undesirable URL's being visited because the undesirable domains are known, and it would be trivial for them to periodically crawl the undesirable domains for all known internal URLs and build a dictionary of hashes.
In my mind, the big deal is that it means a company is actively facilitating governmental mass surveillance. And yes, I believe China does maintain a record for every citizen, and is using machine learning to score them.
How do you get that from what's happening here? Apple is actually doing the exact opposite of that. They're obfuscating traffic prior to sending it out to the government entity (Tencent) and only sending partial matches of the obfuscation and then processing the return completely on the local machine. If anything, that appears to be them actively fighting governmental mass surveillance. Even if they do maintain a record of IP addresses, the model for something like that based on URL hash prefixes alone would need to be staggeringly large. They'd probably need more storage and computing power than is available right now.
I'm really curious how you've come to that conclusion and I mean that sincerely. What makes you say that?
They are sending data to what is essentially a state-owned company that can facilitate building user profiles (with enough compute) and is, besides that, explicitly used to censor the web.
What is not to understand?
Apple should send precisely nothing to the CCP if they care about their users privacy on principle.
People unfortunately don't get that and are being slow cooked into accepting being spied on.
I paid for device and then you prevent me from doing perfectly legal action and then notify authorities about it. That's what it is, and coming from privacy hero, Apple.
The purpose of this system is to censor the internet, no? Isn't that enough?
But that being said, I do not believe that hashes are enough, because all they really need is to know you visited one (hash of) a certain website out of thousands. They do not need proof, and they don't need it to work reliably, and they do not need the model to be precise or good.
For the system to work in China, this is sufficient. It need not be perfect, it just needs to loom in the background.
People will do the rest themselves.
This, I think, one should not forget, especially if you really, really don't see an issue with what Apple is doing.
No it wouldn't. If, as an example, google.com and piracy.com both resolved to the same hash prefix, how would that help you at all? The whole point is that the server sending the blacklist doesn't get the details of the page that's being visited. piracy.com/info, google.com/searchingstuff, and whatthefuck.com/isthis might all have the exact same hash prefix. They don't because I just pulled that out of my rear to use as an example but there's no way for someone to ascertain the content of a website just form the prefixes of these hashes. The browser doesn't even send the details.
Yes it would. All China needs is a correlation. Most users don't input URLs manually, they follow links. So if a user starts from 'taiwan.gov' and goes from there to 'freetibet.org' and 'IhateXi.com'* the firewall would eventually detect from enough matches - even matches with partial hashes - that the user is a 'bad-thinker' and deduct social credit accordingly.
Apple could have tried to avoid helping the Great Firewall (e.g. a proxy could hide times and anonymize IPs) but chose not to.
* fictional names inserted only for example's sake.
I’m not sure I’d call warning people about malware “censoring the internet”. It’s the same system in use in the rest of the world through Google SafeBrowsing. And not just Safari, most other web browsers have some flavour of this as well. But that Google API is not available in China so they use the closest equivalent. AFAIK, the warnings can be bypassed the same in China as in the US, so anger about censorship here seems odd to me.
It’s absolutely not an “overly sensationalized piece” when apple does nothing to offer ip-based controls, such as offering the user a choice of which ips to connect to. You have to install little snitch or get super comfortable with packet fence. Apple certainly has the budget to figure this interface out and has for decades but has not. The issue is not capability but chosen business models.
These hashes also reveal domain names. Most users visit many URLs on a small set of domains. If a user requests 1000 hashes that all can map to Reddit, it's very likely that the user is indeed reading Reddit. Another way to look at it: if the same person appears in a crowd on hundreds of photos, it's trivial to notice that there is something special about this person, even though in all cases the person was k-anonymous.
Except that the problem with that is that these hashes likely won't reveal domain names.
For example, the hash for reddit.com/r/gifs would be different from reddit.com/r/funny and so the prefixes would be different for both of them. Unless the requested hashes are saved for every single user, it would be way too computationally expensive for them to get anything useful out of that. Not to mention the fact that hashes would return the same prefix for any thousands of URLs. Narrowing down which domains those URLs are rooted on would be incredibly hard.
> Unless the requested hashes are saved for every single user,
I don't know why or what part you think this is that hard. Do you think a map from User -> set [Requested URL Hashes] is hard to build? Or that building the URL Hash -> set [possible domains] is hard?
Maybe I'm missing a piece of this.
Building something simple to start guessing domains visited seems pretty easy. If a user has 10 URL hashes and the same domains show up in each hashes' possible domains you're probably requesting pages on that domain. If you're lucky and all the pages from a domain fall into a single hash, all it takes is two or 3 hashes from known outbound links to show up to confirm this.
It's not foolproof but hardly infeasible? Or maybe I don't fully understand the algorithm.
You’re ignoring that it’s not the full hash, just the prefix. The prefix could potentially match millions of URLs many of which would be duplicated with random pages from other URLs. You would need a very, very extensive model of every single IP that was mapped to each person and that would not be trivial. That’s exactly why the actual matching is done on the client and not somewhere else.
> Unless the requested hashes are saved for every single user,
It's 4 bytes by request. Google is keeping my whole history, much more than 4 bytes by request, and they do it for advertisers. I have no trouble believing a company partially owned by the government could afford 4 bytes by request.
Let say it's a billion address for each 4 bytes. You do 2 requests, how much of them will be on both list? Let's be generous and say half! You would only have to visit 30 uniques pages on that URL to find the domain. How often do you go on 30 different pages of a website? I feel it's quite regularly.
That's for a billion matches for each 4 bytes prefix, in reality it would be much less than that and there would certainly be much less than 50% matches between each prefix.
You can even do it in reverse even more easily. Go to a bunch of forum that you want to silence the users. Find their URL which allow to post on it. Get the 4 bytes prefix. Now you got a bunch of timestamped comments with username, and they are most likely pretty unique in the Tencent database. Now you found which IP is for which username.
Yes, I do. I linked to that very same site. The prefix is the only thing that’s sent and the hashes are computed as unique for each combination of URL. The docs explicitly say that http://evil.com/foo is unique from http://evil.com/foo?bar.
If you have the computing power to brute force sha256 then there are few security precautions a user can take against you.
That opens up forging SSL certificates, signing Debian packages, and a whole slew of other things that you could pull off with more computing power than has existed in the whole of the universe.
Suppose a public PGP key is communicated in the URL.
I don't think there are enough PGP keys in the world that simply trying those keys in a certain prioritized order wouldn't find a lot of keys.
This way you could recognize who someone is talking to based solely on their 'malicious browsing' send hashes.
In general, anytime you know a secret in the URL comes from a low (min-) entropy distribution you can use the truncated SHA-hash as an oracle to (approximately) figure out the secret.
Obviously, if the secret is a 128 bit session cookie, you won't find that by brute forcing. Heck, in that case your 32 or 48 bits of hash prefix won't even be enough to uniquely identify the session cookie.
When you get to a 64 bit secret (say again a session cookie), with a 48 bit prefix, things already get dicey. It seems possible that some state actors can already brute force 64 bit secrets, and with a 48 bit prefix, you'd get about 2^16 ~ 64000 (more, but dont wanna compute exactly) candidate session cookies. Those can actually be tried at the website to see if they work.
>I doubt Apple is actually "handing over IP addresses" - rather it's a feature of connecting directly to Tencent via the IP internet. Though they could conceivably provide some sort of proxying, caching, and batching if they wanted to improve privacy.
I vaguely remember that back when maps.app used google maps, the app would contact apples's servers (presumably a proxy) rather than google's directly.
I feel like the entire tech world is overthinking this. First, the Great Firewall already blocks the known "dangerous" sites. If you manage to access them via VPN, then chances are Tencent is getting your VPN's IP, and it's not your problem.
If it's a hot new URL that hasn't been firewalled yet, then it's also not going to be in the safe browsing database, which means that your browser won't make that second request (with the full hash) to verify it.
You can think of various other what-if scenarios. But the other thing is that all internet providers within China are in bed with the government and can cough up your network history on demand. Why bother going through Tencent?
People are pillorying Apple over a vague theoretical risk that is only really practical in the free world.
EDIT: I found out that there isn't 2 requests, only one and only with the prefix. The prefix is first lookup in the client database, and if it's there, a request is made to get the URL related to that prefix. My idea still works but then they would need to give theses specific prefix in the database already, which will be pretty obvious.
> If it's a hot new URL that hasn't been firewalled yet, then it's also not going to be in the safe browsing database, which means that your browser won't make that second request (with the full hash) to verify it.
Who say you need the full hash? Keep all prefix with the timestamp and their IP address and believe me, you can get a pretty good picture.
All you need is to keep a bunch of URL you don't block but consider harmful, and a bunch of of others URL from theses same domains.
Here's a partial hash that I'm pretty sure would be quite useful:
From your database, find everyone that called for 18ec. It may matches many other URL, but use your backup for Reddit and the timestamps. You now got the username linked to their IP.
Someone doesn't post? Use 1483, B5E7, 13F9, 5B58, 0859, etc...
The funniest is that you could probably find the URL yourself using theses prefixes... but yeah the goal is to find out who went to theses URL, not really which URL they went to. You don't need that many matches to confirm they truly went there.
I wonder haw many pages on a typical website you'd need to browse to let Tencent de-anonymise that k-anonymous hash-prefix scheme?
If Tencent added ihatetheccp.com and ihatetheccp.com/about and ihatetheccp.com/news and ihatetheccp.com/blog and ihatetheccp.com/donate to their "unsafe page" list - I suspect the pattern of fetching of the full list of urls for some of those specific hash prefixes in a short time would allow Tencent (or Google) to make reasonably accurate assumptions about which site you're actually visiting.
That's true, but if you scroll down further it mentions that they try several permutations of the url, up to 30. So while grabbing 1 of the 2^32 buckets won't say much, grabbing 5 buckets (once for each permutation) in a specific order may very well indicate that you browsed a specific url.
That’s on the client, not on the server. The server would just provide the responses for the specific hash prefix. While I agree that, in time, it would eventually get good enough to categorize people, all it would take is a change in the hash and most of the old model would be trash.
But that required by the system to work. Otherwise you have to choose between having really poor granularity (if you only check the domain), or really large black lists (if you only check the whole url). Point is, there's no way for this system to work effectively and be anonymous.
>all it would take is a change in the hash and most of the old model would be trash
Pretty sure browsing _one_ url won't give too much useful away.
Without digging into details myself, I'm under the impression from recent reporting that Safari is sending 32 bytes of a SHA256 hash of the (canonicalised somehow) url. So there's 2^24 possible hashes that you might be wanting to check in the returned list. Presumably some small percentage of those will actually be on the list, what I'm not sure about is how many of them are plausible and completely innocent urls.
Kinda pointless that they're hashing the urls, as to tell if that url was malicious google/tencent would have had to generate that hash themselves as well. The associations are already there, so the hashing step adds nothing (except to prevent a MITM attack where the bad actor didn't have a lookup table).
If Apple wants to create a privacy-oriented safe browsing experience, they need to ship that dataset with the phones themselves.
This assumes there can be many prefix matches for any given url hash.
How many times (probability) do you think a 32-bit hash prefix matches more than one 256-bit hash? Very very unlikely; it's one-to-one practically speaking.
Nearly 100% probability. Someone else[0] already did the ballpark math and came up with ~10,000 unique URLs per 32-bit hash prefix if you use the number of pages indexed by Google. My math has it coming out to be ~10 billion per 32-bit hash prefix. I simply did: 30,000,000,000,000/(36*8), but I might be doing something wrong, happy to be corrected.
The issue is, as mentioned in that comment, with visiting a handful of pages you can get a clearer picture of similar URLs that are being returned among these hash prefixes, which can be used to build a profile of your browsing history.
You're right, I was using 0-9A-Z, when it should have been 0-9A-F. I still need some coffee :)
So for 6,984 URLs per 32-bit hash, wouldn't that be evenly distributed since it's the result of a hashing function? Therefore we'd expect fairly close to 6,900 URLs per prefix? In what situation would you expect a 1-to-1 of 32-bit prefix to URL? Note: happy to be disproven, this is not my specialty at all.
We are working in bits, use bits instead, that will avoid theses kinds of mistakes.
> In what situation would you expect a 1-to-1 of 32-bit prefix to URL?
Oh yeah sorry I misunderstood it, yeah it's pretty unlikely that you would get 1 url for a prefix(but still possible). I would have to get out my old probability books to find that out but it's not worth it, the probability would be way too tiny.
I thought it was about certainty to be able to match it with the real URL. In theory it would takes only a few page hit to be certain of the domain and thus the URL (if there's no unknown string in the URL).
A prefix of URL hash more or less gives away the site you are visiting, given a large enough database. And in China, an IP address can be tracked down to a person, given that most sim cards can be identified due to regulation.
>A prefix of URL hash more or less gives away the site you are visiting, given a large enough database.
That database would be ridiculously huge. If they truly are using the same method for Tencent as they are for Google, the prefix has would give you several thousand different domains potentially. According to Google's docs, the full URL is hashed, not just the domain. (https://developers.google.com/safe-browsing/v4/urls-hashing)
The top return for a google search "how many pages does google have in it's index" says Google has "30 trillion web pages" (that's a 2013 article, but let's roll with that number for now).
The Google Safe Browsing docs says to send the first 32 bits of a 256 bit SHA256 hash. So there's 2^32 = 4e9 possible prefixes (And 2^224 = 2e67possible hashes for each prefix.)
30 trillion = 3e13 webpages. If you assume they're evenly distributed across all the hash prefixes (a reasonable assumption for a cryptographically strong hash function) there's about 1e4 or 10,000 urls matching each prefix. (And that's a lower bound, using 2013 vintage estimates of the number of known urls...)
I _think_ that means visiting 13-14 unique urls from a site in Tencent's lists would be enough to guarantee they could tell which site and pages you'd just visited? (since 2^14 > 1e4)
1: I'm pretty sure browsers keep a local list of known malicious hashes, and only request a list if the URL matches it. So those 13-14 URLs would all have to happen to have a prefix on the safe browsing list (presumably quite unlikely). I guess tencent could just advertise every hash as being on the list, but I feel like that would've been discovered if that was the case.
2: Even with 13-14 hashes I don't think that _guarantees_ a match, it's just the average you'd expect to need in order to find a match.
3: This becomes significantly harder when a user browses multiple websites at a time (which most people do, even if its just because they click a link from one website to go to another).
2) Right. I was imagining some sort of binary search through the hash space - which might be way off base. It might actually take all 10,000 pages to guarantee it? That seems wronger in my head than binary search?
And those 10000 URLs per hash are not equally likely. Chances are one or two of those URLs get visited and the other 9999 are so obscure they can almost certainly be ruled out. Sending a 32-but hash isn't that far off sending the URL.
When the link to the privacy document worked, it basically said we collect everything, but in a fuzzy jumble of poorly worded english that looked like it took a few laps through google translate.
You'd think that if the CCP wants the browsing history of its citizens, it will make that mandatory (ie. all browsers have to upload their histories to the CCP), rather than through some sort of convoluted scheme like this.
(i) It is not static. It actually changes very frequently.
(ii) You would get a fair amount of cache misses, so a proxy if anything would be a better approach.
(iii) The providers do not allow it (source: have tried to work with at least one of them). I am not sure exactly why they want IP, but the sense I got was it did have something to do with training their models.
> If it is true what the article claims, that they give hashes to Tencent, then this statement is false
Is today really the first day all of y'all are hearing about Safe Browsing and how it works? The _prefix_ of the hash is sent to the blacklist service, not the entire thing.
With all due respect, this makes no sense whatsoever. You can't make any meaningful conclusion (in isolation, or even with small enough/random enough datasets) about the content that was hashed from only a part of the resultant hash, that's the whole point of how hashing works.
It gives a 32-bit hash prefix (derived from Sha256) for a subset of URLs that matched a local database of suspicious prefixes. Those prefixes are used to request the provider send a list of malicious sha256 hashes with that prefix.
> Apple may deny users in China VPN protection, it may deny Hong Kong democracy protesters an app used to avoid police, and it may remove references to Taiwan in localized versions of iOS 13 for the Chinese-controlled Hong Kong and Macau.
With recent HKmap.live deleting this is EPIC FAIL, amigos. Seems like now China just bought most Apple shares and make new Huawei. Just imagine it 3-4 years ago.
It's not a backdoor, it's just safe browsing. And safe browsing is controversial since the beginning because this (but now most people have forgotten it).
But don't forget that cryptographic techniques are already used to protect the privacy of users, Mathew Green just wrote a good analysis.
> To address these concerns, Google quickly came up with a safer approach to, um, “safe browsing”. The new approach was called the “Update API”, and it works like this:
> * Google first computes the SHA256 hash of each unsafe URL in its database, and truncates each hash down to a 32-bit prefix to save space.
> * Google sends the database of truncated hashes down to your browser.
> * Each time you visit a URL, your browser hashes it and checks if its 32-bit prefix is contained in your local database.
> * If the prefix is found in the browser’s local copy, your browser now sends the prefix to Google’s servers, which ship back a list of all full 256-bit hashes of the matching URLs, so your browser can check for an exact match.
> At each of these requests, Google’s servers see your IP address, as well as other identifying information such as database state. It’s also possible that Google may drop a cookie into your browser during some of these requests. The Safe Browsing API doesn’t say much about this today, but Ashkan Soltani noted this was happening back in 2012.
> It goes without saying that Lookup API is a privacy disaster. The “Update API” is much more private: in principle, Google should only learn the 32-bit hashes of some browsing requests. Moreover, those truncated 32-bit hashes won’t precisely reveal the identity of the URL you’re accessing, since there are likely to be many collisions in such a short identifier. This provides a form of k-anonymity.
> The weakness in this approach is that it only provides some privacy. The typical user won’t just visit a single URL, they’ll browse thousands of URLs over time. This means a malicious provider will have many “bites at the apple” (no pun intended) in order to de-anonymize that user. A user who browses many related websites — say, these websites — will gradually leak details about their browsing history to the provider, assuming the provider is malicious and can link the requests. (Updated to add: There has been some academic research on such threats.)
> And this is why it’s so important to know who your provider actually is.
> What does this mean for Apple and Tencent?
> That’s ultimately the question we should all be asking.
> The problem is that Safe Browsing “update API” has never been exactly “safe”. Its purpose was never to provide total privacy to users, but rather to degrade the quality of browsing data that providers collect. Within the threat model of Google, we (as a privacy-focused community) largely concluded that protecting users from malicious sites was worth the risk. That’s because, while Google certainly has the brainpower to extract a signal from the noisy Safe Browsing results, it seemed unlikely that they would bother. (Or at least, we hoped that someone would blow the whistle if they tried.)
> But Tencent isn’t Google. While they may be just as trustworthy, we deserve to be informed about this kind of change and to make choices about it. At very least, users should learn about these changes before Apple pushes the feature into production, and thus asks millions of their customers to trust them.
This is spot on, and I'm going to call it for what it is. Just another way of Google creeping on user data to improve their products. The same way they do with fonts, email, ads, etc etc etc.
I for one I'm appalled that nobody really cares too much about this because of the covenience provided by Google tools, but we clearly have a situation building up where another RMS will come up with the equivalent of the free software foundation but from a privacy perspective.
> I'm going to call it for what it is. Just another way of Google creeping on user data to improve their products.
You called it spot on and then came to a conclusion that fundamentally disagrees with the article:
* This is Apple and Tencent, not Google
* The author does not believe that Safe Search (Update API) is a way for Google to spy; or if it is, its privacy risk is marginal compared to the huge security win
* The author's beef is not at all with this functionality, but in how it was communicated
it appears to me their current leadership also would have said <<they‘re one of our biggest platform drivers. let‘s just put flash on idevices. screw „the right thing“. also, money money money!>> when adobe was bitching and mourning.
sure, there have been issues under jobs. but those were of different quality. cook counts beens and has no balls. schiller goes on stage and says „courage“. ive has nobody measuring his and his teams ridiculous design decissions — flat it must be!
i would bet money steve would have put the iphone 6 down, notice it wobbles, pull up his eye brows and sent the team back down the design bunker telling them not to emerge until that bullcrap is fixed.
Apple should rely on its own server to check for safe browsing lists with Tencent, Google and other providers. An Apple API that encapsulates these external APIs.
China has many other more effective means of blocking websites for users within China mainland.
Since Tencent is used for a malware list instead of Google for all users who have their iOS region set to China (even if they aren’t physically in China) this would technically allow them to falsely flag websites as malware even for some people outside of the country. But, this is a user configurable setting, and you can always bypass the warning.
That would be the most incompetent implementation of censorship ever. All you'd have to do to bypass this is disable the safe browsing feature. At least school firewalls require you to use some sort of proxy service like google translate.
If you want to condition people that a site is bad but more subtlety than a outright block, it is a viable method. If you can visit it, but every time you do you are misleadingly told it is malicious, then that may have a subconscious effect (assuming you don't disable it outright), and you don't have to stop everyone from visiting to influence society as a whole.
In fact, there may be cases where being blocked by the Chinese authorities lends credibility to site, whereas being flagged for malware does not.
These are definitely edge cases, and no doubt the China government would to completely block a site in most circumstances, but it is another tool they could use. It also increases their coverage - the current censorship tools only cover China, but this covers any _device_ where the region is set to China (meaning it can apply to some people outside of China, and continues to apply to people who have temporarily left China).
Unless I'm mistaken, that's not true. The hash is computed from the full path and the prefix is the first 32-bits of the hash (or 48-bits if using Safe Browsing).
Safe Browsing has a canonicalization step. Given a URL the Safe Browsing implementation calculates several hashes, each of which is checked. They're basically cutting the URL up into gradually less specific URLs, so e.g. maybe the site malware.example.com is a friendly site discussing Malware, no harm there, but https://malware.example.com/samples/ is entirely full of sample malware, don't want that, so when you try to go to https://malware.example.com/samples/windowsmacker.exe the browser hashes not only malware.example.com but also malware.example.com/samples and malware.example.com/windowsmacker.exe
This lets Google flag the actual problem, without freaking out for users who aren't in harms way. Millions of users visiting the discussion site see nothing, but everybody directed to an actual malware installer gets a warning.
So a "prefix" operation happens in the code twice, once in processing the SHA-256 hash to get a 32-bit prefix which is all Google / TenCent are ever shown but also before that during canonicalisation to figure out a list of hashes (not a single hash) for each URL visited.
Privacy minded people I know disable safebrowsing even when it's provided by Google, and although Google dropped its "Don't be evil" long time ago, it is still more trustworthy than Tencent backed by Chinese government.
Out of the box Firefox, even as-installed on Debian through apt, has sponsored content on the home page and new tabs. It's basically advertising.
There are other annoyances, I forget, every time I setup a new user or new profile with Firefox I'm reminded how much worse things are and have to go change a bunch of obnoxious stuff.
The problem with a Linux laptop is that you are then bound to shitty UI.
A problem which already begins, if you merely switch browsers. For instance the only browser other than Safari, who has the close buttons on a tab right is Opera. I think you can tweak Firefox to do it, but you are out of luck with Brave or Chrome. That is just one example.
KDE is certainly a step up from Windows, but it is not where the Mac is. This is a fact. It takes more than a HN comment to explain that fact. If you can't discern that yet, lurk more in the Mac-based scene.
Besides, I didn't call KDE shitty but the UI of a Linux laptop in general. Here's the thing, the commenter I responded to did express ethical considerations as the driving motivation for a switch to Linux/GNU from BSD/Mac. Not technical ones.
I can relate to that. I myself would rather switch to an open OS, but the truth is nothing compares to the comfort of the Mac. Currently at least.
> To accomplish this task, Safari receives a list of websites known to be malicious from Google, and for devices with their region code set to mainland China, it receives a list from Tencent.
From Apple's statement, Tencent's API functions exactly the same as Google's k-anonymity model, which you can read more about the api[0] and how it works[1] (also via this JAMIA paper[2]).
0: https://developers.google.com/safe-browsing/v4/update-api#ch...
1: https://blog.cloudflare.com/validating-leaked-passwords-with...
2: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2528029/