Firefox and Google Chrome probably have the equivalent of many small high quality libraries embedded in them, implementing 'business' logic or protocols, that could be reused in more places.
I guess a large scale study on github could be done, with a graph analysis to show potential "cut off" points in codebase.
It’s a bit telling that they linked to the GitHub repositories rather than specific lines of code they were talking about.
To me it looks to be trying to uphold your anonimity until you commit (click) through to the site/link. But certainly other ways they can approach this if it really bothers people.. I'd prefer DDG doing the lookup.. or having no fav icons.. over my computer going and downloading all my bookmark or other source icons
Maybe this should be an opt-in rather than an opt-out feature?
Edit: as pointed out by warpspin in another comment, this is about the DDG Browser, not search results.
This is mostly a UX issue IMO.
Not that I'm against making money. But there's a tipping point associated with some height value in a pile of cash, and once you cross that point then the pile controls you. DDG probably hasn't crossed that point yet, but self-justification is one of the steps on that path.
Do Chrome, Firefox, or Safari do this? I would assume they do it on-device.
CREATE TABLE meta(key LONGVARCHAR NOT NULL UNIQUE PRIMARY KEY, value LONGVARCHAR);
CREATE TABLE icon_mapping(id INTEGER PRIMARY KEY,page_url LONGVARCHAR NOT NULL,icon_id INTEGER);
CREATE TABLE favicons(id INTEGER PRIMARY KEY,url LONGVARCHAR NOT NULL,icon_type INTEGER DEFAULT 1);
CREATE TABLE favicon_bitmaps(id INTEGER PRIMARY KEY,icon_id INTEGER NOT NULL,last_updated INTEGER DEFAULT 0,image_data BLOB,width INTEGER DEFAULT 0,height INTEGER DEFAULT 0,last_requested INTEGER DEFAULT 0);
CREATE INDEX icon_mapping_page_url_idx ON icon_mapping(page_url);
CREATE INDEX icon_mapping_icon_id_idx ON icon_mapping(icon_id);
CREATE INDEX favicons_url ON favicons(url);
CREATE INDEX favicon_bitmaps_icon_id ON favicon_bitmaps(icon_id);
I haven't looked at Firefox and Safari but I assume they do something similar.
Also, in SQLite, note that LONGVARCHAR is the same as TEXT, and that you don't need to specify both UNIQUE and PRIMARY KEY (it is redundant), and that if it is not a INTEGER PRIMARY KEY and not WITHOUT ROWID, then it isn't the real primary key but just an index (same as UNIQUE); add WITHOUT ROWID if you want to make it a real primary key, but note that the way the data is stored differs then, and WITHOUT ROWID is inefficient with tables storing large blobs.
DDG is unneccessaryly producing (aggregating), transmitting (and collecting?) very sensitive user data here, which is just the opposite of data protection. I can't even understand why they try to justify their actions. It's like omitting the seat-belt in a car, then telling customers that this was required to make the in-car entertainment system more usable.
The transmission of ip address alone, which is necessary for the TCP request to happen, deanonymizes the request enough to not be considered anonymous within the GDPR framework.
GDPR Article 5 (1) c:
"Personal data shall be ...
adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed (‘data minimisation’);" - this is the "Datensparsamkeit" you mentioned.
GDPR Recital 30
"Natural persons may be associated with online identifiers provided by their devices, applications, tools and protocols, such as internet protocol addresses, cookie identifiers or other identifiers such as radio frequency identification tags."
Oh, and the fact I'm downvoted for a purely informational comment additionally does not shine a good light on DDG.
"When assessing whether consent is freely given, utmost account shall be taken of whether, [..] the performance of a contract[..] is conditional on consent to the processing of personal data that is not necessary for the performance of that contract."
As DDG's favicon-hack is not strictly neccessary for operating the DDG-browser, DDG would need to give users the option to opt-out of the favicon-retrieval, otherwise they may have "forced" the users to consent to the data processing, thereby voiding that consent as far as the GDPR is concerned.
Wouldn't they need to give users the option to opt-in, under GDPR?
So whoever thinks my interpretation is overly broad should first have the decency to step forward and actually explain why, instead of hammering a button and second, talk to me again after he had a meeting with the responsible authority and listen to THEIR interpretation of the GDPR ;-)
People should not mistake my interpretation with endorsement of the overly broad text of the GDPR itself.
Any human readable ways of dealing with that?
Or put another way, a TCP request sent by your app from my computer can not be considered anonymous.
IP-adddresses are considered personally identifying information. TCP requests transmit IP addresses.
Under the strict interpretation of the GDPR, a lot of things which are common outside the EU might be illegal, like e.g. embedding Google Fonts. To be on the safe side, people usually at least list these external dependencies in their privacy policies to construct some kind of "consent", but till we have more actual court rulings, this is a huge problem area.
For the problem at hand, it is pretty clearly illegal, as it's not only an ip address transmitted, it is a combination of ip address plus visited unrelated domain. This allows the creation of profiles. It does not matter for the GDPR, if the profile is ACTUALLY created, the pure possibility of creating it any time is enough to be a problem.
Art. 4 GDPR (1) clearly makes the (ip-address, visited domain) tuple personal data
Art. 4 GDPR (2) defines "processing" data, and the pure "collecting" of data, even if immediately thrown away, is usually already considered "processing", therefore the GDPR applies.
Do you really think then there is "nothing of interest for the GDPR" just because they do not actually permanently record that information? It would clearly be a violation. But to the GDPR, the importance of that data is equal. In fact, the domainnames might actually be more important to the law, as article 9 establishes event stricter rules for "sensitive" data about e.g. health or sex life of a person, and the domainnames might just leak that information.
If the TCP request carries personal data like the name of a visited website plus the user's IP address, then it "breaks the GDPR rules" in so far as you now have to fullfil your GDPR transparency/consent etc. duties /before/ sending that request.
Maybe not all website names look like sensitive data to you, but some website visits you surely want to be treated as sensitive, personal data (like names of hospitals, doctors, political parties, religion etc.).
Sure they can. Doesn’t mean you have to believe them.
Seems like time to get SearX a try now: https://searx.me
The service is private as we do not collect any personal information (e.g. IP addresses) on any requests for this or any service and the requests are all end-to-end encrypted.
Potentially saving a few requests here and there is certainly not worth phoning home with that kind of data regardless of what records you keep how much you do to anonymize it. This is especially true for a company that has built its brand on promises of privacy!
Besides, favicon requests are small potatoes compared to the kind of tracking, ads, metrics, and other often-unnecessary page resources that bog down most of the modern web. And a well-designed website can mitigate the issue pretty easily.
This is troubling.
If you say the service is anonymous and does not leak data, prove it.
If it's not present, then you have options, and yes, using your weird API is an option (which I still don't like, but ok).
But sending private information to your servers even when sites follow the standard show either that you're probably not trustworthy, or that your product team is so painfully incompetent that I'd be afraid to use their browser at all.
This sentence is 100% meaningless. I understand you have good intentions, but these things must rely on proof, never on trust. Either you get this information or you don't; whether you say you "collect" it is inconsequential.
It's a really bad look and you should ditch it.
Which presumably means you've already created the logic for determining favicons. I'm not sure why this couldn't be implemented in the browser.
I don't think here's a need for adjectives here. Why stress that it's anonymous (when that's hard to verify) or that the search engine is private, when that too is starting to come into question? Repeating these things won't will them into the reader's perception.
> In addition, doing it this way avoids another request (and potentially multiple) to the end site.
This isn't true, unless I'm missing something here? When I access a website, the HTML response I get from that website includes all the information my browser needs to, on its own, get and display the favicon. Can you clarify why you think/say this avoids one or more requests? What mechanism is this service a substitute for?
Not as worse as publicly denouncing an honest engineer while referencing his paygrade. I hope there is no affiliation you have with DDG to be honest, because this is much, much worse.
I was once an honest engineer too, publicly. Being honest in private is enough for me now. It’s a lesson worth learning.
That said, it’s not like a single HN comment will make or break a company, so if they’re really just a rank-and-file engineer, I hope the company won’t come down on them too hard. A simple “don’t do that” would suffice.
I use DDG and the possibility of getting a statement directly from an engineer conveys much more trust than a carefully crafted PR statement ever could. I would think again about using it if the company does indeed come down on employees that live the values the company writes on its flags to have honest and transparent business practices.
That said, I am careful too when I state things about my company, even if I believe there is nothing to hide. Still, people that think it isn't the place for others with knowledge to comment are often not too impressive and would have difficulties in convincing me that privacy and transparency are real goals instead of just looking decent enough.
Furthermore the naming of management of DDG creates a stark contrast to the suggestion for more professional distance. I don't like PR very much as you might have guessed, but like a good design it needs some congruence.
If people find out that you just shut up for your company, it might give people the wrong impression about their business.
By commenting on an ongoing PR crisis without consulting management, you are both undermining their ability to respond in an effective way — imagine how strange it would look to see a “Hey, X from <company> here” after an existing one was already posted — and you’re acting on your own rather than in a team. You’re a part of a team; how could you think it’s a good idea to act alone?
Of course, I am talking to my former self with this comment, since that’s exactly what I did at S2 when working on HoN. It was a mistake, and I gave the community the wrong impression about the company’s priorities.
You have to understand, when you’re given money to do a job, you’re not given authority to become that job. Just because your job is getting beat up on social media doesn’t mean you should just jump in and go “Hey, that’s not true!” It doesn’t matter whether it’s true. Here, let me pretend to be DDG:
“Hi, Shawn from DDG here. You’re right; this was an oversight on our part. Obviously we dropped the ball on this. To clarify, we were unintentionally gathering the data as a side effect of our favicon service. <some technical details here>. We’ll be acting immediately to reverse this, and we’ll be enacting policy changes to ensure that user privacy — our core mission — is maintained going forward.”
But that’s not what they said. And if you’re gonna tell the community the opposite of what they want to hear, you’d better be in charge of the company’s Telling The Community Things division.
Sounds like you'd prefer him to have run a message past management/public relations first?
All you have to do is to read this comment thread to see the kind of damage that a single statement by someone affiliated with the company can do.
Agree, didn't mean to imply that.
If that's true, then I am so glad they ghosted me when I applied there.
I'ts not like we couldn't have predicted this disaster.
One more reason to never trust a companies "word". Show me the code.
I have read your explanations in good faith and they don't cut it. This behavior cannot continue. Good privacy promises are not based on trust - they're based on not ever handling private data in the first place. If you don't quickly admit your mistake and roll this back, it will jepoardize your entire brand - and rightfully so. If you believe this behavior is okay, then it demonstrates incompetence; if you don't believe this behavior is okay but do it anyway, it demonstrates malice.
This is the one thing you Should Not Have Done.
Edit: I'm speculating here. But specifically because of the way you've replied here and on Github, my actual level of trust in DDG team went down.
Generally speaking. Mine is shielded with lead.
The repeated handwaving that no one in your company is ever going to do something bad or stupid when the browser phones home for what amounts to a cute sticker is extremely suspicious.
Curious to know why this is an issue.
An online favicon generator will create these variations
<link rel="apple-touch-icon" sizes="57x57" href="/ico/apple-icon-57x57.png">
<link rel="apple-touch-icon" sizes="60x60" href="/ico/apple-icon-60x60.png">
<link rel="apple-touch-icon" sizes="72x72" href="/ico/apple-icon-72x72.png">
<link rel="apple-touch-icon" sizes="76x76" href="/ico/apple-icon-76x76.png">
<link rel="apple-touch-icon" sizes="114x114" href="/ico/apple-icon-114x114.png">
<link rel="apple-touch-icon" sizes="120x120" href="/ico/apple-icon-120x120.png">
<link rel="apple-touch-icon" sizes="144x144" href="/ico/apple-icon-144x144.png">
<link rel="apple-touch-icon" sizes="152x152" href="/ico/apple-icon-152x152.png">
<link rel="apple-touch-icon" sizes="180x180" href="/ico/apple-icon-180x180.png">
<link rel="icon" type="image/png" sizes="192x192" href="/ico/android-icon-192x192.png">
<link rel="icon" type="image/png" sizes="32x32" href="/ico/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="96x96" href="/ico/favicon-96x96.png">
<link rel="icon" type="image/png" sizes="16x16" href="/ico/favicon-16x16.png">
<link rel="manifest" href="/ico/manifest.json">
<meta name="msapplication-TileColor" content="#ffffff">
<meta name="msapplication-TileImage" content="/ico/ms-icon-144x144.png">
Nonetheless, the browser can see this when parsing the page and choose the appropriate path.
Besides, how do you handle Intranet, VPN sites, and auth-only sites where DDG's god-tier favicon parser in the cloud couldn't fetch the URL anyway?
Uncheck the very last option "Site Icons"
The issue, as I understand it, is that the Android app loads the favicon service for search results you actually open in the app.
Sidenote: The more I use pi-hole the more I realise how essential it is!
Those simpler popover-ads which can be closed clicking an X in the upper right corner still are blocked tho...
Spilling my secret tho (and YouTube execs hate me for it!): i block YouTube in my mind and only rarely go to it if i really need to watch a video (which, for me, is rarer than i ever thought it'd be).
Or mpv (for single videos, or local playlists), or mps-youtube, or youtube-dl.
Dear DDG, you are getting complaints on GitHub and Hacker News. This is not the general public, it’s people who understand the issue. You should definitely reconsider whether you’re doing something wrong.
That must be the worst justification for this possible. Favicons. Complicated to locate? Who are you trying to fool, 5 year olds?
Can you tell how many visited site A and also site B?
If you think the next time I hit the shitter I'm not going to be looking for a new browser, you're dead wrong.
Just do the basic checks and then fall back to a DDG logo, no one cares that much about the favicon.
... is completely irrelevant. Even if they were trying to save babies from a fire (which they really aren't) it wouldn't excuse the fact that they're doing something orthogonal to their stated policy and sole reason for existing.
Everyone makes mistakes, that's not the point. The point is to correct them when they're found, instead of digging one's heels in the ground and pretending it's nothing.
Nobody in the company at any point thought that it could be a problem?
How if you type a url into the browser how do you stop the browser from sending that url to ddg to get the favicon?