
DuckDuckGo browser seemingly sends domains a user visits to DDG servers - commotionfever
https://github.com/duckduckgo/Android/issues/527
======
yegg
Hi all, Founder and CEO of DuckDuckGo here. I’m literally just waking up and
reading the comments here.

I’m new to this issue and happy to commit us to move to doing this locally in
the browser and will have us move on that ASAP.

That said, I want to be clear that we did not and have not collected any
personal information here. As other staff have referenced, our services are
encrypted and throw away PII like IP addresses by design. However, I take the
point that it is nevertheless safer to do it locally and so we will do that.

~~~
ddevault
Thank you for re-opening and prioritizing this.

However, this problem demonstrates gross incompotence for a browser team
supposedly concerned with privacy. Will you please do a post-mortem on how
this code made it through your code review process in the first place, as well
as how it managed to stay in place for a full year after it was pointed out
that it represented a privacy problem?

"Sends every URL you visit to the vendor's servers" is the _single worst
thing_ DuckDuckGo could have done for privacy in this web browser, and that
needs to be accounted for. There was a major failure in the code review
process, ticket review process, and in how you treat your community. A
standard marketroid "by design" response with washy promises that "we'll take
very good care of this highly sensitive personal data, just trust us" is not
something I want to see in the future from this team.

[reposted from GitHub]

~~~
elliekelly
I’ve worked with many companies who have demonstrated “gross incompetence”
when it comes to privacy and information security. This is absolutely not an
example of gross incompetence.

I agree that for a company built around privacy even the appearance of
impropriety needs to be avoided. DDG holds themselves to a higher standard and
their users hold them to a higher standard.

This was a design flaw and a process flaw. DDG prioritized speed and
efficiency over privacy (or in this case, perceived privacy) and I suspect
there isn’t a soul on HN who hasn’t made that trade off at some point. They
assessed the cost/benefit and risk/reward and it turned out their assessment
was wrong. Now they’re fixing it. It happens. But to call this gross
incompetence is really blowing it completely out of proportion.

~~~
ddevault
I'm not blowing it out of proportion. This one specific "design flaw", if
we're being generous, has been raised many times with many different browser
vendors and add-on vendors as a very bad thing that you cannot do. There is
plentiful wisdom on this issue.

The first rule of privacy is never handle the private data in the first place.
An accidental leak is one thing, but deliberately designing a feature whose
side effect is exfiltrating heaps of private data, then doubling down on it
_for a year_ after it's pointed out to you, then doubling down _again_ when
it's raised on HN - this _is_ gross incompetence.

------
tagawa
DuckDuckGo staff here. As mentioned in the linked page, the purpose of the
request is to retrieve a website's favicon so that it can be displayed in
certain places within the app or on the results page. We use an internal
favicon service because it can be complicated to locate a favicon for a
website. They can be stored in a variety of locations and in a variety of
formats. The service understands these edge cases and simplifies retrieval
within our apps and our search engine.

Like our search results, the favicon service adheres to our strict privacy
policy[1] in that the requests are anonymous and we do not collect or share
any personal information.

[1] [https://duckduckgo.com/privacy](https://duckduckgo.com/privacy)

~~~
st3fan
Take the code from Firefox iOS or Android-components. We spent a lot of time
on these and it is all on device.

[https://github.com/mozilla-mobile/android-
components](https://github.com/mozilla-mobile/android-components)

[https://github.com/mozilla-mobile/Firefox-iOS](https://github.com/mozilla-
mobile/Firefox-iOS)

~~~
antpls
I wonder how many lines of code from big open source applications are generic
enough to be reused in other projects.

Firefox and Google Chrome probably have the equivalent of many small high
quality libraries embedded in them, implementing 'business' logic or
protocols, that could be reused in more places.

I guess a large scale study on github could be done, with a graph analysis to
show potential "cut off" points in codebase.

~~~
sillysaurusx
Yeah... the gesture is nice, but good luck extracting any code from a massive
project. Might as well say “Here’s some free oil; all you have to do is dig
for it.” Unlike oil, this might not be worth the excavation.

It’s a bit telling that they linked to the GitHub repositories rather than
specific lines of code they were talking about.

~~~
_hl_
Here's their kotlin implemenation, looks fairly straightforward/self-
contained: [https://github.com/mozilla-mobile/android-
components/tree/ma...](https://github.com/mozilla-mobile/android-
components/tree/master/components/browser/icons)

~~~
st3fan
Android components is a foundational technology for our browser products on
Android but also for many other applications. We’ve designed things in such a
way that you can pull in just those things that you need. If that is not the
case, file an issue and we will take a look at how to improve things.

------
throwaway_pdp09
There's an interesting disease showing up here in the responses.

I accept DDG's statement that this is about a favicon and that they "do not
collect or share any personal information", and despite that, I also agree
with others that DDG should be on the safe side and just stop doing this small
thing. It's just the safer and more moral thing to do (So DDG, as many are
suggesting, plz stop doing it. Today is good).

But... the reaction here is "they made a mistake, let's pile on like kids in a
playground" ignoring the genuinely huger issue of the amount of info and
mining that google et al. do. There's no measure of proportion in the
responses, someone is making a mistake then there's a wolfish, pack-like
desire to get stuck in and hurt someone.

Which is why politicians rarely admit mistakes, because it's taken as a sign
of weakness, not strength, to admit you were wrong. DDG isn't the big evil on
the web but from reading some of these you'd think it was the 2nd google.

This isn't about DDG, just the proportionality of responses in public errors
and what society you'd like to have.

(no affiliation to DDG)

~~~
warpspin
I think what angered people was actually that a company saying to hold privacy
high was simply refusing to change something after a mistake was pointed out
and instead kept on defending it with a technical argument, which makes no
sense at all.

The reaction would have been actually a lot different if someone from the
company admitted the mistake and promised it will be changed.

Update: Gabriel Weinberg has promised to change it, linking it here so it does
not get buried in the pile of comments:
[https://news.ycombinator.com/item?id=23711597](https://news.ycombinator.com/item?id=23711597)

~~~
koheripbal
Read your comment again.

You are faulting someone for defending thier own argument. You suggest that
people who do not cow and apologize to the mob deserve the anger and
retribution the mob has to offer.

People have a right to think differently and express themselves without
threats, bullying, or shaming.

The mob does not deserve apologies. The comment above is spot on - we've lost
all sense of proportionality.

It is an indication of the modern online mob sickness that they always demand
others beg for forgiveness.

What emotional void are mob participants trying to fill with the apologies of
others?

~~~
411111111111111
i cant fully agree.

you obviously should be allowed to make a mistake and be forgiven for it. that
does not mean that i personally would ever forgive any `company` that markets
itself as pro-privacy after its been caught gathering data on its users.

i could forgive the people working at the company and would definitely expect
future employers not to hold that against them, however.

but if a `company` does something while claiming to stand morally opposed to
exactly that.... proves that it doesn't actually care about the topic. it just
wants the publicity for marketability, discrediting them entirely for all
future communication.

in this particular case, i wouldn't go that far however. they weren't
gathering any data on their users if i understood it correctly. it was just a
badly implemented feature, which will get changed

------
davidhyde
Ubiquity did the same thing with their routers. They couldn’t understand why
users had such a problem with their phone home feature that was on by default
when the purpose of it was to ultimately “improve” the user experience. I
didn’t buy their router as a result. I also removed kaspersky from my computer
because I didn’t like their phone home feature. Turns out they were selling my
data despite holding my trust as a security company. DDG, don’t turn this into
a PR nightmare. We don’t trust anyone anymore. Privacy policies are worthless.
Nobody cares about favicons anyway.

Source:
[https://www.theregister.com/2019/11/07/ubiquiti_networks_pho...](https://www.theregister.com/2019/11/07/ubiquiti_networks_phone_home/)
[https://palant.info/2019/08/19/kaspersky-in-the-middle-
what-...](https://palant.info/2019/08/19/kaspersky-in-the-middle-what-could-
possibly-go-wrong/)

~~~
pelliphant
wow, didn't know that, thnx for the heads up.

Time to check what my ubiquity router is up to once I get home from work.

------
CivBase
This is a bad look for a company that is trying to build its brand on privacy
and trust. Even though I don't use the DDG browser I hope they own up to this,
rectify it quickly, and learn from it.

~~~
dabbernaught420
In my view, anyone who trusts ddg is a bit silly - founder has a bad track
record on user privacy. Founded Names Database[1], a social media website
designed to collect user information as aggressively as possible, before
selling all the information to classmates.com.

[1][https://en.wikipedia.org/wiki/Names_Database](https://en.wikipedia.org/wiki/Names_Database)

~~~
abc-xyz
Also worth mentioning they're closed-source, US-based and for-profit. Why
exactly do people trust them? Simply because they write a few articles/ads
saying "privacy is important"?

If you're willing to sacrifice search quality for privacy, as in switching
from Google to DuckDuckGo, then you might as well take a step further and
switch from Google to Searx/Ask.Moe.

~~~
leereeves
As a DDG user I don't feel like I'm sacrificing anything. Two sets of results
are better than one (I can see Google results by adding !g, which I do less
than once per day on average) and, ironically, DDG bangs are the easiest way
to use even Google services like Translate and Scholar.

~~~
abc-xyz
Ask.Moe also support bangs (I know !g and !gt works, not sure about scholar).
Searx doesn’t but I imagine they’d gladly accept a pull request for it.

~~~
leereeves
I don't know much about those search engines. What's the advantage of Ask.Moe
or Searx?

~~~
abc-xyz
For me the fact that they’re open source. This thread is a prime example of
why it’s so important. We really have no idea what DuckDuckGo is doing because
they’re closed source. For all we know they could be forwarding users’ IP to
Microsoft/Yandex/etc.

If you want to market yourself as a champion of privacy, then the absolute
minimum criteria should in my opinion be that your codebase is open source.

------
jpangs88
The favicons on the duckduckgo browser are often worse than other browsers in
my opinion. For example the BBC website where DDG interestingly enough just
uses /favicon.ico and the other browsers use the apple touch icon.
(Information I found from just looking at the pages headers)

Don't really understand why they do extra work to get worse results... This
feels to me slightly worse than just a privacy concern, it's a
misunderstanding of their domain which leads me to the question of what else
do they not fully understand.

The good news is that you can have the DDG search engine as a default in other
browsers.

(I understand that the DDG browser is probably not their main focus and any
lack of knowledge can potentially be just on their mobile browser.)

------
tananaev
Very weak argument for why they do it. Using a service to retrieve a favicon?
Surely there's a way to implement the same logic locally.

~~~
tagawa
We had already had created this anonymous favicon service for our private
search engine. In addition, doing it this way avoids another request (and
potentially multiple) to the end site.

The service is private as we do not collect any personal information (e.g. IP
addresses) on any requests for this or any service and the requests are all
end-to-end encrypted.

~~~
oska
Please don't downvote this reply from a DDG staff member (as I'm currently
seeing).

Even if you don't like the reply it's good that we're _getting_ replies.

~~~
oefrha
You can read essentially the same reply as the first comment on the linked
issue, so not much value is lost even if gp is downvoted into oblivion.

------
rickyc091
Looks like this was an issue posted in 2019. From the looks of it, the code
remains unchanged.

[https://github.com/duckduckgo/Android/blob/b2131d7d2f47fb09d...](https://github.com/duckduckgo/Android/blob/b2131d7d2f47fb09d88e1a7768c67454a639518b/app/src/main/java/com/duckduckgo/app/global/UriExtension.kt#L83)

------
mhaberl
Product description (play store):

"Tired of being tracked online? We can help."

And then they track you.

Yes, that might not be intentional and is used "just" for the favicon, yes
they might not use the info on the domains you visit for tracking you today,
but the data is there.

Why not use that data tomorow "just" to see what kinds of pages their
customers (browser users) are visiting so they can better place their ads..
and then maybe some other idea.. this is a path that many such companies went
("don't be evil").

You either respect the user privacy or you don't - there is no middle "just
for this little feature" ground

------
zeckalpha
Seems a bit much, but k-anonymity could work here. Hash the domain, take the
prefix, get a batch of favicons back. They won’t know which you visited, but
still get the benefits of consistent favicon support.

~~~
bad_user
And how would they get the "batch of favicons" in the first place?

~~~
jhasse
When there's a miss, send the domain to the server so that it can fill the
cache. You probably won't even have to do that as DDG could have already
filled the cache with their search results.

~~~
bad_user
OK, at what point does it become complicated enough to just implement that
shit client side? :-)

------
niftylettuce
Formerly worked with DuckDuckGo

My advice:

Install ungoogled-chromium: [https://github.com/Eloston/ungoogled-
chromium](https://github.com/Eloston/ungoogled-chromium)

Install these extensions:
[https://github.com/gorhill/uBlock](https://github.com/gorhill/uBlock)
[https://github.com/ilGur1132/Smart-HTTPS](https://github.com/ilGur1132/Smart-
HTTPS)

There is also a Chromium extension that lets you install from Chrome Web
Store: [https://github.com/NeverDecaf/chromium-web-
store](https://github.com/NeverDecaf/chromium-web-store)

Set duckduckgo.com as your default search engine with a blank home page. But
you could also use @pkrumins home pages of
[https://techurls.com](https://techurls.com) or
[https://finurls.com](https://finurls.com) as nice home pages.

Use Mullvad VPN: [https://mullvad.net/](https://mullvad.net/) (They are EVEN
available on F-Droid now, which is AMAZING)

Security harden your Android device: [https://niftylettuce.com/posts/google-
free-android-setup/](https://niftylettuce.com/posts/google-free-android-
setup/)

Security harden your Mac:
[https://gist.github.com/niftylettuce/39597a7b3bc0660ffe1e09d...](https://gist.github.com/niftylettuce/39597a7b3bc0660ffe1e09d77588bcf6)

 _P.S._ If you need email forwarding for your domain name, you can use
something I made. [https://forwardemail.net](https://forwardemail.net) \- it
is 100% open source.

Follow me @niftylettuce on GitHub and Twitter for more

~~~
the_duke
Seems a bit off-topic for the concrete issue.

Advertising your Twitter for the advice of "switch to somewhat well-known
browser X, install these very common extensions and use a VPN" is also a bit
... odd.

~~~
austhrow743
An ex DuckDuckGo employee recommending people use an alternative browser over
the DuckDuckGo browser, in a post about the DuckDuckGo browser spying on its
users, is about as on topic as you can get; after the current employee giving
an explanation.

~~~
smolder
They said "worked with" duckduckgo. I'm not clear on what that means. I think
they'd have said "worked for" if it was direct employment.

------
marcinzm
This is concerning because it indicates a lack of care in terms of privacy and
understanding that the best privacy is achieved by knowing the least. Does
this approach permeate their backend as well?

~~~
Stevvo
A privacy focused browser shouldn't even have a backend.

~~~
fulafel
I'm not so sure. I think the privacy-functionality trade-off is understood and
expected by the users, it woul be used by very few people if it were extremely
spartan.

(But this instance, the favicon service, is not a good privacy-functionality
trade-off)

~~~
bad-apple
What would a browser even need a backend for? The only valid use that I can
think of is Google's Safe Browsing list, but if ad blocking can be implemented
totally on-device, surely that can, too?

~~~
jeltz
The use case I was going to suggest was a safe browsing list. Possibly also
something like Have I been Pwned. So, yes, there are valid reasons to have a
backend but they are very narrow and privacy is key when building them.

------
bad_user
Speaking of leaks, I never understood why people use DDG's bangs.

By using bangs you're sending your search history to DDG even when using
search engines that aren't DDG.

~~~
pelliphant
but.. that is obvious?!?

If I don't want ddg to know what I google, ofc I won't do it by typing in !g
in ddg...

I actually use bang functions because I want to help ddg out by informing them
when I'm unhappy with the results I got from them.

~~~
bad_user
So ... you're expecting them to log your searches for later analysis, or at
least meta data about your searches.

That's what they promise to not do and the whole point behind many people's
decision to use them.

And it might be obvious for some, but it still makes no sense :-) given the
browser is capable of doing it.

~~~
pelliphant
well, this isn't something that I have given that much thought...

But yes, when I go to ddg and type in "!g <thing>" after having typed in
"<thing>", I'm on some level hoping that this is somehow informing ddg that
their search results for <thing> could be better.

Now, if they have somewhere stated that they are not going to do this, then
yes, it would be bad.

------
hota_mazi
> At DuckDuckGo, we do not collect or share personal information. That's our
> privacy policy in a nutshell

Except that you do, exactly in the way that the reporter of the issue
explained to you.

But you choose to patronize them and ignore the issue.

------
renewiltord
Haha, amazing to witness. This is the problem with catering to this crowd:
your audience is suddenly full of people who just want to see you fail. Good
luck, DDG.

~~~
mhaberl
I don't agree.

I really like DDG - it works good, it is fast and it does not use my personal
search for giving me "better ads" or "better search results" that put me in a
filter bubble.

But there is a different issue here at play; because of errors like this the
whole DDG brand gets a bad rap - and thats not only bad because of the risk of
people losing a google alternative but because it is real easy to exploit
situations like this for google-like companies to give an impression that "all
this privacy thing is bs, all companies work in a same way". There are a lot
of people that are not really sure is this "privacy thing" is worth the
inconvinience of swiching to some other search engine/browser/app and
situations like this one are not helpful in that regard.

Lot of folks are aware of this and are displeased for risking brand confidence
of such a visible privacy-concerned company for miniscule gains like
performance gains for fetching a favicon for the first time - just fetch the
favicon after you display the rest of the page and cache it, maybe dont even
try to fech it if the connection is poor - who cares really

------
coronadisaster
I don't really trust DuckDuckGo, but I use their search service because I
trust Google less... I still trust Firefox more for a browser although it
won't take much at this point to make me switch.

~~~
stan_rogers
There's always Startpage. You use Startpage; Startpage uses Google _for_ you,
with their own user agent - no history, no tracking.

~~~
Nextgrid
How does that work?

What does Google have to gain from it? Google has pretty aggressive anti-
scraping protections to protect against this exact behavior, so why would they
allow Startpage to get away with it?

What does Startpage have to gain from it? Unlike DDG, they don't seem to have
any core product, so they fully depend on Google's goodwill which is very
shaky grounds when it comes to a long-term business.

~~~
sidibe
> Unlike DDG, they don't seem to have any core product, so they fully depend
> on Google's goodwill which is very shaky grounds when it comes to a long-
> term business.

Isn't that exactly like DDG but Google instead of Bing?

~~~
Nextgrid
My understanding is that DDG is using Bing results as well as their own.
Whether their in-house results can stand on their own is another matter, but
at least they're trying to reduce their dependency on Bing, where as Startpage
is not.

~~~
Kiro
No, their own crawler is just used for Instant Answers and other widgets.

[https://help.duckduckgo.com/duckduckgo-help-
pages/results/so...](https://help.duckduckgo.com/duckduckgo-help-
pages/results/sources/)

> We also of course have more traditional links in the search results, which
> we also source from multiple partners, though most commonly from Bing (and
> none from Google).

So basically it's all Bing and I see no effort to reduce that dependency as
you claim.

------
olafure
I think we're due a full disclosure on this favicon service, what information
is collected and what is stored.

DDG has repeatedly said that they have "not collected any personal
information".

For example,

1\. Does the service store the fact that it got a request for a domain?

2\. Does it store any ID along with that information and if so, how unique is
that ID? How is it generated and what is it linked to?

3\. What other information is stored along with the request?

4\. How does DDG process this information?

5\. Who has or can get access to this information?

------
sonicggg
Something is not adding up. Why would you go through so much trouble and over-
engineer a favicon retrieval service? Really, favicon? Since when did they
become so essential?

I'm pretty sure 90% of websites provide one in a standard way. If not, just
draw a letter there, or anything.

But I don't know. I think that either there is more to this story, or DDG team
completely lost common sense.

------
mikaeluman
I don't want to have to trust everything follows a policy.

It's much easier if I don't even have to trust you. Please change this.

------
lopmotr
Nevermind privacy. How are favicons so complicated that they need a special
service that understands edge cases. Just do it one standard way and if a
minority of websites don't work, then exclude them. We've been through this
mess before with all kinds of web standards devolving into mess.

~~~
fiddlerwoaroof
The standard way involves a meta tag, right?

~~~
jakejarvis
That's my understanding of how it's worked for decades...

1\. Check for <link rel="icon" ...> tag(s)

2\. Check for /favicon.ico

3\. ...give up?

Someone correct me if I'm wrong!

~~~
FalcorTheDog
One major issue is that the "standard" favicon size has historically been
16x16 pixels... which, in the age of high density displays will render either
comically small or comically blurry. There are other meta tags like Apple's
"apple-touch-icon" which has some higher resolution options. But already you
can see the logic here isn't trivial.

~~~
akersten
It still seems trivial to me. Parse all meta icon tags, prefer one that
matches exact client resolution of current display (don't even have to
download the icons - the sizes are defined as part of the meta tag), else use
the largest-resolution one, else in desperate bid try site/favicon.ico,
otherwise give up.

Really shouldn't be complicated enough to need a special service to handle
"edge cases".

Another comment mentions web manifest - I guess try those first before meta
tags, or whatever order the standard says to use. I mean, we're talking a _web
browser_ here, it's designed to do these kind of tasks.

~~~
plorntus
I guess the problem is when you want to quickly provide a favicon with the
search result (beit in the omnibar or actual search page), as you're typing
and results are being displayed you cannot send off a request to the site,
wait for HTML html to finish up downloading, send off a request to the
manifest if its defined for multiple sites at once.

On a technical level of course its doable but in reality it's a complete waste
of data and processing, not to mention it could take a long time to show up. I
imagine they have these favicons all cached on their side so they can quickly
send the right file down and/or do this processing if needed.

That being said maybe they should just not use a favicon if it's that big of a
deal.

~~~
akersten
Oh, this is for putting an icon next to search results? Yeah, that changes the
calculus considerably. I thought this was about showing the favicon in the
browser for a site the user visits (per the issue title).

In that case, yeah, I don't think the icons are necessary to show at all...

~~~
fiddlerwoaroof
Even if you wanted to implement this, the logic of the service could be
directly embedded in the browser as an extension or similar. There’s no reason
to depend on a network service for this functionality.

------
foxhop
DDG mobile apps ~= Web Browser or == Web Browser

I think that distinction needs to be made. I think DDG should treat this app
as a web browser which means phoning home to this endpoint is unacceptable.

------
Angeo34
Whoever ever put their trust into American for profit companies which use
slogans like "private secure and fast" should not be surprised at leaking all
their data.

I never got how people trust companies like ddg or Brave. If you don't trust
Google and Apple why would you trust a smaller company in the same
jurisdiction. They will be forced to hand out all data as well regardless what
they say.

------
trashburger
From st3fan's links, this[0] seems to be something that DDG developers can
use. Took me about 30 seconds to dig it up from the repository.

[0]: [https://github.com/mozilla-mobile/android-
components/tree/ma...](https://github.com/mozilla-mobile/android-
components/tree/master/components/browser/icons)

------
pochamago
This doesn't seem like an issue to me.

------
WClayFerguson
It is a hilarious excuse for DDG to claim they are doing this for a favicon.
Even if DDG is legitimately not _using_ the data they are definitely
collecting the data.

The problem with that is that it requires users to "trust" DDG, which is not
how the world works today. If you are a company that collects info, and you
expect users to trust that the info will remain safe, secure, and never get
misuse that is downright foolish for anyone to believe a word of that.

We all know that DDG cannot claim it's impossible for them to get hacked and
have all that data leak out. Hacks happen all the time and so the solution for
DDG is to simply NOT collect the data, rather than collect and claim it's all
secure.

And we all also know DDG has (or will) get a NSL (National Security Letter)
from the NSA to secretly turn over the data anyway, and when that does happen
the DDG employees are not even allowed to admit it ever happened.

------
awinter-py
seems like the ticket author found this by reading code (presumably was
grepping for duckduckgo.com URLs)

this would _never_ happen with a consumer-facing product from apple or google;
someone would have to MITM their whole OS to discover phone-home

~~~
LeoNatan25
It actually happens very often. People monitor their networks often, and pay
close attention to such problems. Just search for “Apple phone home” and
you’ll find many cases where people complain about Apple’s various services
making worrying requests.

~~~
awinter-py
right sorry -- didn't mean to say that they'd never get caught, rather that
you can't read source

------
akent
Do they release the source of the webservice? Seemingly not. This is extremely
shady.

~~~
Carpetsmoker
Would it matter? I can release all the source I want, but what guarantees this
is also what's running on the actual endpoint?

------
classified
By now it has been sufficiently proven that it is physically impossible to
even exist without sending surveillance data to someone on the internet. We
should probably update the laws of thermodynamics to include that.

------
polycaster
If DDG cannot fetch the favicon in different, reasonable way, then the
question is whether or not the ability to display a favicon in search results
is really worth it.

Personally, no.

------
markholmes
Does this only occur in DuckDuckGo’s Android browser?

~~~
jakejarvis
And the iOS browser it seems:
[https://github.com/duckduckgo/iOS/blob/1ae03d7221180bd6791cf...](https://github.com/duckduckgo/iOS/blob/1ae03d7221180bd6791cf6f7f06922a96335cf75/Core/AppUrls.swift#L43)

------
eightlimbed
Can someone please explain like I'm five how this line of code sends the
domain a user visited to DDG's servers?

~~~
oplav
Every time you visit a website on the Android version of the browser, instead
of requesting the favicon from
[https://example.com/favicon.ico](https://example.com/favicon.ico), the app is
calling out to
[https://icons.duckduckgo.com/ip3/example.com/favicon.ico](https://icons.duckduckgo.com/ip3/example.com/favicon.ico).

Since DDG owns the icons.duckduckgo.com service and the domain you were
interested in is in the request to icons.duckduckgo.com, you've sent the
domain to DDG's servers.

------
gowthamgts12
created a new issue to track this again:
[https://github.com/duckduckgo/Android/issues/876](https://github.com/duckduckgo/Android/issues/876)

------
Geee
I guess they collect statistics of the sites that people visit. This is
anonymous but valuable information.

~~~
jacquesm
Your guesses are both wrong (see upthread) and irrelevant.

~~~
Geee
I just reread the thread and don't know what you're referencing.

~~~
ricardo81
IP addresses are considered PII

------
bangonkeyboard
_EDIT: I didn 't notice that this topic was about the DDG browser (which I
didn't know existed) and responded assuming this was about the site/extension.
For a browser, yes, a client-side solution is possible and probably
preferable. Please check and upvote other comment trees._

This makes sense to me and is not alarming. Getting favicons actually is
difficult to do robustly; many applications and websites use Google's service
to do so, which then leaks the request to Google:
[https://www.google.com/s2/favicons?domain=ycombinator.com](https://www.google.com/s2/favicons?domain=ycombinator.com)

Putting this logic in the client is not feasible. You want to send requests
directly to every shady site that shows up in your search results, load their
pages in the background, work through network delays and HTTP errors, and
parse out the location/format of the favicon files?

DuckDuckGo hosting this functionality themselves is also a positive. They have
previously been burned when the Web of Trust service they were originally
using was found to be farming data, and turned it off immediately once
discovered. Processing, hosting, and serving the icon themselves prevents that
from happening again.

This is not to say that DDG is perfect: links you click do seem to be
redirected through a /l/ page on their domain, which can cause problems:
[https://lapcatsoftware.com/articles/duckduckgo.html](https://lapcatsoftware.com/articles/duckduckgo.html)

~~~
pwdisswordfish2
"This is not to say that DDG is perfect: links you click do seem to be
redirected through a /l/ page on their domain ..."

I am surprised the user is not complaining about this instead of the favicons.
Their privacy policy goes on about the privacy implications of Referer headers
and instead of calling out browsers for sending Referer by default, they
instead give themselves power to record all the user's clicked results
themselves. The Referer problem is something that can be solved by the user at
the browser level through, e.g., using a client that does not send Referer,
browser extensions/plug-ins that can control headers sent, or perhaps with a
local proxy to remove the Referer header.

Unless DDG has changed, these prefixed result URLs are the default. It is
possible to get unprefixed result URLs using the "lite" version of DDG however
that is not the default. "Privacy-focused" search engine chooses less private
default. News at 11.

I recently noticed that DDG has started redirecting queries submitted via POST
to /lite/. The redirect is to the same domain. No explanation. I have a custom
client that does not follow redirects and I now have to submit two sets of
HTTP headers instead of one.

These guys are trying to make money from advertising just like everyone else.
They have to be very particular in the methods they use to do it -- check the
exceptions in their privacy policy -- but it is the same game. Ads and
affiliate links. That sort of business and privacy are always going to be at
odds with each other.

~~~
bscphil
This is done only for browsers that don't support setting the referrer policy.
[https://caniuse.com/referrer-policy](https://caniuse.com/referrer-policy). It
should not be happening with any modern browser, and (like the other reply) I
don't see it in mine (Firefox).

See here:
[https://help.duckduckgo.com/results/rduckduckgocom/](https://help.duckduckgo.com/results/rduckduckgocom/)

~~~
pwdisswordfish2
Are you saying DDG is forcing you to send a User-Agent header?

DDG's privacy policy also goes on about the privacy implications of User-Agent
headers combined with IP addresses.

So let's say I take what they have put in their privacy policy to heart and I
stop sending a User-Agent header. In response DDG sends prefixed result URLs?
WTF?

Using haproxy, for example, I can use a "modern browser" and send no User-
Agent header. I still get prefixed result URLs.

~~~
bscphil
You can disable this on the settings page. Frankly I'd infinitely rather send
my request through a redirect served by DDG than send my referer to every site
I visit. I think this is a perfectly sensible default for the average user,
who both doesn't block / obscure their user agent and doesn't have referer
masking software in place.

You're free to disagree, but in that case you can make use of the option to
change the default that they provide for you.

~~~
pwdisswordfish2
The settings page requires that the user enable Javascript.

I would not call haproxy "referer masking software". In any event, a proxy is
not even needed.

Modern browsers are open source, right? Users can edit the source and remove
the code that sends Referer header.

Even easier, I wrote my own http client. I can send any header I want, or none
at all. According to DDG's privacy policy this is a good thing.

------
twirlock
I didn't know they had a browser. I'll have to give it a try. Can't be any
worse than Google's browser. Or their OS. Or their video monopoly. Or their
search monopoly. Or their secret partnerships with governments. Or their ad
monopoly. Or their email monopoly.

------
glouwbug
Looks like its time to ditch duck duck go

------
rygh
Don't think it's air tight. But still better than most browsers.

------
BuckRogers
Essentially a way to collect where people are visiting. I believe them that
it’s anonymous, this valuable info wouldn’t need to be identifiable to be of
value.

They should probably change the behavior to how it’s suggested in the thread,
but I’m still going to use DDG over alternatives for the bang feature.

------
tomtomtom777
I have a hard time understanding the problem.

The favicon is acquired from DDG servers for the result you've just retrieved
from DDG servers.

How is this leaking anything? What additional privacy would you gain from
getting the favicons from the domains directly of search results delivered by
DDG?

~~~
marcinzm
This is the DDG mobile browser app, NOT the DDG search engine.

------
TabbyCatKirk
Everyone is missing the point here. Let me break this down as simple as I can:

1\. End user does a DDG search for "food" 2\. The "food" query returns a list
of search results, these results have each have a link, DDG wants to display
the favicon for each link. 3\. To be clear, DDG does not store or log the IP
address of the user doing the query. They do, however, know what was queried,
so they know "somebody" somewhere searched for "food". They have to know this,
they are a search engine after all. 4\. Since DDG wants to show the favicon
"privately", and they dont want to put that logic/work on the client side
(which could leak your IP), so instead DDG finds the favicon internally. 5\. A
DDG server, completely separate from anything search-related is then tasked
with finding the favicon for your "food" query results, lets say the #1 result
is www.allrecipes.com, so a DDG server goes to www.allrecipes.com and finds
the exact favicon location. 6\. The "found" favicons are then stored in a
cache, and displayed from the cache like this: [https://external-
content.duckduckgo.com/ip3/www.allrecipes.c...](https://external-
content.duckduckgo.com/ip3/www.allrecipes.com.ico) (and if no favicon is found
in the local cache, you get a grey arrow by default) 7\. I'd like to note,
even with all this action, DDG doesn't know if you actually "visited"
www.allrecipes.com, they simply know that some anonymous user did a search for
"food", www.allrecipes.com was a search result, and a favicon was displayed.
They dont know who searched for it because the users IP is not stored
anywhere, they dont know if you visited www.allrecipes.com, they prevented you
from leaking your IP to allrecipes.com since they didn't force the end user to
load the favicon.

So whats the issue? What am I missing here?

PS: You know this works because after doing all these searches for food and
seeing allrecipes.com (and even clicking allrecipes.com result in the DDG
Mobile App or browser extension), guess what? allrecipes.com doesn't follow
you around with re-targeting ads! Why? Because DDG prevented that from
happening!

~~~
ddevault
This is not duckduckgo.com. This is the DuckDuckGo-branded end-user web
browser.

