
Ask HN: Deep web size? - c-47
It is frequently quoted that the deep web [1] is 400 to 500 times larger than the surface web. That number is attributed to a 2001 study [2].  Is that stat even close to relevant today?<p>Is anyone aware of a more recent study or have any potential insight on the current deep web size compared to the surface web?<p>[1] By &quot;deep web&quot;, I am referring to web pages not indexed by commercial search engines.<p>[2] http:&#x2F;&#x2F;quod.lib.umich.edu&#x2F;cgi&#x2F;t&#x2F;text&#x2F;text-idx?c=jep;view=text;rgn=main;idno=3336451.0007.104
======
chatmasta
Think about it. How many sites do you log into each day? As a percentage of
text on your screen, how much is only visible while you are logged into a
given site?

The "deep web" is very interesting, but a more precise definition would
clarify any measurements of it. For example, does content that is visible to
all authenticated users comprise the same "deep web" as content that is
visible to individual authenticated users? If I can see a wiki only when
logged into a service, but so can every other user of the service, is that the
same "deep web" as my personal dashboard that only I can see when
authenticated?

Also consider the many mobile apps that do not have their content indexed by
search engines, like whisper, yikyak, tinder, etc.

Perhaps the next great search engine will be the one that nails the problem of
gaining insights from the "deep web".

