

Show HN: A Search Engine for Developers, Marketers, Web Designers, and Hackers - nerdydata

Watup Hackernews,<p>We have created a new type of search engine that searches on the full source code of webpages, not just the on-page text.  It&#x27;s called NerdyData (nerdydata.com) and we&#x27;re launching today!<p>Our search engine is different from search engines you&#x27;ve used before. Traditional search engines are geared towards providing answers, whereas our goal is to give you the best list of results for a query. Our crawler has visited over 140 million homepages and collected terabytes of HTML, Javascript, and CSS code. We&#x27;ve also designed several search interfaces that allow anybody to query against the source code of webpages, or download a list of sites containing a specific term.<p>We even offer a search interface specifically for SEO&#x27;s and marketers that allow you to search within specific HTML tags like meta descriptions and meta keywords.<p>About us: We&#x27;re two 23 year olds who recently graduated from Stony Brook University in New York, and we are ridiculously passionate about making search better for our fellow nerds out there.<p>If you&#x27;d like to know more, you can email me at dave@nerdydata.com<p>Thank you so much for you time!
======
jwarkentin
I thought it would be really cool until I realized the whole credit thing. It
often takes many permutations of a search to find the results you're looking
for. If each attempt costs money and the whole thing might be a waste in the
end then it's hardly worthwhile.

I've always been annoyed that Google won't search for symbols. As a programmer
that makes searching for certain things anywhere from difficult to impossible.
I was hopeful that this would be a search engine I could use to search for
code snippets or queries that involve symbols.

Let's say, for example, that I was browsing through some PHP code and saw a
double dollar sign ($$) and wanted to know what it was used for (yes, I
already know, it's just an example). I might try searching for something like
'PHP $$'. This search engine is useless for that. That's the real problem I
need a search engine to solve.

~~~
brodney
Check out [http://www.symbolhound.com/](http://www.symbolhound.com/) for a
programmer friendly search engine.

~~~
jwarkentin
You just made my day :)

------
dgunn
This could be great. You need to monetize in a better way. Make searching free
for everyone (cause no one is going to pay for it). Then, package up some pre-
built searches (backlink search, competitor stuff, etc), build a special
interface, and sell that to SEO's as a tool. This is just my first thought.
You have data from over 140 MM pages. You can almost certainly use that data
to make a product worth money to someone.

I just can't imagine selling searches will work. The big players in the space
have made search into a human right. I know you have to make money somehow but
charging for searches almost has to be the wrong answer.

------
brbcoding
Whoa... Searches must be based on IP. Shared with a co-worker and they can see
all of my searches, and I can see theirs.

------
aram
Very interesting service. At first it didn't occur to me why would I want to
search the source code, but your examples were good.

What is the logic behind the search criteria? I entered `<div id="main">`[1]
and the first result is LinkedIn with `<div id="main-wrapper">` highlighted,
which shouldn't match my query directly as I entered a closed tag. Some of the
other results also didn't contain exactly what I was searching for - e.g.
`<div id="main" class="main">`, `<div id="main" class="clear">` etc. A
technical explanation on this would be great.

Also, one minor note - David's photo in the front page footer is broken [1]

[1]
[https://search.nerdydata.com/search/#!/searchTerm=<div](https://search.nerdydata.com/search/#!/searchTerm=<div)
id="main">/searchPage=1/sort=pop

[2] [http://www.arc.losrios.edu/Images/Images-
arc/Foreign_Lang_De...](http://www.arc.losrios.edu/Images/Images-
arc/Foreign_Lang_Dept/Faculty%20pictures/no%20image%20avatar%20man-1.jpg)

------
michaelmior
Looks pretty cool. I really dig the design. I was a little annoyed that I seem
to have 10 credits without noticing and used them up just messing around. (I
can't really tell because clicking on the credits doesn't do anything). It
would be nice to make this more obvious up front. Anyway, I can see this being
a pretty useful tool in some scenarios.

Also note that "Locater" is spelled "Locator" (sidebar on the left)

------
nerdydata
Clickable link: [http://nerdydata.com](http://nerdydata.com)

------
jwcrux
Just a small request - it would be _way_ cool to have a "site:" filter like in
Google searches. A lot of the results found are irrelevant to me as it is.

Cool site - great work!

~~~
nerdydata
We have a Refine Tool that allows for AND,OR,NOT filters. It's only available
for paid users, but really allows you to push the limits and get the exact
query you want. Thanks for the feedback!

------
013
Is there any content filtering? When I went on to Image Locator, then "See an
example" it will search for imgur.com. The first page of results shows
'thebiglistofporn.com', 'entensity.com'(Not sure if it's NSFW, but it looks it
from the thumbnail.)

I know it's not too much of a big deal, but porn thumbnails wasn't something I
was expecting to see on a websites example page.

~~~
nerdydata
We do not filter content, but thanks for letting us know about that thumbnail
on the example page.

------
bapi
Search terms like consumer_secret ou api_secret and you'll find lot of
things... :(
[https://search.nerdydata.com/search/#!/searchTerm=consumer_s...](https://search.nerdydata.com/search/#!/searchTerm=consumer_secret/searchPage=1/sort=pop)

With great power, comes great responsibility.

------
ScottWhigham
Love it - great idea and great presentation. I only tried a few cursory
searches but it was definitely bookmark-worthy. Kudos :)

I'd love to hear you guys talk about the risks of searching for "dicey"/risky
characters/terms and how you protect your data/server from malicious searches.

~~~
nerdydata
When the launch craziness winds down, we plan to start a blog and nerd the
fuck out. We had so much fun making this and want to help others get past the
insane obstacles we faced while building a search engine.

------
ollerac
I don't get how to use the seo search. How do I search for all websites that
have a certain name in their title for example?

    
    
        <title>name</title>
    

Just returns the exact match.

And how do I search meta descriptions and meta keywords?

~~~
nerdydata
SEO Search is a premium feature which lets you search inside of pre-parsed
html tags. You can learn more about it here
[https://search.nerdydata.com/documentation/#seoSearch](https://search.nerdydata.com/documentation/#seoSearch)

------
nerdydata
Did we mention our "Basic" account is free and gives you 200 credits to use?
[https://search.nerdydata.com/pricing?hackerNews](https://search.nerdydata.com/pricing?hackerNews)

~~~
bapi
No , because yesterday it was 49$ or 99$ Idon't remember exactly :). Good move
I think to let people like me use it and find value in it as the promise is
great.

------
ScottWhigham
Anyone tried comparing to [http://searchcode.com/](http://searchcode.com/) ? I
haven't - just ran across it right now doing an unrelated search.

------
kyle_martin1
This would be a very good tool to find XSS vulnerable websites.

~~~
cwings
Or find all sites with a vulnerable installation of wordpress.

------
sachin0235
while it is good, i am wondering what it can do which google can not? For
example i searched for android accountmanager and results i got were far less
productive than what google returns me back. If you want to build it, build it
the way it perceives user perspective well

------
kevrone
This is awesome. Especially for finding use cases and implementation examples
of front-end libraries.

~~~
nirvanatikku
That was my first thought, too. I checked out where my various projects are
being used on the web, something I've been wanting to see for some time, and
was pleased to learn about various properties using it.

------
ilikehunny
your site has a really bad user rating?
[http://www.mywot.com/en/scorecard/nerdydata.com?utm_source=a...](http://www.mywot.com/en/scorecard/nerdydata.com?utm_source=addon&utm_content=warn-
viewsc)

~~~
sadkingbilly
I've never trusted wot. They are abused by groups of "wot cops" that organize
in the forums and target web sites to bash (deserved or not).

As far as the site goes, it looks like they used to be a more generic seo
search engine, as recently as a few months ago.

------
snake_plissken
Good god, my attention span just reached terminal velocity. I like. Very cool
and original.

------
ErikAugust
Do results sort based on mozRank, Alexa, PageRank? Or something proprietary?

------
jayzalowitz
First off Stony Brook! Second, open crawl? Third, what's your stack?

~~~
nerdydata
1\. Go Seawolves!

2\. We built our own distributed crawler with a couple of dope servers.

3\. Not ready to share our stack setup yet (work in progress) but we will have
a write up shortly!

------
adamnemecek
Honestly, I'm not a fan of the name.

