
Web Crawler in Golang. [Rejected] - yashmurty
https://github.com/yashmurty/go-web-crawler
======
yashmurty
My submission was recently rejected.

The problem statement was: Create a simple web crawler which creates sitemaps
in a textfile.

I am wondering where I can improve it, and how not to be rejected the next
time.

~~~
gus_massa
Where did you submit it to? Who rejected it?

~~~
yashmurty
A tech startup, whose name I will not be sharing here. :)

~~~
gus_massa
OK, I think it's fine to avoid naming them. But the context of the original
question was not clear enough.

~~~
yashmurty
Here's the original problem statement:

\-----

We'd like you to write a simple web crawler in a programming language you're
familiar with. Given a URL, it should output a simple textual sitemap, showing
the links between pages. The crawler should be limited to one subdomain - so
when you start with _[https://google.com/*](https://google.com/*), it would
crawl all pages within google.com, but not follow external links, for example
to facebook.com or mail.google.com.

We would like to see your own implementation of a web crawler. Please do not
use frameworks like scrapy or go-colly which handle all the crawling behind
the scenes. You are welcome to use libraries to handle things like HTML
parsing.

\-----

I hope this adds more context to my post. :)

