
Ask HN: Online web crawling services that are affordable and support UTF-8 - anfractuosity
Hi,<p>I&#x27;m just wondering if anyone can recommend any web crawling services that support UTF-8,<p>I&#x27;ve had issues with 80legs not supporting that when I used it some years ago, which gave me results which I couldn&#x27;t use unfortunately.<p>Ideally I would be looking to pay less than £100 to crawl a site with a large number of pages.<p>Any advice would be most appreciated!<p>Cheers
======
db48x
If your budget is that low, perhaps you just want to use wget. If you use a
fairly recent version of wget (1.14 or newer), you can even produce a WARC
file, which is great for archiving. (It captures your requests as well as the
responses, with full headers).

