
Ask HN: Work to search for if I enjoy web scraping and browser automation - latexr
I’ve always worked freelance. Mostly as a graphic and web designer, including coding the front-end. I’m mostly self-taught as a programmer, skilled in Bash, HTML, and CSS, and competent in JS and Ruby.<p>For a while now I’ve started to notice that what I really enjoy is the web scraping and browser automation I do for myself. I tend to use Ruby for the task, relying mostly on Nokogiri and Watir. I do it to automate repetitive tasks on websites or to build personal RSS&#x2F;JSON feeds of websites that do not provide them (or are limited in what they provide). I don’t feel tied to those technologies. I enjoy them but feel confident I can learn another language&#x2F;library enough to get the job done, if need be.<p>1. Having never worked professionally as a programmer, and having no formal training, what kind of job or contract&#x2F;freelance work should I be searching for?<p>2. What’s the job title of someone that does this type of work?<p>3. Are there any specific skills I should be cultivating? For example, if most available work for these skills is in testing (is it?), should I be focusing on learning a testing framework?<p>4. Are there any specific technologies you’d recommend I learn? Why?<p>5. Any other advice?
======
Dangeranger
What you are describing as browser automation is quite similar to automated QA
and end user testing.

Many QA departments need people who can automate their manual interaction step
process to ensure that a service is behaving correctly.

It’s costly to manually check all the user stories that an application
supports, and automating this can be a career path for an employee or a
contractor.

My recommendation would be to get your foot in the door at a company that has
a large manual QA process and is looking for someone to write for them
automated tests, or convert their manual steps into tests.

------
dgarud
There are 2 fields - Web scraping is done to get data from the web, automation
is done to ensure websites keep functioning as planned.

1\. For web scraping, you may get consulting jobs. For automation testing,
search for job title of 'QA Automation Tester'.

2\. Consultant / Tester / Analyst (Depending on seniority till you become
Testing Manager)

3\. I think there is a much larger market for testing. You can start by
learning selenium. There are many testing frameworks, but it comes down to the
company - which programming language and framework they use.

4\. You can think of learning appium for mobile (android and iOS) automation.

5\. Maybe you can try to search for work in Ruby for starts.

------
africajam
Hi there, a sector in which web scraping is quite important is real estate. I
have written an open source scraper for that sector (using Nokogiri and Ruby
on Rails):

[https://github.com/RealEstateWebTools/property_web_scraper](https://github.com/RealEstateWebTools/property_web_scraper)

Take a look and let me know what you think. I have a few ideas about how to
monetize. Get in touch if this is something you would like to work with me on
( ed at propertywebbuilder.com )

------
justaguyhere
I have no clue about the legality of what these guys are doing but they are
interesting - [https://www.aggdata.com](https://www.aggdata.com)

Might give you some ideas about markets, pricing etc

------
pouta
We need someone with those skills right now. Drop me an email. moura at oko
dot ai

------
borplk
4\. Javascript, Puppeteer

------
malux85
Hey - reach out to me, contact in profile

------
sharemywin
create a simple web presence site

then create a portfolio

then start blogging.

~~~
sharemywin
create a free tool people in your target market can use

reach out to other web dev/phone shops and mention what you specialize in and
see if they come across this work and want to refer it to you or resell it,
etc.

create social media profiles linking back to your site.

