

Does Mint.com use screen scrapers? - fatalerrorx3

Just curious if anyone knows the answer to this question..I'm pretty sure when it first was started they were basically scraping these financial institutions, but I don't know for sure.  If they were initially screen scraping, do you think by now they have their own special access to the financial institutions that they deal with?<p>The reason why I'm interested is that I'm currently working on a site that's similar in nature but for a totally different industry and I'm worried that the websites I'm scraping from will be updated and will no longer work...I'm not sure how often the websites in question are updated but my guess would be that as the industry evolves over time the websites will be getting updated more frequently and it's currently only a 1-man engineering team (me)
======
mountaineer
No, they used Yodlee initially, and now use the Intuit aggregator. However,
Yodlee does some scraping, but rumor was (this was 2008) scraping was less
than 10% of financial sites and they had more direct access to other 90%.

~~~
fatalerrorx3
Thanks, I remember hearing about Yodlee when reviewing Mint...in the industry
that this new service is targeting there is not very many things that are
standardized, so there isn't currently a platform that I can use to make
integrating with them easier...I wish there was but then again that's half the
fun =D

------
byoung2
_I'm worried that the websites I'm scraping from will be updated and will no
longer work_

When I worked for Internet Brands, they bought a site called Tjoos (a
coupon/deal site) that amassed what was at the time the largest number of
merchant sites using a proprietary screen scraping technology. It is a very
fragile method, and a good portion of the codebase was dedicated to config
classes that needed to be updated with each change to a given site. The
founders had to keep hiring developers to keep up with the demands of
constantly updating scraping definitions, so the site had a lot of revenue,
but little profit when IB bought it. Surely by now Internet Brands has updated
the site to use the same engine that powers UlitmateCoupons, BensBargains,
DealLocker, etc.

~~~
fatalerrorx3
The newer coupon/deal aggregators are most likely using RSS feeds which is a
more standardized format when compared to screen scraping, and thus less
chance of breaking. There are still a surprisingly large amount of websites
that are still stuck in the 90s in terms of data portability though

------
aquark
There was a previous discussion of this last year:
<http://news.ycombinator.net/item?id=1537825>

I also recall (but can't find) another discussion of screen scraping where the
consensus seemed to be 'be very careful of the legalities'. If you are small
you might fly under the radar, but if you are successful you might just be
opening yourself up to a whole can of worms.

Regardless of the outcome of any battle over this, the battle itself would be
a severe distraction.

~~~
fatalerrorx3
Good advice, I read the discussion you posted and there was a lot of
insightful tips as well, which I've kind of worked out along the way. In terms
of legality, this service will be scraping on behalf of the user, meaning they
provide their login credentials and are granting permission to the server to
perform the scrape..We'll eventually need to have a Terms and Privacy section
which covers the possible legal issues, this is still in the prototyping phase
but it's good to know before launch what is allowed and what isn't by each
website

