
Show HN: CloudQuery – Turn any website to serverless API with SPA support - timqian
https://github.com/cloudfetch/cloudquery
======
Jaruzel
Nice. I've been toying with the same idea, but with optional RSS formatting to
be able to generate an RSS feed from _any_ website.

What's stopping me though is the EUs Article 13, and I can't see this avoiding
that either.

~~~
hiccuphippo
I was able to do this 10 years ago with Yahoo!Pipes :(

~~~
timqian
Yahoo also have a similar tool nowadays called YQL, I tried it but faild to
make a query..

~~~
franze
YQL is dead since 3rd of Jan. 2019
[https://developer.yahoo.com/yql/?guccounter=1](https://developer.yahoo.com/yql/?guccounter=1)

------
nreece
This looks good!

* _Shameless plug_ *: Our little startup, Feedity - [https://feedity.com](https://feedity.com), helps create XML/RSS feeds for any public webpage, even JS/XHR/SPAs and social networks (Facebook, Instagram, Twitter), via a visual feed builder and REST API.

------
dmitriid
Where's Yahoo! Pipes when you need them...

~~~
morenoh149
dead. so sad.

------
mrspeaker
Very neat idea! I like do this locally in Emacs (by scraping `M-x eww`
output!), but having an API is a great idea! How would you go about using it
on sites that require login (for prototyping private apis for example)?

Also, it says it uses serverless-chrome for running chrome on AWS lambda... is
that "expensive"?

~~~
timqian
For pages require login, it is not implemented yet as this is a little
complicated , but it is not impossible, the tool need to record actions user
do and replay it in the remote browser.

About pricing, AWS lambda provide 10 million free invoke and the billed invoke
is cheap too($0.2 for 10 million invoke) you can check the pricing detail here
[https://aws.amazon.com/cn/lambda/pricing/](https://aws.amazon.com/cn/lambda/pricing/)

~~~
fefb
*1 million

------
orliesaurus
Does anyone remember the Kimonify extension? It reminds me of that! Cool!

~~~
LikeAnElephant
LOVED Kimonify, especially how it (somehow!) figured out the pattern of data
after clicking on a few different items. This is very close to that, minus the
pattern recognition.

------
aboutruby
I think the correct keywords for this would be "tool assisted continuous
scrapping"

------
bobbydreamer
So people are starting to miss YQL

------
leowoo91
Looks great, reminds me of Kimono labs.

------
lugrugzo
What's the use case of this tool?

~~~
tobyhinloopen
My guess: Rapid prototyping of tools that use scraping as a source of data.

We actually have multiple tools that do realtime scraping as the primary data
source. Many of these tools act as a simplified interface to features from
another service.

For example, there is some webapp we've been using that we were using for a
single feature, and that app doesnt have an API available. Using the app
required many clicks, and page-loads were slow. By inspecting the HTTP
requests, we figured out the minimum amount of HTTP requests required to
perform our common task.

Using a simple GUI that focusses on that simple task, the user can initiate
the task from a single form, and the server will perform the correct HTTP
requests and notify the user whether the process was successful or not.

We have plenty of these "micro tools" that encapsulate / wrap around web apps
to simplify the usage of that tool. Usually our micro tools are easier to use
(because its focused around our use-case), add integrations with other tools
and commonly are significantly faster as well.

They are easy to build (usually within 40 hours) and are a real time-saver,
because the users don't have to keep track of all the logins, don't have to
load slow web apps, and don't have to have a guide with screenshots where they
need to click.

These web-apps sometimes change their request/response structure but after
building a few of these tools, but it doesn't happen that often and the tools
are updated within hours.

