Hello HN! We are Andres and Edgar and we are building MovingLake (
https://movinglake.com). We are creating real-time, bidirectional data connectors which aim to solve automated ETL (eg moving data from an API to Snowflake), reverse ETL (we pipe the data straight to your CRM instead of going through the data warehouse), and transactional integrations (e.g. for an ERP integration) in one swoop. Polling, webhooks, websockets, REST, GraphQL, SOAP—we merge it all into one reliable, replayable, real-time data stream which can be sent anywhere.
Here are two demos: Whatsapp to Google Sheets (https://youtu.be/Qy4eInGgIhw) and Postgres to Webhooks (https://youtu.be/cmrkUan8o1w).
Moving data around is still difficult and a pain. Companies spend thousands of hours and lots of money making data pipelines. General solutions are difficult in that there are too many sources and too many destinations. It also entails a ton of schlep that no one really wants to deal with. At the same time, the API / data connector market is vast. Even when there are already multiple billion-dollar companies in the space, we continue to see big verticals unattended.
We previously worked at big tech as well as have been CTOs of Series B startups in Latam. We got into this problem when we had the experience of paying for pre-built data connectors, but at the same time having to do custom integrations to the API, at which point it didn’t make sense to have a paid data connector. After a few paid connectors and manual connectors we thought that there should be an event-driven, realtime data connector company which could solve all of our API integration needs in one go.
We also are not huge believers in operationalizing the data warehouse, i.e. the trend to use the data warehouse as a processing tool out of which data is extracted and pushed to other systems. Since data warehouses are built on an OLAP frame of mind, we think using it as a computation source for automated workflows is not the best idea. At MovingLake we propose using rather realtime connectors with a transformation pipeline which is specifically designed to do these things.
We combine polling with webhooks and websockets to provide a single reliable stream. If there's websockets, we use them. If there's webhooks we use them. If there's only GET endpoints then we poll as fast as the API lets us. Either way we ensure you'll get the data as fast as possible.
We provide destination adapters so that this data can be sent anywhere to as many destinations as you want for the same price. We provide CDC (Change Data Capture) plugins for databases to pull data as it is written and then send it anywhere. We also provide automatic JSON to SQL converters with Schema Evolution.
On our roadmap we still have to add the data transformation layer as well as add support for more bidirectional connectors.
“Get started” on our home page takes you to a “book a demo” thing, but there’s an open beta at https://app.movinglake.com which everyone is welcome to try out. Everything is free for now, though eventually we’ll have to charge. Although we wanted to charge a standard fee per event extracted, this would have tied our hands to deliver tougher-to-build connectors. So most of our connectors are charged at $0.00004 per event, but some which include scraping are charged higher (we want to be very explicit about which connectors are charged differently).
Please let us know what you think! Roasting our product would be super helpful :)
> bidirectional data connectors which aim to solve automated ETL (eg moving data from an API to Snowflake), reverse ETL (we pipe the data straight to your CRM)
I need exactly this at work. However, my requirements are :
- it has to be "self-hosted", usually in my clients' VPC, sending data to a third party is a big no-no.
- most of them use SAP, Salesforces and many other esoteric CRMs & ERPs. It has to work for at least the most important ones.
- the target medium is usually the cloud provider's object storage (S3, ...), snowflake is quite rare. It also have to be flexible enough to support our own versioning strategy.
Would it work with your product ?