
Ask HN: What do you use to move (copy) data between systems? - willnewby
Do you centralize your data? Do you run an iPaaS of some kind? Do you like it?<p>Context: I&#x27;m currently in the middle of a project where I&#x27;m hooking up an ERP system (SalesPad) to an eCommerce website (WooCommerce) and I&#x27;ve found I need to move many different kinds of data between them (Inventory&#x2F;Orders&#x2F;CustomPricing&#x2F;etc), and I haven&#x27;t found a good tool so I started building one.<p>Has anyone else run into this? How did you solve it? Did you like your solution?
======
eb0la
Take a look at kafka ;-) Just put the file path as the key and the content as
the message value.

You get TTL/size garbage collection for free: no need to get/develop a
sophisticated client/server.

------
sitkack
rsync, rclone

really depends on what formats the data is in, how it will be used, how long
it needs to last, how long the code that reads and writes the data will last,
how much out of band information vs inband information will be sent along with
the data.

You talked about movement, movement is the last of your worries, formats and
provenance should rank higher on your list of concerns.

My gotos are sqlite, xml and avro. Try to keep things under 2GB so you don't
run into lame data transfer issues.

------
linsomniac
Many of these sorts of things we will push to an S3 bucket, and have other
parts of the system pull from that bucket. We also use "rsync" with SSH keys
to copy some data. Some of our vendors provide data to us via FTP, similarly
it could be provided via HTTP

Maybe Zapier can help?

------
Jugurtha
Can you configure database replication so your database handles the transfer
to the target system, and then make the ecommerce site accessible for orders?
The two databases will be kept in sync.

------
toomuchtodo
Have you checked out [http://n8n.io](http://n8n.io)? Sounds like you’re just
pushing JSON blobs around on a bus.

------
beeve
Lots of data integration software is available. In the open source world,
Apache Nifi and Apache Camel are two examples.

------
alexchantavy
To add to the rsync answers, I think robocopy in a Windows environment is
great.

------
nailer
rsync. Because it's efficient and well understood and emphasises accuracy.

