
Show HN: I'm working on an easy way to add data to files. Thoughts? - padho
https://github.com/midas-science/midas
======
mtmail
You have extractor and loader classes, I think the industry term at least in
data warehousing is a
[https://en.wikipedia.org/wiki/Extract,_transform,_load](https://en.wikipedia.org/wiki/Extract,_transform,_load)
system. Enrichment sounds cool, too.

I like it, at least more than Zapier and similar pipelines. We have a couple
of clients that use our API (geocoding[1]) to enrich their records inside an
ETL. Be prepared that some APIs are rate-limited or quota-based, need API keys
or other authentication and might return 'over-quota' error messages, never
mind latency.

[1] [https://geocoder.opencagedata.com/](https://geocoder.opencagedata.com/)

~~~
padho
Hi, thanks for your comment and the link to your Geocoder. I think it's
exactly the use case that I faced several times in the past.

I just saw that the Geocoder API is free with certain limitations. If I find
some free time, I would like to create an enricher for it. Maybe it can help
your clients (and other people of course) to save some time :)

------
lozzo
my thoughts: I was looking for the way you handle exceptions. And found none.
For me any enrichment API needs to be designed to handle exceptional case and
make it easy to deal with them.

Say that during enrichment I might want to do some lookups what if one of them
is not working...

~~~
padho
Hi, thanks for your feedback. You’re right. There is no exception handling
yet. I also plan to add some kind of state handling, so that jobs can be
paused/resumed. What do you mean by making lookups?

