
Ask HN: Tools to call REST API and export to CSV? - gabrielsroka
I&#x27;ve written my own in JavaScript, Python, PowerShell, and I&#x27;ve used Microsoft Power BI&#x2F;Power Query, but they are slow, don&#x27;t work well, require custom code, etc.<p>I need to:<p>1. call REST APIs (with Authorization HTTP header) and get JSON (arrays of objects), also parent-child endpoints (parent has IDs of child objects) -- nested loops<p>2. paginate using the Link HTTP response header, observe rate limits (HTTP response headers, to avoid 429 errors) and concurrency limits<p>3. (nice to have) select certain fields from the JSON object; also, data can be &quot;L-shaped&quot; (not every record has the same fields)<p>4. export to CSV<p>Open source, cross-platform would be good. Needs to be easy for not-very-technical end users to use.
======
nhayfield
datatables.net is a good modular javascript library that handles pagination
and child tables and export to csv/excel. you will still need to write some
code for formatting the api calls into a useable format.

~~~
gabrielsroka
Thanks for the tip.

I gave it a try, and got something really basic to work. It took me several
hours (part of it was my fault, part of it was the documentation and its
defaults that expect the data in a particular format, and the error messages
that weren't very helpful).

It's a complex tool, and it does a lot, but not what I need.

~~~
nhayfield
There are no silver bullets here. It is always alot of work.

~~~
gabrielsroka
I agree. I've already written 1,000s of lines of code for this. I was trying
to a) see if the problem had already been solved, b) see if it was worthwhile
for me to continue trying to solve it.

The MS Power Query / BI tools come really close, but not quite.

As I like to say, they do the first 80%, but it's that last 80% that you still
need :)

------
majkinetor
What was wrong with powershell ? You could make it faster if you run using
jobs/runspeaces and/or distribute on several machines. Should be dead easy.

~~~
gabrielsroka
Nothing was wrong with it. It works fine, but I need, basically, a GUI or
something simple that non-programmers/non-technical people can use.

I guess I need to wrap my code with something easy. I was trying to avoid
reinventing the wheel, but, so far, I haven't found a wheel.

