| ||Ask HN: Cost-effective hardware for text processing?|
3 points by jason_slack 5 days ago | hide | past | web | favorite | 3 comments |
|I need to process 15,000 text files each day. This includes un-raring, manipulating data, creating a CSV, etc etc. it’s a lot. Each file is from 1mb to over 100mb. I can’t change how I get this data.|
I currently use a combination of rar, gnu parallel, python, pandas, rename, sed, in a bash script. Debian 9.6.
Having 60gb backlog plus new data each day my laptop doesn’t keep up. I need to have a dedicated setup for this.
Ideas on doing this cheaply but still pretty quick? What are hardware ideas? Would a Pi cluster be effective? A used PC would be fine but there are so many processors and motherboards all slightly different.
Edit: I could switch to C++ using system() calls. Then eventually replace these calls with code to do the work that makes sense.
| Apply to YC