Hacker News new | past | comments | ask | show | jobs | submit login

Sounds obvious... of course, sometimes ORM and other abstraction layers conspire to hide that you are doing queries in a loop.



Years ago I was part of an effort to migrate a compute footprint from one data center to another that was across town.

When the DB VMs were migrated suddenly we had complaints about certain ETL jobs going from minutes to days.

I opened up a packet capture and took a look (thankfully at the time the queries weren’t encrypted). What I saw was that this big name ETL tool was loading data one row at a time using one query per row. This apparently was ok when the DB and app server were in the same building but adding a few additional milliseconds to each query totally screwed the jobs.

Thankfully there was a parameter that could be set in the DB driver to fix the issue and we just had to convince the ETL tool vendor to set that.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: