ok, so 20k * 60 minutes * 24 hours * 365 days * 3 years = 31,536,000,000 rows. You are querying 31.5 billion rows on a machine with 4 cores and 8gb ram? Are queries that return in 5-10 seconds running over the entire table? or small portions of it?
small portions of it, sometimes an hour or two, sometimes a day, or a week.
most of the times there are 5-6 other conditions.
pandas will have to full scan entire dataframe for any query to filter, while SQL uses index seek
