At the time it was hard. There wasn't Google Patent search which is really useful for checking these things.
Basically I would look at what patents the fledgling startup had filed. Make sure they didn't suck, and that there wasn't lots of prior art.
You can be a start up and get bought for nothing more than a single patent that someone wants, so making sure the patents you have are not going to fall down when a bigger company wants them was important.
Often I would also be checking who beat the company to what ever they were doing, and then check those people's patent portfolio.
We didn't worry much about things like the Fat32 patent, we worried about things like hey, this isn't a real innovation these guys just read this other guys thesis paper and knocked it off. (Like Nick at Summly did)
From there a VC could decide if they were going to double down or cut their losses.
