This isn't content that will ever perform well in search engines. I'd even go so far as to say these results pollute the search results. They are an address and hopefully a tiny copied description. What added value does this have for a visitor? What added value does this have for a search engine to present high up in the results?
Look at how Yelp or LinkedIn got people to enter content on their listing pages. Or even Yahoo, look at all the widgets added to listings, to increase the content size and relevancy.
Conclusion: No (manual) penalty, but bad non-engaging duplicate content listings combined with a poor site architecture that allows zillions of page combinations with meager to no content on them. The issue is confounded by not making sure your site is canonical:
Site architecture and SEO is massively important for the success or failure of these kinds of websites, and both seem quite poorly thought out at a glance.
If the site is young, the initial boost in rankings (honeymoon period), will decay overtime and settle together with your page authority, content quality etc. If you see a decrease in visitors, you could attribute it to this initial popularity, instead of a penalty. Current visitor numbers might reflect your content / site quality a lot better.