I don't think any of our startups would exist without open source software. We use free operating systems, compilers, scripting languages, databases, web servers, content management systems... where would your startup be if you had to pay for all of these? I think those of us with successful startups owe it to those who came before us to help out those who come after us.
Stripe supports open source software by running an open source retreat. Tarsnap doesn't have unicorn-scale resources, but sponsors FreeBSD developer summits and the FreeBSD Foundation. How does your startup contribute back?
Everything we do[1][2] at Fogbeam Labs is Open Source. And I mean, really, truly Open Source, not "faux-pen source". Our code is on GitHub, most everything is Apache ALv2 licensed, the issue tracker is public, and we use open mailing lists / IRC for project communication.
We try to avoid the "throw it over the wall" style of development in that sense. The biggest wart with our approach right now is that we don't have a lot of community collaborators who aren't part of Fogbeam, so there's not a lot of discussion with "the community". But that's because we haven't grown much of a community yet to discuss with.
We are also open to receiving pull requests / patches and don't ask for a CLA or any copyright assignment. The only expectation is that any contributions are licensed under the Apache ALv2 or corresponding license (which will always be a mainstream, OSI approved OSS license, not some weird license we make up ourselves).
[1]: http://fogbeam.github.io/
[2]: https://github.com/fogbeam