You lose the ability to tweak and test one of the most important parts of the customer acquisition experience. There's a >10% conversion rate gap between different checkout flows I've tested on one of my ecommerce sites.
I've written my own subscription code for Authorize.net; it's around 200 lines, pretty straightforward (take the form data, POST to authnet to create a customer profile, POST to authnet to create a payment profile to store the card, POST to authnet to authorize the first payment, display any errors, otherwise save the reference token to the database and display a thank-you). It'd be smaller if you used a library like ActiveMerchant. Rebilling is a 15 line script as a nightly cron job.
I've also done integrations with Spreedly and Recurly. I didn't find they saved me any time; I had to write lots of code to send them information about the customers and process all their postbacks that you simply don't have to write if you do the charges yourself. Recurly was also changing its pricing and publicly arguing with its customers at the time, which killed my trust in the company.
Outsourcing your subscriptions to one of these startups is a business risk. They will forever be able to hold your cash flow hostage if they want to. If, however unlikely, your account is suddenly closed or they raise prices beyond what you can afford, you not only have to rip out all the integration you did and find another way to bill customers, but you will lose some of your customers while you're explaining why they can't manage their subscriptions at whatever.com anymore and begging them to fill out another payment form so you can continue billing them through someone else.
I read a few posts on forums that if the customer financial data is ever on a page you wrote, you come under the realm of at least some PCI compliance because of a possible XSS attack getting that information out of your forms.
That made the hosted "checkout" pages seem a lot more attractive to me... curious how you handled this.
Thing is, everything PCIDSS says you have to do when payment data passes through your servers is security 101. If you're not already doing 90% of these things, you're just waiting to be hacked anyway. Instead of thinking of it as a burden for payment processing, think of it as something every professional online business should have been doing anyway.
Lots of bark, absolutely no bite. There is not a single case where a bank will shut you off and stop getting your processing fees.