

Ask HN: What "infrastructure" do you use at your startup/company? - paulsingh

I've been thinking about the idea that most startups/companies are generally the same -- at some point, they all have to deal with issues like billing, accounting, provisioning, newsletter/email delivery, banking, IT, etc.<p>What tools/services do you currently use? If you had to do it all over again, what would you do differently?
======
jasonkester
We're a Microsoft shop, so the sweet spot for us is a small number of fast
servers. We pay for tools and server licenses, but surprisingly less than you
might expect.

Hardware: Half a cage at a colo with one fast box for web/db server, and a few
cheap pizza boxes for build server and misc. This used to be two boxes in a
garage behind a business DSL until Twiddla went big.

Content storage & delivery: S3/Cloudfront

Heavy lifting: EC2. We spin up a half dozen boxes each night for a few hours
of processing for S3stat.

Email: Google Apps (they're terrible, but free and functional)

Tools: MSDN + EmpowerISV = $400 for all the dev tools for the shop. Small
Business Server for the boxes in the cage. ReSharper, CodeSmith & RedGate's
SQL stuff are actually the biggest per dev cost, but they make life so much
better that they're worth it.

All in, it works out to about $500/month to keep up and running. Total, across
all our sites (and there are quite a few). The colo is expensive, but it's so
nice. Even with the cloud stuff where it is today, I'd still put a box in a
datacenter if I had it to do over again.

~~~
grep
"until Twiddla went big"

How big to have to change?

"half dozen boxes ... for S3stat"

What kind of boxes do you use and how much do you need to process?

(I use 2 of your services and I love it)

~~~
jasonkester
We moved into the Colo when Twiddla got accepted into the SXSW Interactive
Awards. Good thing too, because we won our category, got written up by every
major tech blog simultaneously, and spent a few days dealing with 1000
pageviews per second. This was before EC2 came out with Windows instances, so
the machine was pretty much on its own. Today if the same thing happened, we
could spin up a dozen EC2 boxes to deal with the spike.

That one box handled the load remarkably well, especially considering that I
hadn't done much backend optimization at all, and that we were still
persisting every single message to the database. It went a long way towards
solidifying my faith in the MS stack, and making me scratch my head whenever I
see somebody's blog fall over from a minor HackerNewsing.

As to S3stat, it runs on EC2 m1.small instances. Towards the end of the month,
when each bucket takes longer to process, we'll step up the number of workers
we fire up each night, and occasionally pull in a few c1.medium instances to
speed things along.

It's a fun architecture, and I've been meaning to write it up for a while now.
It uses six distinct AWS offerings to do its thing.

------
nethergoat
Business: Social games for gamers (<http://www.ea2d.com>)

Languages: Java backend, AS3 frontend, Python + Ruby misc

Note that we're very new (and hiring!), so much of this is still being built
out.

Software lifecycle: git for SCM (GitHub as central repo); ant for builds; ivy
for dependency management; Bamboo for build automation; Crucible for code
review; JIRA for issue tracking; Greenhopper for sprint planning; Vagrant for
containerized development environments

Systems infrastructure: AWS (EC2 w/ELB+ASG and S3 for now, more services to
come); nginx-fronted Tomcat for JVM; Cassandra for persistent DB; chef for
configuration management (currently chef-solo, but evaluating Opscode
Platform); PoolParty+Fabric for one-click deploy server fleets; Nagios +
Pingdom for monitoring; Mixpanel + Kontagent for analytics; Dynect (likely)
for DNS, possibly for GSLB; Akamai (likely) for CDN, possibly for GSLB

Misc: PagerDuty for on-call stuff; Google Apps for email, shared docs, etc;
Confluence for wiki; OpenFire (Jabber) for chat

~~~
ido
Sounds neat, but is a subsidiary of EA really considered a "start up"?

~~~
nethergoat
I definitely wouldn't consider EA2D a startup, but fortunately the OP left it
open ("startup/company").

That being said, I do think we (EA2D) have a lot in common with later-stage
startups. Three big factors are our autonomy (near-complete), self-sufficiency
(product, dev, marketing, ops, etc. is internal to our studio) and size (just
27 people across all disciplines). I bet our stack more closely resembles that
of a startup than that of an enterprise.

------
enjo
* Code Management/Bug tracking: <http://unfuddle.com> \- I love these guys. The bug tracking is fairly solid and they serve as an excellent source code host

* Hardware : Amazon EC2. We also use cloudfront (CDN) and a couple of other services. It's worked out amazingly well really.

* Accounting: My wife (PhD in Accounting). She uses Quickbooks I think. We use ADP for our payroll for quite awhile and that was _nightmare_. They were late with W2's and failed to pay the proper taxes they said they were paying. Just awful. My wife does payroll now.

* Email: We use authsmtp as our gateway. They've been very solid. Very few spam box issues. We also use mailchimp to manage our actual contact lists.

* Analytics: we use a combination of homegrown tools, google analytics, UserFly and CrazyEgg for usability and conversion analysis.

* Email: we use Google Apps. That's also worked out extremely well for us. No real issues and we rely on Google Docs quite a bit.

* Other services: We use Twilio to handle phone-tracking (it's an integral part of our product). We used UserVoice at one point and that was excellent.

I'll update with other stuff as I think about it.

------
dangrossman
Billing: Merchant accounts and Authorizenet's gateway, plus PayPal

Accounting: For analysis/forecasting, a custom programmed dashboard. For
bookkeeping, Quickbooks.

Newsletters: MailChimp

Everythign else e-mail: Rackspace Email

Banking: The local bank

IT: If hardware breaks, Softlayer, Linode and Amazon take care of that.
Anything else is my job.

------
pegmanm
Business : Cirrii.com

Email: Google Apps works fine for everything we need.

Code Management : Local Redmine install. Mutli repo and multi project is just
what we need for a bunch or side projects or prototypes. Github for personal
stuff. The issue tracking is not where it needs to be yet for an production
project yet but getting there.

Deployments: Fabric and Chef-Solo. Internally a lot of virtualbox vm's.

EC2, Rackspace and hosteurope.de with : Haproxy lb's, powerDNS homegrown
GeoDNS, AWS Cloudfront CDN.

HTH

------
scorpion032
Servers: Linode, Slicehost, EC2

Project Management/Version Control: Unfuddle, Github

Billing and customer tracking: freshbooks, highrise

~~~
shabda
I saw your comment and though, damn they use _Exact_ same tools as us. Then I
saw your username. :)

------
ehoward
Billing: Recurly.

Accounting: Quickbooks

Provisioning: Home grown on EC2/GoGrid/Rackspace

Banking: Local bank

Newsletter: AWeber

IT: Depends. Most I handle, but some of the mundane is outsourced.

SCM: HG @ Codebase

If I had to do it over again, I wouldn't obsess as much as I did over the
infrastructure. Check and re-check your backup & disaster recovery plan.

~~~
zackattack
Is Recurly worth it? One of their team just sent me an email suggesting I
check 'em out.

~~~
dangrossman
I would never, ever trust those guys again.

They pulled a massive bait-and-switch on their first users. Overnight, without
notice, they changed their long time published pricing from pay-as-you-go (fee
per transaction) to high monthly fees. For many, the cost was orders of
magnitude higher. They arbitrarily decided the API would only remain
accessible if you bought the highest priced plan.

It'd be fine if they gave notice of the change or grandfathered people in, but
at least initially, they didn't. They just wrote condescending, insulting blog
posts telling us all how we should appreciate their rate hikes. If you weren't
happy with (or able to pay) the higher prices and had already integrated
Recurly with your site, you were essentially held hostage. They have all your
customer billing information, not you. If you want to keep billing customers,
your revenue!, while you strip out Recurly and implement some other payment
system and get all your users to sign up again, you've gotta pay to keep your
account open in the meantime.

Eventually they figured out that screwing with their users like this wasn't a
good idea after all, backtracked and came up with a new pricing scheme more
like the original, but it's really too late at that point. Revenue is what
makes a business. You don't play games with who you trust that to. And nobody
should ever trust it to Recurly, since they're more than willing to play games
with you.

If you want to outsource subscription management, check out Spreedly and
Chargify. They've proven themselves more trustworthy than Recurly by far.
Really, I recommend doing it yourself though, it's not that bad. PayPal
subscriptions and IPN can take you a long way with minimal coding.

------
edanm
I use Codebase to for source code management (mercurial) and for project
ticketing.

I use Google Wave for most in-project communications.

Python/Django + jquery for dev.

WebFaction for web hosting <\--- Opportunity here, there's definitely an
opportunity to make simpler hosting for Django site. I believe there are
people working on it already, though.

------
paolomaffei
WordPress (and a lot of plugins) DropBox Remember the Milk Basecamp Linode +
Virtualmin/Webmin for hosting Paypal G Adwords G Analytics G Apps for emails
and calendar

~~~
paolomaffei
Actually I forgot Zoho invoice, Coda, Subversion and OpenOffice

------
iuguy
When we started, we had a single Linux server and some laptops. Over time we
added a VMWare server for testing and a VPS. Now our main file server is a Mac
(as we didn't want to run Windows servers and needed PGP Whole Disk
Encryption), we have a test VM Server and a couple of VPSs.

------
gosuri
Code - Github Billing - Recruly Accounting - Freshbooks Newsletter - Mailchimp
Email Delivery - Sendgrid

