
Balanced's Architecture - steveklabnik
http://blog.balancedpayments.com/balanceds-architecture/
======
jtchang
Solid architecture. For those not familiar with payment industry terms PCI
stands for Payment Card Industry. You usually know this as PCI DSS (Data
Security Standard). Basically all the payment card providers got together and
said that you need to follow these rules if you want to accept credit cards.
The rules are pretty straightforward ( have a firewall, don't save CVVs,
encrypt credit card #s at rest). You then pay an auditor to audit against PCI
DSS.

If I were looking for vulnerabilities I'd probably start with any XSS. Chances
are the credit card data is locked down tight and encrypted. But what if I can
scoop it up as it gets transformed into a token? Also look at where you store
the encryption keys to decrypt the card data. There are hardware devices you
can use that are especially hardened.

The problem with bitcoin is that it necessitates an even more secure
architecture because you don't have a 3rd party to run to if things hit the
fan. Suppose all your credit #s got stolen in this case. You can run to
Visa/Mastercard and they will invalidate the card #s in bulk. Or at least do
monitoring on them. What do you do when all your bitcoins are stolen?

~~~
steveklabnik
With the Coinbase integration, the OAuth secret is the equivalent of the card
number (and stored in knox like all other sensitive data, even though OAuth
tokens are outside of PCI scope), and so if they got stolen, could be
invalidated by Coinbase.

------
tlarkworthy
With a name like balanced, I was expecting double entry book keeping[1], so:
you give an account to a device, e.g. the credit card processor, and measure
how much money is entering and leaving the company per device. Plus you give
accounts to each customer and measure money going in and out their separate
channels.

Then you check these two different modalities of tracking money conclude your
the same liquidity measure for your company.

If they do not then you have sprung a leak somewhere and can halt everything.

That would be an extra layer of security. Passing loads of tokens round is
still a single point of failure at the conceptual level.

[1] [http://en.wikipedia.org/wiki/Double-
entry_bookkeeping_system](http://en.wikipedia.org/wiki/Double-
entry_bookkeeping_system)

~~~
steveklabnik
This post wasn't talking at all about ledgering, just about high-level
components.

If ledgering is interesting to you, [http://blog.balancedpayments.com/the-
ledger/](http://blog.balancedpayments.com/the-ledger/) and
[http://blog.balancedpayments.com/state-
machines/](http://blog.balancedpayments.com/state-machines/) are what you're
going to want to read.

~~~
tlarkworthy
oh great reads. Thanks

------
wtracy
(Former PayPal employee here.)

I first have to give you a thumbs-up for calling your fraud detection layer
"precog".

Does AWS let you firewall off Knox from the open internet? PayPal's
architecture has most of the machines that touch payments isolated behind
hardware firewalls, with only certain front-end machines able to punch through
the firewall.

~~~
TheHydroImpulse
Amazon has VPCs which are virtual private clouds. They let you configure
networks with specific requirements, such as being closed off from the outside
world.

Once closed off from the world, only your servers within the public subnets
can access those in the private subnet. By default, the private subnet can't
talk to the outside world. You'd typically setup a NAT instance in your public
subnet that tunnels your private subnet's internet to the outside world
(because the NAT is in a public subnet, it can access the outside world).

That's just an example setup. It's a very powerful tool for securing your
infrastructure. For example, you should typically put your databases, and
anything that isn't password protected that stores information or something
(except web servers) in a private network so that only your public servers
have access to them.

User -> Public Network -> Public Server -> Private Network -> Private Server
-> NAT (Tunnel) -> Public Network -> Internet

VPC does take quite a bit of effort to setup, but after that, it's pretty
straightforward.

------
whouweling
I wonder: is EC2 secure enough for this type of credit card store? What if the
management layer running the underlying hosts is vulnerable or a XEN zero-day
vulnerability shows up?

I'm sure Amazon does a lot on securing its infrastructure, but for credit card
data wouldn't a physical, fenced off server be more secure?

~~~
wolfwyrd
I suppose it comes down to the amount of investment available. Amazon can pour
resources into security, monitoring and have a large staff actively keeping an
eye on such things. They're signed off for PCI compliance Level 1[0] (Any
service provider that stores, processes and/or transmits over 300,000
transactions annually) which helps isolate you from a lot of costs around
getting your dedicated hardware audited yourself.

It's also worth noting that Amazon.com itself is hosted off AWS (since ~2010)
though I'm struggling to find a good cite for that

[0] [http://aws.amazon.com/compliance/pci-dss-
level-1-faqs/](http://aws.amazon.com/compliance/pci-dss-level-1-faqs/)

[1] [http://www.dummies.com/how-to/content/amazoncom-runs-on-
amaz...](http://www.dummies.com/how-to/content/amazoncom-runs-on-amazon-web-
services-aws.html)

------
vladgur
I love that plexiglass on the wall type of whiteboard. Is this a
premanufactured solution or homegrown?

~~~
mtamizi
We repurposed the glass tabletops from IKEA desks and threw them on the wall
for white boards ;-)

------
rch
Am I wrong to want to know more about how (for instance) communication between
the networks and components is implemented? This post, while well written,
doesn't really answer the questions I might ask about any given architecture.

~~~
steveklabnik
You're not wrong! You can only fit so much in one post, and I wanted to keep
it fairly high-level here.

I got asked this on Twitter, you might find the thread worthwhile:
[https://twitter.com/moritzheiber/status/442037431089786881](https://twitter.com/moritzheiber/status/442037431089786881)

It seems like a lot of people are interested in hearing more details, so I'll
try to get into that eventually in another post. Always more to talk about!

~~~
rch
Thanks very much. The question about latency is certainly relevant, and I
would add reliability to that too. I'd definitely be interested in a technical
follow-up, when you have the time.

------
Avalaxy
I would really love to use Balanced, I'm really excited for the escrow and the
bitcoin features... If only they accepted customers from the Netherlands.

~~~
steveklabnik
Yup, it's a bummer. We're working on it, but it's really hard. :/

------
agerlic
Great post and diagrams. How do you manage/centralize logs ?

~~~
steveklabnik
Thank you, they took forever to draw... My handwriting is terrible.

We wrote about logging here: [http://blog.balancedpayments.com/status-
page/](http://blog.balancedpayments.com/status-page/)

    
    
      > We already log these to a centralized server using RSYSLOG, so I already had
      > a data source to draw from. Next, I went and brewed a fresh pot of coffee and
      > bestowed it upon bninja for his prescient work in building our log parser,
      > Slurp. We wrote a quick Slurp script that read the HTTP status code from each
      > request and then fed them into Graphite buckets. Each bucket was based on
      > service name (DASHBOARD, API, JS) and then response code family (2xx, 3xx,
      > 4xx, 5xx, and a special case timeout for slow requests).
    

If infrastructure stuff is interesting to you, you may want to check out
[https://github.com/balanced/balanced-
infra](https://github.com/balanced/balanced-infra) . If there's interest, I
might blog about it in the future.

~~~
rafekett
what do you do when rsyslog drops log events?

~~~
steveklabnik
I am not an ops person, but my understanding is that rsyslog does 'store and
forward', which would allow replaying of the log if something was dropped:
[http://www.rsyslog.com/storing-and-forwarding-remote-
message...](http://www.rsyslog.com/storing-and-forwarding-remote-messages/)

------
KedarMhaswade
Great way to explain on a blog. Well done!

------
robbiet480
Who removed (YC W11) from the title of this, and why?

~~~
gwillen
The HN mods typically sanitize titles to match the title of the underlying
article. (Except when they don't, of course.) They are generally unwilling to
explain themselves, even when their edits are clearly making the title worse.

It's honestly a mystery to me why the system even lets submitters specify
titles.

~~~
robbiet480
The problem I have is that Stripe, a competitor to Balanced, seems to keep the
(YC S) emblem every time they have a story. It feels like YC is trying to hide
the fact that Balanced is a YC company I guess. Pair this along with the fact
that (IMHO) Balanced is killing it, but I only ever see PR about Stripe
(except for a few weeks ago, when Balanced announced a partnership with
CoinBase, another YC company)

But I could be paranoid.

~~~
subsection1h
> The problem I have is that Stripe, a competitor to Balanced, seems to keep
> the (YC S) emblem every time they have a story.

That doesn't seem to be the case:

[https://www.hnsearch.com/search#request/submissions&q=Stripe...](https://www.hnsearch.com/search#request/submissions&q=Stripe&sortby=score+desc&start=0)

------
thinkcomp
Yet another unlicensed money transmitter handling Bitcoin transactions
(through another unlicensed money transmitter). What could go wrong?

~~~
erichurkman
Are you posting on the right thread? Balanced Payments handles credit card and
ACH payments, not Bitcoin.

~~~
simonk
Balanced did start accepting Bitcoin through Coinbase
[http://techcrunch.com/2014/02/20/balanced-coinbase-
bitcoin/](http://techcrunch.com/2014/02/20/balanced-coinbase-bitcoin/)

~~~
steveklabnik
I expounded more on the technical details of that integration here:
[http://blog.balancedpayments.com/more-details-about-
bitcoin/](http://blog.balancedpayments.com/more-details-about-bitcoin/)

~~~
thinkcomp
Expound on the details of your legal status. You're an "open company," right?

~~~
icelancer
Ah yes, the scheduled trolling of payment companies by Aaron. Wouldn't miss it
for the world.

