
How ZenPayroll processes billions of dollars in annual payroll - sergiotapia
http://stackshare.io/zenpayroll/how-zenpayroll-processes-billions-of-dollars-in-annual-payroll
======
tonyhb
It's really cool that they're very public about the things they've built (ACH
formats, fixed width file gems etc.). They seem like a very friendly and open
bunch of chaps with their customer's needs put first.

I'm surprised to see that they're solely ruby-based, though. Especially for an
SOA setup. I tried building a payroll system in rails one time and didn't like
it. That a dynamically typed language with strong metaprogramming was handling
people's money really didn't feel right. Then again, tools like Grape (which
looks awesome) must definitely outweigh that non-issue.

~~~
panh29
Disclosure: I am an engineer here with ZenPayroll.

In terms of language choice, it is actually quite nice to see definitions of
taxes, rules, and forms for different states and counties in a dynamic
language like Ruby without verbosity of other languages like Java. As
mentioned in another block post[1], when launching to new states, we paused
and built tools to help us automate the verbose tasks, and Ruby really helps
us there. We have good test coverage so we have high level of confidence when
refactoring (which we do frequently).

In terms of JRuby, we are only using it on one of the components that needs to
talk to a Oracle DB using JDBC.

We are early in the service oriented architecture path and will be learning
(hard) lessons a long the way with mistakes and from new talented engineers
joining us. What I am most impressed with this team of engineers is a
collaborative culture and willingness to go extra miles for customers and for
ourselves.

That said, I wouldn't be surprised that one day ZenPayroll technical stack
would be polyglot.

[1] [http://engineering.zenpayroll.com/benefits-of-writing-a-
dsl/](http://engineering.zenpayroll.com/benefits-of-writing-a-dsl/)

~~~
teacup50
... except that the choice isn't between Java or type-unsafe code handling
people's money.

That's a silly (and false) dichotomy.

------
mandeepj
Few questions if someone either from ZenPayroll or having similar experience
can answer -

1\. How you handle maintenance like upgrades in applications hosted on
multiple machines? Do you have a particular maintenance time period where you
bring down the whole application? For some applications, it is not possible to
shutdown the website completely. Even if you try to upgrade the application
servers one by one there might be active users on that server. How do you
handle this situation?

2\. The database tier is the hardest to scale. How do you do it? There is an
option to go master\slave route but it is hard. For performance purposes we
can't always talk with database. How you sync data in case you plan to use
cache.

Thanks.

~~~
volkadav
(I'm not a ZP employee, but I have worked in environments of significant scale
a fair amount. The below is just my opinion...)

#1: The usual approach is something like remove app servers from the load
balancers' rotation, upgrade them, run some sort of tests to ensure the
upgrade is ok, add them back into the LB rotation. Some environments might use
user traffic as the test step (e.g. send 5% of your users at the newly
upgraded machines, watch monitoring to see how response times and error counts
behave).

#2: The answer here really depends on what scale you're operating at. DB
replication (e.g. read slaves if your traffic is more read-heavy that write
heavy) and sufficiently high-spec machines combined with intelligent use of
caching can take you a fairly long way. I hesitate to give hard numbers, but
off the cuff probably 99% of sites will never need more than this. There's not
really a recipe to follow for that, the data needs and design/ops parameters
of an organization tend to be fairly specific to that org; more or less it's
good schema design, thinking hard about how fresh data has to be at the edge,
caching what you can with careful management of expiry times, and having an
ops team comfortable with managing replication. Beyond that scale you start
getting into globe-spanning multiple-datacenter voodoo rocket science. :) Ask
amazon, google or facebook (some of their work has been mentioned publicly,
things like cassandra from fb and various papers published by google/amazon).

------
G650
Just amazing.

~~~
pbreit
Perhaps. But orders of magnitude more payroll has been conducted by supposedly
prehistoric banks for the past century or 2.

~~~
enraged_camel
Yes, but how many of those banks openly and transparently discuss their
technology stack and processes?

~~~
smackfu
A bunch of old PL/I programs that run once very pay period is not very
exciting.

------
zdw
Ah, but can they generate [http://ledger-cli.org](http://ledger-cli.org)
files?

