
Netflix Billing Migration to AWS – Part III - samber
http://techblog.netflix.com/2016/08/netflix-billing-migration-to-aws-part.html
======
hrayr
It continues to surprise me how (why?) Netflix builds their infrastructure on
their #1 competitors platform. They're basically funding Amazon Prime. They're
undoubtedly smart, so they must have a lot of incentive in doing so.

~~~
thisisbad
It's a bad decision. It's similar to companies deciding to outsource their
communication to other parties (e.g. Slack) that can be hacked [1] and you can
be banned at any moment.

Don't outsource your infrastructure to 3rd parties (especially competitors)
just to save few dollars in the short term. Have control of your assets.

    
    
      [1] https://techcrunch.com/2015/03/27/slack-got-hacked/

~~~
frankchn
I don't think it is a big concern so long as the risks are acknowledged and
contracts are in place. Apple outsources production of its Apple Ax SoCs to
Samsung too.

------
secabeen
Netflix is well-known for running much of their infrastructure in AWS. What
infrastructure do they not run in AWS? I'm aware of their CDN, which is the
core of their business, and which they manage in-house. Are there other parts
of the Netflix service that they self-host?

~~~
keepper
It's pretty amazing Open Connect[1] doesn't get as much "fan fare" as Netflix
use of AWS. This is arguably the core of what they do ( push tons of bits to
the end user ), and something the cloud could not do for them.

[1] [https://openconnect.netflix.com/en/](https://openconnect.netflix.com/en/)

~~~
chupy
I am curious if the other streaming video providers are doing something on the
same principle to the Netflix Open Connect.

Just by assuming that 5-10% of their actual Netflix video traffic gets served
from AWS and the rest gets served from the Open Connect servers, it's probable
that any other competitor (except Amazon Video) would see huge savings with a
similar implementation.

------
jasonjei
Does Netflix have an integration with the ERP? Is the ERP self-hosted, on
cloud, or SaaS (like NetSuite)? Very curious to read if G/L is integrated or
if transactions are posted manually on an end-of-month basis.

Edit: it seems like the earlier blog post mentions there is a data feed to the
GL, but leaves this rather vague. Very interested to read their ERP and data
feed setup. What ERP does Netflix use too?

~~~
lmickh
Remember reading years ago somewhere that it was Oracle ERP, but can't find a
reference. That might have changed as they got closer to completing their
cloud migration.

------
nstj
> Some source tables had columns where Number meant an integer, in other cases
> it was used for decimal values, while some had really long values up to 38
> digits.

I'd be interested if anyone had any guesses as to what data type may require a
38 digit number.

~~~
twoodfin
log2(10^38) ≈ 126.

I bet these were 128 bit integers encoded in decimal.

~~~
yeukhon
Can you elaborate on this? I have not seen this technique.

~~~
nstj
Not that I condone using an integer as a UUID, but here's some discussion
around using a 128 bit integer to store one[0]

[0]: [http://stackoverflow.com/questions/18439520/is-
there-a-128-b...](http://stackoverflow.com/questions/18439520/is-
there-a-128-bit-integer-in-c)

------
merb
last time I asked why not PostgreSQL, now I have my answer. I love netflix's
posts. Way more than most other tech blogs and their responses on the how and
why is mostly more accurate than others ( _looking at uber_ ) especially their
expertise in evaluating stuff is extremly good. What I especially wondered was
why DRBD (i didn't had good experiences with it, neither with psql nor with
mysql) guess with a lot of engineers + the hope to migrate to aurora at some
day makes this a reasonable choice (i guess). Still funny that their whole
billing infrastructure runs on like two nodes for a while (the masters) while
I guess a lot of things there has a need for writes especially when the
monthly billing takes place.

~~~
brianwawok
You could send a monthly bill to the entire world in 2 servers I suspect.

~~~
Consultant32452
I wonder if you could bill the entire world from a single Raspberry Pi. I
suspect it could be done.

------
samlachance
I would love to see that monthly bill...

~~~
MrMullen
I wonder how all that works out for Netflix. They must get a massive discount
but do they buy reserved instances? or are all of their instances reserved
instances?

~~~
tomschlick
According to @jedberg... they don't get any discounts other than hitting those
top level tiers which are discounted. They pay the same rates as us peasants
for the lower levels.

And yes, I'd imagine they heavily use reserved instances as their traffic
patterns would be pretty predictable by now.

~~~
vidarh
I know people using a tiny little fraction of that which have negotiated
discounts well beyond the published prices at usage well below the top tiers.

If they do pay the public prices, they are being totally fleeced to the point
where you'd have to ask what in the world they are doing.

~~~
tomschlick
Thats what I would have thought too... just going off of what he has said in
previous comments directly to me.

~~~
vidarh
I guess he could be technically right in that they might overall be so far
into the "Contact Us" tier that they've just been given a steep enough
discount for that part of their usage to nullify the premium on the lower
tiers. Having Netflix people say they're paying the regular rates certainly is
good for Amazon PR.

------
user5994461
I'd love to see an estimation of NetFlix for running the same infrastructure
on GCE.

All the instances are cheaper and faster.

The networking is substantially faster, especially on smaller instances (I can
imagine they have a lot of network IO given their business).

No more reserved instances to manage. The discount is automatic on GCE when
instances ran 24/24 for the month. (Reserved instances are a management
nightmare when you have many servers, you can actually loose money).

Replace all spot instances with the equivalent on GCE, which has fix pricing
(about 70% discount) and is more dependable. (I imagine the core video
business is extremely spot friendly).

------
Scarbutt
With all the praise and hype postgresql gets, you wonder why all this awesome
services are mysql first, RDS? mysql first, aurora? mysql compatible, google
cloud sql? mysql, netflix? mysql, github? mysql.

I think mysql users are just there quietly making money while postgres users
are bashing mysql ;)

I also think services like aurora are strong selling point for starting new
tech businesses with mysql.

~~~
ben_jones
I'd be interested to see how many large companies have the courage to switch
their databases after they've reached a "mature" stage as a company. My guess
is a very small number have the technical expertise in order to make it worth
doing in the first place, meaning in many cases the database the CEO picked
when it was him in a basement learning to code for the first time is the one
used in production many years later.

~~~
brianwawok
Sure it depends on use. If someone is all ORM all the time they may change
fairly easily. If someone is all stored procedure all the time, I doubt they
ever migrate.

