
Web Service Efficiency at Instagram with Python - ksashikumar
https://engineering.instagram.com/web-service-efficiency-at-instagram-with-python-4976d078e366#.s1st93t8g
======
piquadrat
Instagram is a great story for Django, it's the best argument to bring against
doubters of Django's scalability. But I wonder how much Instagram contributes
to Django itself. I follow Django's development relatively closely, and I'm
not aware of any major work contributed by Instagram engineers. They aren't
listed on Django's fundraising page, neither.

[https://www.djangoproject.com/fundraising/](https://www.djangoproject.com/fundraising/)

~~~
fefifofu
The fundraising page is depressing. They're only asking for US$200,000 and
they only have 19% at half way through the year. In other news... Instagram
hits half a billion monthly active users... but not on Django's fundraising
page.

I'm not knowledgeable about open-source issues, pros/cons, etc., but man...
only 19% of $200K. Doesn't seem right given Python is one of the major
languages and I believe Django is the most popular web framework in Python.

~~~
J0-onas
You'd think that companies who profit from open source projects would support
them financially. Considering how big django is there should be many other
companies who profit from the projekt.

200k/year should be collected in no time...

~~~
wrigby
I've found that the engineers who build products on open source software very
rarely have the authority to donate company money, and the businesspeople that
do have the authority rarely understand the benefits of it.

I think a guide on how to get your company involved in donating to OSS
projects would be beneficial. Has anyone come across any good advice, or have
any experience in influencing your employer to donate?

~~~
wiz21c
Funding an open source project you use means funding the competitors.

A is a company, B is an individual with lots of spare time.

B feels that he can help the company by writing a tool for them. He also
envisions that the tool could be used by other companies and so wants to open
source it so he can provide services around the tool.

B says to A : see, I'm going to develop the tool for you for free, in exchange
you test it.

Note the tool is the n-th implementation of common business practices, such as
an ERP. The tool is not about something specific to A.

B thinks : I'm independent, I'm not hired by A, I write the code as free
software as it pleases me.

A then says : wtf? you're going to publish that code on the web ? Therefore my
competitor might get it so, no way. I'd be liable for giving away some value
of the company to the competitor

So, instead of saying something like "hey, I use that program so let's fund it
a little", A actually says "by helping B to develop the program, I may
actually help my competitors, I'm in legal trouble".

B is somewhat screwed :-(

(I let you figure out if this actually happened to me :-))

~~~
Svenskunganka
I don't think that this applies to this very case though. Django was not made
for Instagram, it was made available to the general public for free, as an
open source framework. Instagram engineers only decided to use what was on the
table.

I also believe that Instagram _needs_ Django to be OSS, and the same goes for
any other framework/language/library that other companies use, as OSS in
itself is an extremely powerful label for a project. It brings together
developers to form a community with lots of great minds and inputs. It helps
feasible projects become production-ready and stable much faster than if it
was closed source as its use-cases are far more wider than what a single
company would provide.

Taking this from a competetive standpoint isn't very realistic in my opinion.
I dare say that if either Django or Python weren't open source software,
Instagram wouldn't be built on it in the first place.

I think it's absurd how Instagram can grow financially fat from the hard work
of the Django Software Foundation without even contributing a dime to their
fundraising or provide engineers exclusively to the DSF. IMHO Instagram should
bring that fundraising up to at least 100%, considering Instagram revenue is
estimated to hit $5.8 _billion_ USD by 2020, $160,000 USD per year is a small
price for the amount of value Django brings them.

It's a topic of morale though, Instagram does not owe the DSF anything, but I
guess this shows where Instagrams appreciation lies.

~~~
wiz21c
You're right

>>> It's a topic of morale though, Instagram does not owe the DSF anything,
but I guess this shows where Instagrams appreciation lies.

That's the point I was actually making. My experience that when you have
people brainwashed with "competitive advantage", "profit maximization", and
such, well, morale just doesn't exist anymore. In my case, I was surprised to
see how quickly someone I trusted can actually switch to this brainwashed
brain...

------
meteorfox
This part I'm skeptical:

    
    
      "Compared to CPU time, CPU instruction is a better metric,
      as it reports the same numbers regardless of CPU models
      and CPU loads for the same request."
    

CPU instructions will be more stable than CPU time for sure, and it does show
that their metric is stable based on their chart but a single CPU instruction
can take multiple cycles, especially if there are stalls in the pipeline or
other processes are "polluting" the cache. Depending on the CPU model, the
number of uops that can be issued concurrently varies, and the latency of
these instructions will also vary on the size, and access pattern to their
memory hierarchy.

Also, what about a change in compiler version? That can also vary the number
of instructions. Unless they are referring to a Python bytecode instruction as
a CPU instruction.

Would measuring CPI be a better indicative of their efficiency? They could
also track both, no need to settle for one.

~~~
zerohp
I agree with your skepticism. They will not detect a performance regression
when there is a change in instruction mix or cache miss ratio.

------
wheaties
Wow, CPU is the limiting factor? I've never been CPU bound in a Web app, only
I/O bound. Am I doing something wrong?

Granted, I don't use python anymore for Web dev. I'm using languages that can
handle 10x the traffic per box that Django used to be able to handle. This
isn't a knick against python. I love python. Just such a different
perspective.

~~~
hardwaresofton
This also immediately popped out to me as the wrong assumption to make (and
start off with, for that matter)... But I could see it being true if:

\- You are essentially maxing out incoming and outgoing network IO

\- You have no database to speak to (or your DB is in-memory)

In that case, I guess it is just a race to do some computation and fetch the
right data from RAM, then the limiting factors would be RAM access and the
number of the computations you could do to try and keep up with inbound
requests?

------
sorenjan
If engineering time wasn't an issue, how much would a company of Instagram's
scale save by switching from Python to a lower level language like C++?
Pretend you could wave a magic wand and overnight have the entire code base
transformed from Python to well written C++, how would that affect the
hardware requirements, and future development?

I guess what I'm really asking is what pros and cons are there with using
dynamic and relatively slow languages compared to faster languages like C++,
Java, or C#? My uninformed reaction is that it's easy for startups to get
started with these easy languages but that it leads to problems when it's time
to scale up. Dropbox is writing their own Python implementation, and Facebook
have their PHP to C++ compiler.

~~~
Scramblejams
It's important to remember that for a lot of network server purposes, your
code only has to be fast enough to saturate your network interface, and after
that there isn't much point in trying to speed things up.

Not sure how often that's the case in Instagram's use, but I've been using
Python and Twisted to saturate network interfaces effectively since 2003 :-),
and in those situations I've been more interested in the correctness
advantages of a good statically-typed language rather than the performance.

~~~
abiox
time-to-saturation is also worth considering.

------
spullara
They should really show what they are doing with their CPU. How many requests
per second and their characteristics? Generally Python (Ruby, etc) is terribly
inefficient relative to the next tier of languages like Java and Go. At
Twitter we were able to reduce number of servers by 10x and decrease latency
by 10x by moving the web application from Ruby to Scala/Java with no
architectural changes, essentially byte for byte (and bug for bug) compatible.
My guess is they have done this math and would rather pay for more servers and
take the latency hit rather than rewrite it.

~~~
deno
Did you at the time compare it with a JIT implementation like PyPy (for
Python, not sure what would be the Ruby equivalent)?

~~~
pritambaral
A JITing implementation of Ruby would by JRuby. Depending on JRuby's state at
that time, Twitter could have used JRuby and gradually rewritten parts of it
in Scala. Perhaps.

------
lifeisstillgood
I think on reflection however we need to remember, with all our sprints and
agile iterations, that just in Instagram billions of web requests are being
served by just dozens of developers based on code written over 25 years now,
following good practises and hard work - kudos to the Python Dev team on 25
years.

~~~
dalke
To be fair, that's more like "started 25 years ago". Very little of the
original Python 1.0 code exists (especially given the great renaming when the
"Py"/"PY"-prefix occurred.)

I like to think of the thousands of developers involved. ;)

For example, assuming Instagram uses gcc to build Python or any of the other
components, gcc started in 1987.

Parts of glibc have copyrights dating from the 1980s, like netinet/tcp.h
Copyright (c) 1982, The Regents of the University of California .

~~~
lifeisstillgood
Of course ...

------
instakill
If any Instagram people are reading this, please for the love of all that is
sane, please I beg you add some kind of progress indicator for your API.

3 weeks in, I find myself loading my app's API client page to see if it's
still stuck in a greyed out "go live" mode.

I've got an "Instagram API" search saved for twitter and StackOverflow to
check checking if anyone else gets to slip through but others struggle just as
much.

Just a little transparency and predictability is all that we ask for.

------
pm90
Wish they would share some of that profiling code...

------
lifeisstillgood
I am digging into a gnarly bug in thousands of lines in python 2.6 right now
and, well, a custom cProfile, C types support and auto-reporting sounds ...
Lovely :-)

But I _know_ that they will have their rough edge cases and developers banging
their heads on the desk making this custom code for their needs.

But the grass does seem greener today :-)

------
acd
Is Pypy used?

[http://speed.pypy.org/](http://speed.pypy.org/)

------
fovc
I wonder if they looked into using existing APM solutions before building out
the profiler and monitor

