
How we keep GitHub fast - janerik
https://github.com/blog/1252-how-we-keep-github-fast
======
aggronn
Should I feel bad knowing that if I were project manager, I'd feel like
someone wasted valuable time making the dashboard look so pretty? Am I an
idiot just for assuming that was done in house?

Edit:

I'm not trying to make a case either way, but I look at the color scheme,
specific typography decisions, and other small things that would have taken a
non-trivial amount of time to work on or think about (ie, more than 10 minutes
of actual focus) and think to myself: "If that were me, I'd feel like I was
just goofing off"

Get the information there. Make sure its nice and and usable. This does more
than that though, which is great. I personally just would feel like I was
wasting time and energy worrying about picking a color scheme when I had other
work to be done.

Third thought: I appreciate the romantic justifications mentioned in the
replies. I think they're fairly superficial, but they are romantic, and thats
fine. I think the real difference here (and the root of the disagreement) has
to do with why I associate this with 'waste of time'. There is obviously
opportunity cost of doing this. I expect people most in favor of this stuff
are salaried and not really concerned with opportunity cost. If I were
salaried, I wouldn't feel like I was doing something wrong by doing this work
because I'd do in _in addition_ to my other work. I'd stay late and do it
because I enjoyed doing it. I don't have that luxury where there isn't much
opportunity cost because I'm paid by the hour. I don't work over-time. I'm not
going to spend an extra hour at work one day to take care of this, so if its
going to get done, its going to have to be prioritized over my other work. If
picking colors (however important it is) is the most important thing on my to-
do list, I have a problem.

 __TLDR __: I'm paid by the hour. Thats probably why I feel this way.

~~~
kalid
"When you’re a carpenter making a beautiful chest of drawers, you’re not going
to use a piece of plywood on the back, even though it faces the wall and
nobody will ever see it. You’ll know it’s there, so you’re going to use a
beautiful piece of wood on the back. For you to sleep well at night, the
aesthetic, the quality, has to be carried all the way through."

-Steve Jobs

~~~
stephengillie
That's not actually true. You'll use the plywood because it's stronger and
because it's not seen. You get both the cost savings and increased structural
stability, which will get you more customers.

~~~
bkbleikamp
That's the difference between Ikea and a company like
<http://www.urbancase.com/>. Craftsmen don't think that way.

~~~
majormajor
Craftsmen absolutely think about what's stronger and more durable, not just
what's nicest looking (especially for non-visible parts!). I've got a few
pieces of pretty decent antique furniture that's made its way down to me in my
family (stuff made by craftsmen, not by mass production), and the hidden
pieces aren't finished or styled to the same degree as the front ones. The
purpose of the back and other non-visible parts is to hold stuff together, not
to look nice. I would also point out, from your own link, the specs for their
"the ledge":

 _"available with a solid domestic walnut top, sides and doors with a plywood
bottom and low voc finish or painted mdf."_

(Of course, these little details about what's "the right way to do it" are
rather irrelevant to the greater point, which is that if you're the type to
want to do things the right way, you're going to do things that way. Just
don't hate on plywood, it has its uses (and feel free to come up with the
programming equivalent!))

Edit -- some more notes on plywood: a sizable piece of plywood (like for the
back of a chest of drawers or bookshelf) is going to be stronger (especially
with regard to bending, IIRC) and vastly cheaper than a solid piece of wood
since it only needs thin pieces of veneer with good grain for the outer
layers. And IMO, it looks nicer than having several individual solid-wood
boards jointed together in parallel.

~~~
jonah
It's pretty universal.

We have some amazing antique Persian and Korean chests in the house. Hand
carved and split grain matched fronts and maybe sides, but the backs are plain
and/or rough hewn. These pre-date plywood, but the same goes - don't waste
effort on parts that will never be seen, just build them to do their job.

------
wamatt
Github is generally pretty nippy for me, and it's a pleasure to work with.

Just 2 performance concerns I've noticed:

1\. The network graph
[https://github.com/<username>/<repo>/network](https://github.com/<username>/<repo>/network)
seems very sluggish to render

2\. The API is slower than normal for HTTPS

Usage: $> time curl -i [https://<site>/](https://<site>/) For comparison (2nd
request's):

* <https://www.googleapis.com/> _187ms_

* <https://graph.facebook.com/> _175ms_

* <https://api.github.com/> _400-700ms_

Not sure if anyone else noticed this

~~~
lazyjones
That's easy: googleapis.com and facebook.com use 1024 bit RSA keys, github
uses 2048 bit. So the initial handshake is slower and the performance on your
end is probably the main bottleneck.

(by the way, RSA labs deprecated the use of 1024 bit keys in 2003(!), so one
could say that googleapis and fb use snake oil rather than ssl ...)

~~~
wamatt
Good find!

Didn't realize Github were using 2048 bit keys. I would be interested in
knowing what the actual performance differences are between key sizes, given
RSA is used in the initial negotiation before symmetric SSL takes over.

>so one could say that googleapis and fb use snake oil rather than ssl ...)

Given that Paypal, my bank, Google and Facebook use 1024 keys, I think
labelling 1024 bit as "snake oil", might be a stretch. ;)

~~~
runejuhl
>Given that Paypal, my bank, Google and Facebook use 1024 keys, I think
labelling 1024 bit as "snake oil", might be a stretch. ;) It's no less snake
oil just because every con man sells it...

------
pestaa
A couple of questions:

\- What does the top chart represent that blends into the background?

\- What is so costly about rendering pages for Googlebot that it takes almost
twice as long as the average rendering time for public views? The throughput
is especially interesting as it is less than 30% of its public equivalent.

\- What does 1 cpu unit equal to?

\- Also interesting to me is the fact that API requests require the most
horsepower, yet also provide almost the lowest response time and very high
throughput. How API requests are so different from "browser" requests, since I
suspect both are powered by the same backend?

Sorry for my naivety.

~~~
kneath
The top chart is the throughput for the whole stack over the time period
listed in the dropdown.

Googlebot tends toward the opposite of the pages we optimize for. For example,
Googlebot is _really_ interested in the 38,291th page of your history.

I think 1 CPU unit is a core, but I'm not sure. It's mostly interesting in
terms of relative numbers.

And yes, both api and the web live in the same application servers.

~~~
pestaa
Thanks for your answers, nice insight. Great job on the dashboard, its
interface is really inspiring.

I still don't grasp how API can be so efficient while doing basically the same
thing as the public-facing site minus probably the HTML rendering, which leads
me to believe that rendering is damn resource-heavy in Rails.

~~~
kneath
Keep in mind that a huge portion of our requests are rendering code. Syntax
highlighted code. With line numbers.

------
spullara
Doesn't something have to be fast in order for it to be kept fast? My
experience using the Github web frontend is one mostly of frustrating
slowness, especially as of late.

~~~
wamatt
Upon further investigation, it's not just the API, it's the entire site.

A significant cause of the issue is their SSL implementation. Specifically it
negotiates a new connection _every time_ instead of resuming the session, eek!

[http://security.stackexchange.com/questions/5511/ssl-
session...](http://security.stackexchange.com/questions/5511/ssl-session-
resumption-and-ids)

[https://www.ssllabs.com/ssltest/analyze.html?d=github%2ecom&...](https://www.ssllabs.com/ssltest/analyze.html?d=github%2ecom&s=207%2e97%2e227%2e243)

~~~
nodesocket
Is this a problem with the default setup of Nginx out of the box, or has
GitHub intentionally disabled session resumption?

~~~
wamatt
Haven't tried the latest nginx install, but according to the wiki,
_ssl_session_cache_ controls this, and is set to _'none'_ by default.

<http://wiki.nginx.org/HttpSslModule>

As for Github, could just be a misconfiguration, or perhaps they had some more
deliberate reason in mind.

Good news is I just received confirmation from one of the ops guys. He says
they know about it, and are working on a fix. :)

~~~
nodesocket
Awesome, GitHub ops on it. :) Curious if there will be a speed improvement
once they enable session resumption.

Any idea why the default in nginx is: none. Soft off: nginx says to a client
that session can be resued, but nginx actually never reuses them. Seems like a
bad default to not cache.

------
DigitalSea
Wow, those are some of the nicest reporting tools I've ever seen. Usually any
application that involves measuring metrics like page load, average response
times and whatnot looks horrible. It might be a small thing, but wow the
design of that dashboard is really nice.

Github are absolutely killing it right now.

------
nodesocket
The performance dashboard and mission control bar look amazing. Would be great
if they could be open sourced. :)

~~~
technoweenie
Take a look at <https://github.com/nesquena/query_reviewer> (if you're using
MySQL).

------
skylan_q
This is how they keep an eye out for slow pages. This isn't how they keep the
site fast. :(

------
emehrkay
I find myself saying this a lot, but the Github guys do really cool work and
whenever I see something new from them, I just want to get home and code.

------
blaze33
Reminds me of [http://www.codinghorror.com/blog/2011/06/performance-is-a-
fe...](http://www.codinghorror.com/blog/2011/06/performance-is-a-feature.html)
:

"The simple act of putting a render time in the upper right hand corner of
every page we serve forced us to fix all our performance regressions and
omissions. [...]

Most of the performance fixes were trivial, and even the ones that were not
turned into fantastic opportunities to rearchitect and make things simpler and
faster for all of our users."

Stackexchange open sourced their profiler at <http://miniprofiler.com/> (.NET
& Ruby)

The post provides some data showing how they improved their site performance
over time where Github just states they have a strategy to use "powerful
internal tools that expose and explain performance metrics". Guess it's just
common sense now.

------
digitaltoad
The mission control bar looks very familiar. Although I can't remember the
name of it now, I seem to remember there being a debug bar of sorts that
worked with merb/rails projects that had the same type of look. Anyone
remember the name of it?

Edit:

I was thinking of FiveRuns Tune-Up toolbar.

~~~
particlebanana
Maybe your thinking of the old FiveRuns Tune-Up app. I used that all the time
when I was doing early rails projects. They got bought and shut down awhile
ago but I found an old TechCrunch article on it.

[http://techcrunch.com/2008/05/29/dont-debug-alone-with-
fiver...](http://techcrunch.com/2008/05/29/dont-debug-alone-with-fiveruns-
tuneup/)

~~~
thibaut_barrere
I think it is (I remembered the black bar too).

It seems to be there (but not updated):

<https://github.com/fiveruns/fiveruns_tuneup>

------
veidr
Now that Github is fast and beautiful, I personally hope they make it fast _in
Japan_.

------
jeremiep
Is it me or the new feature that hasn't shipped they mentioned is the source
code search input box next to the download button?

~~~
calvin
If so, that would be incredible. More than once I've found myself wanting to
search a repository for a specific word or variable without downloading it and
acking through it.

~~~
vrish88
You can actually do that already. Just go to github.com/user/repo/search

So: <https://github.com/rails/rails/search>

------
thebigpicture
Is "responsiveness" the same as "responsive design"?

~~~
bertzzie
I don't think it is. "Responsiveness" in this article's context is about how
responsive the page is, as in how fast it renders and shown to the user.

"Responsive design", OTOH, is about a page that adapats to the user's screen
resolution; eg: renders differently on 480x320 vs 1024x768, on the same code
base. HTH.

------
jaymoney
Github is fast now?

~~~
cjdrake
Hehe, I was just thinking this while looking at a Github page watching that
little cat with the spinning border; didn't want to be the one to say it :).

