
Upgrading GitHub from Rails 3.2 to 5.2 - masnick
https://githubengineering.com/upgrading-github-from-rails-3-2-to-5-2/
======
dasil003
No mention of the rails:update rake task, which is a very valuable tool to get
your boilerplate updated. I'm guessing at GitHub there is so much
customization over the years that they wouldn't get much out of it, but it's
still a valuable exercise to run through, and it's worthwhile to keep your
boilerplate aligned as much as possible since it makes gems, documentation,
and everything else more likely to align with shared community experience.

Also, I want to add a big proviso to the lesson "Upstream your tooling instead
of rolling your own". Historically in ruby, and now even moreso in node, the
ease of pushing new packages and the trendiness of the language at times has
led to a lot of churn and abandoning of packages. The trick is to include
stable dependencies, and that requires quite a bit of experience and tea-leaf
reading to do right. Often times maintaining your own stripped down internal
lib can be a performance _and_ maintenance cost win over including a larger
batteries-included lib that ends up being poorly supported over time. For
example, a lot of people got burned by using the state_machine gem that at one
time was very hot and actively maintained but went on to get left in an
aggressive limbo
([https://github.com/pluginaweek/state_machine](https://github.com/pluginaweek/state_machine)).

~~~
caseyf
> Also, I want to add a big proviso to the lesson "Upstream your tooling
> instead of rolling your own".

I feel like this bit needed more of an explanation about how this applied to
GitHub.

If I were to write a post about working in a 10 year old Ruby codebase I'd
definitely include "Kill your dependencies" as a bullet point.

~~~
tcopeland
> I'd definitely include "Kill your dependencies" as a bullet point

Or at least your monkeypatches!

------
lunaru
For my team, this article comes at some interesting timing, since we're
bumping into some of the same issues with Rails.

Rails is now a mature framework and part of the problem is its lack of
consideration for large existing codebases running in production. While there
are nice tools to help migrate (e.g. rails:update) that hit surface issues,
the deep problem is that there are a lot of decisions made going from version
to version that are obviously unfriendly to established projects. e.g.:
[https://github.com/rails/rails/issues/27231](https://github.com/rails/rails/issues/27231)

Additionally, there are a lot of gems that are losing momentum, which are
near-core to Rails. e.g.:
[https://github.com/thiagopradi/octopus/issues/490](https://github.com/thiagopradi/octopus/issues/490).
This is a side effect of the above issue, where the alternatives to Rails are
taking a lot of the community away to focus on newer/shiner things.
Fortunately, we have companies like GitHub and Shopify that are still very
much invested in the success of the ecosystem.

All that said, it's still a great framework to go from 0 to production with a
new idea or project.

Other ecosystems we're entrenched in (Node for example) have their share of
issues as well, but we won't go into those.

~~~
lostapathy
> art of the problem is its lack of consideration for large existing codebases
> running in production

To be fair - this has gotten a _lot_ better. Upgrading 2.3 -> 3.2 was
terrible. 3.2 -> 4.0 was terrible. 4.0 -> 4.1 was rough. Since then, I've
found the upgrades pretty easy - to the point I ran rails 5.2.0-rc's in
production for a while.

As you fairly note, a big problem is that related gems lose momentum and they
don't get updated - which blocks other updates. On the flip side, they usually
aren't that hard to update and submit a PR on, either.

Even not having those patches merged quickly is not so bad in ruby - it's easy
to tell bundler to look at your fork of a gem on github rather than pulling
the upstream.

~~~
bhaak
What was your problem with going from 3.2 to 4.0?

I think the upgrade path from 2.3 being terrible is generally accepted as
being true. But I don't remember any hard problems from 3.x onward.

At least for Rails itself. Gems dependencies are another problem.

Like at one project we had pains every time Rails upgraded their minor version
because the previous devs thought using Squeel instead of the builtin
ActiveRecord a good idea. Just for being able to write slightly "nicer"
queries and now this is a major stopper for going to Rails 5.

~~~
lostapathy
It's been a while, honestly I'm not totally sure.

Generally speaking at that job, though, we tended to not wean ourselves off
the deprecated features - we'd use the extracted gem to keep the functionality
going. Which works fine for one minor release, but doesn't work when you are 3
minor releases later.

Strong parameters is one that hurt bad. We used the workaround gem to avoid
that for a long time, and it just made it more painful when we had to get rid
of it.

I think generally there were a lot of related gems that were hard to get
updated along the way from 3.2 to 4.0 as well. Seems like a fair amount of
libraries were late to the 4.0 party, then jumped ahead to 4.1 or 4.2, and
never really got ironed out well against 4.0, so you'd have goofy
compatibility issues.

Squeel was a related problem that was horribly painful to remove from that
stack as well, I forgot about that.

------
chris72205
Similar post from Shopify about a year ago on their experience upgrading from
Rails 4.2 to 5
[https://shopifyengineering.myshopify.com/blogs/engineering/u...](https://shopifyengineering.myshopify.com/blogs/engineering/upgrading-
shopify-to-rails-5-0)

~~~
tim333
And some similar discussion on "Shopify now on Rails 5.0. started 12 years ago
on 0.5, the First version released"
[https://news.ycombinator.com/item?id=13448219](https://news.ycombinator.com/item?id=13448219)

------
tjpnz
It's scarily common for organizations to be running ancient versions of Rails
in production. At my last gig we spent six months upgrading a Rails 2.3
application to 3.2 and before that I was working with a team that was
maintaining an application written in Rails 1x. Kudos to GitHub for sharing
this, I really hope they do future posts going into more detail. In my
experience one of the hardest aspects to upgrading Rails is that so much of
the really useful information has either fallen out of Google or succumbed to
link rot.

~~~
stouset
This is true of literally everything.

People don't upgrade their dependencies across the board, and it's a massive
problem for long-term security and maintainability.

~~~
jb3689
My current company doesn't even lock most of their dependencies. At first I
thought it was crazy (and it is crazy) but it does mean we address
compatibility issues immediately. We are mostly running background jobs though
so it's safer than anything customer facing

~~~
stouset
My opinions on this have changed a lot over the past few years. At this point,
I think anything that pins dependencies to specific versions is asking for
long-term maintainability nightmares. Updates of your dependencies, operating
system, language version, etc. should happen _weekly_ , and any instance where
you have to pin a dependency to an old version (e.g., a major version release
that has some compatibility issues) should be dealt with ASAP.

Obviously this isn't a tenable position in every circumstance, but I think it
should be the default. Particularly in a world where the vast majority of
security fixes go without an announcement or CVE.

~~~
maxwellg
Isn't that similar to golang's (now deprecated) stable HEAD philosophy? On
larger projects with tons of human resources that works out OK, but for a
smaller team wouldn't living on the bleeding edge & dealing with every issue
be a ton of work?

~~~
stouset
The only times I've ever run into issues updating dependencies (other than
across major version upgrades of deeply-integrated deps) is when people have
put it off for months or more.

Most updates aren't breaking. The overwhelming majority of updates that are
breaking are trivially discovered and fixed with one or two tweaks. Almost all
the rest can be fixed with a single search/replace.

Being eight minor versions (or two major versions) behind and having to find
and fix _all of these at once_ is when people land themselves into trouble.

------
hartator
> Upgrade early and upgrade often

Not sure about the upgrade early. It’s a different kind of pain to be one of
the first to use a new Rails version vs lagging a couple of months behind.

~~~
gsnedders
I think there's two sane options:

* track master, and notice breakage as soon as possible (you get much smaller deltas when trying to figure out breakage, which is always a big help), and are able to fix it or report it upstream ASAP, or * hold off slightly till the new release has had more bugs found post-release.

~~~
hartator
Main issue is more with other gems that need to be updated as well. It may
take take months. You can't do that on your own.

~~~
craigmcnamara
Fork, use your fork, send a pull request. Almost every rails upgrade I make
forks and send patches on gems that I need that I forgot about. I get good
responsiveness and merge rates. Most of the time a gem needs to be able to
support multiple rails versions, so in that case I usually send pull request
setting up the Appraisal gem to test for appropriate compatibility.

------
jbergstroem
As a comparison, here's gitlab's journey (issue opened March, 2016):
[https://gitlab.com/gitlab-org/gitlab-
ce/issues/14286](https://gitlab.com/gitlab-org/gitlab-ce/issues/14286)

Looks like the first scheduled milestone was 9.5 (a year ago) and the current
is set for 11.4 (next release).

------
Twirrim
> Upgrade early and upgrade often

It seems daft to keep seeing this lesson being learned by tech companies, and
keep seeing blog posts where most of the pain would have been handled easily
by just making upgrading a key feature.

Instead, tech managers and engineers seem to make the same mistakes over and
over again, delaying those upgrades, until suddenly they discover it's a hard
task to upgrade. I get delaying to _some_ degree, it's better to let other
people figure out those sharp bits on the bleeding edge for you, but you need
to set an explicit target for upgrading.

At another large tech company I worked for, it took the security team swinging
the sledgehammer to get teams to upgrade from known-vulnerable versions of
Ruby on Rails. When they came to do it, they discovered the changes were so
extreme that the effort involved in migrating was likely more than the effort
involved in a complete re-write (they did at least have pretty comprehensive
tests)

~~~
cortesoft
It is easy to say that in the abstract, but you always have a finite
time/resource budget for doing work. Effort spent upgrading is effort not
spent doing other important work. Is the other work more important in the long
run? The answer is not trivial to answer.

This is why we call it 'tech debt'.. it is just like any other debt. You take
it on because you don't have the current resources to avoid it, and you
calculate that it is worth taking it on. But then, you are carrying the
interest on it, and if you aren't careful it will grow to be unmanageable, and
all your dev effort goes into just paying the interest without paying the
principle.

~~~
Twirrim
It absolutely matters. Security is a feature. If you see upgrading as merely
technical debt, you're never going to give it the appropriate attention.

~~~
Nagyman
Hrm, your statement seems to suggest upgraded versions are primarily security
updates. Good projects backport security updates (for LTS versions anyway),
and new versions with new features come with their own new security issues.

~~~
Twirrim
> Hrm, your statement seems to suggest upgraded versions are primarily
> security updates

Not in the slightest. But security updates are part of them.

> Good projects backport security updates (for LTS versions anyway), and new
> versions with new features come with their own new security issues.

There's a limit to how far back patches go, even on the good projects. Rails
3.2 hasn't received a patch in over 2 and a half years, and while there is
some desire to backport security fixes, you're dependent on a single
individual having spare time to work on it, where more up-to-date releases
receive patches far faster, and are far easier to test and integrate in to
your existing platform.

I've seen teams, and heard of companies, that are still running services on
top of Rails 2 and the like. Now when they look at upgrading Rails the sheer
number of changes is mind-boggling, and often represents a non-starter.

I'm certainly not arguing to keep up on the bleeding edge, but making routine
upgrading part of your regular workflow is most definitely an important part
of remaining secure.

------
jimnotgym
The regular thread of people piling in to criticise dynamic languages. Instead
perhaps people could suggest a better language/framework that is more
productive than Rails, and has had a long lifespan in a large codebase?

~~~
drb91
> Instead perhaps people could suggest a better language/framework that is
> more productive than Rails, and has had a long lifespan in a large codebase?

I’d suggest most frameworks are better than rails for long lifespans. They
just frontloaded productivity, and it shows. The UI and functionality of
github is largely the same as in 2010.

~~~
int_19h
On the UI, I'm kinda feeling like it's not a bad thing. GitHub UI _works_.
It's neat, most things that need to be done often are easy to find, and
otherwise it gets out of my way nicely. Oh, and it doesn't break my browser
(links, Back button etc). Why change it?

~~~
51
Bitbucket and Jira use React and are far less usable

------
mooreds
Super cool explanation of some of the real world difficulties of upgrading
large rails applications. Really liked the transparency around process and
timeline, as well as the lessons learned section.

------
stevebmark
There are several "we upgraded Rails, it was huge, risky, and took months to
years" blog posts from medium to large companies. I personally take this as a
warning against using Rails. Ruby is one of the most dangerous dynamic
languages to refactor, I don't see how struggling to do it for over a year is
a selling point of the framework. It also feels counter to Rails's mantra of
delivering value fast with little effort, until you need to upgrade, then you
have months of no business value delivery and need to bring in experts to
help.

~~~
nateberkopec
Let me get this straight: you're arguing that a 10-year-old top-100-in-the-
world website taking 4 full-time engineers and having them upgrade their core
framework 2 major version over 18 months is some sort of massive failure, and
that failure would be solved with static types?

Also, you're saying that Github hasn't added any new features in the last 18
months?

~~~
sidlls
Yes, four FTE engineers taking 18 months to upgrade across two major versions
indicates a massive problem, but not necessarily with Rails or Ruby. That's a
cost of $1.5M give or take, just on the engineers, not including the
opportunity cost in new feature development or paying other equally important
tech debt.

~~~
mikemotherwell
$7 billion dollar company, $1.5M cost? $7 BILLION. Your order of magnitudes
are waaaaayyyyy off.

This is the inverse of survivor bias, in that you are retroactively applying
"best practice" at the wrong scale. What gets you from $0 to $7B may hurt you
at $7B. Heck, may hurt you way earlier than that.

However, and YMMV on what problem you want to solve, but saving $1.5M, heck
lets 10X it and call it $15M, saving $15M when worth $7B isn't the problem I'd
personally be concerned with.

------
sigzero
I had no idea that Github even used Rails. The things I learn.

~~~
powersurge360
If you didn't know that then let me share with you this interesting story from
2012. I'm going to repeat it from memory so my details may be a little fuzzy
but I'll include a link which should tell the story more faithfully.

So back in 2012 rails had a default behavior where you could mass assign
values from a POST to a user and there wasn't any scrubbing of that, by
default. Someone realized this was a Bad Idea and issued a pull request that
would have fixed it. Instead of accepting the PR, DHH (I think it was him)
said something along the lines of 'competent programmers would not leave that
setting in place' and rejected the PR.

The exploit discoverer thought about this and tried it against github, which
was known to run on rails and the code worked! From there he was able to
manipulate the permissions on github to get access to the rails repo where he
reopened and accepted his own pull request.

He was promptly banned.

[https://gist.github.com/peternixey/1978249](https://gist.github.com/peternixey/1978249)

~~~
orf
Where is the actual pull request?

~~~
ddeck
Commit:

[https://github.com/rails/rails/commit/b83965785db1eec019edf1...](https://github.com/rails/rails/commit/b83965785db1eec019edf1fc272b1aa393e6dc57)

Github blog post on the incident:

[https://blog.github.com/2012-03-05-responsible-disclosure-
po...](https://blog.github.com/2012-03-05-responsible-disclosure-policy/)

~~~
ydnaclementine
seems like the origin story of strong params

------
dyeje
Wow they've been running 3.2 for this long? That's wild considering the talk
Eileen gave at RailsConf this year made it sound like alot of the Rails 6
scalability stuff was based on GitHub's existing work.

~~~
rst
They've been running 3.2 with local monkeypatches -- which is part of the
reason that upgrading was problematic. (Though certainly not all; over that
span, there were lots of breaking changes to supported and documented APIs.)

------
exabrial
The advice at the end sounds exactly like something I'd say to someone going
from 1.8 to 11 with Java. Great advice for any platform, very interesting to
see the same conclusions from a totally different stack

------
aantix
Using the conditional boot loading, aren’t there structural differences in
ActiveRecord queries/scopes that would run under 3.2 but not 5.2?

Did GH just rewrite those scopes in their respective models and maintain a ton
of if/else blocks for the different versions? And if so, didn’t they run into
issues without the code not being DRY, e.g. someone fixes a 3.2 query, but not
the corresponding 5.2 version?

~~~
barrkel
If you have your test code unified, and have multiple CI pipelines, it should
show up immediately on your build servers.

------
config_yml
What do they mean by off-hours? I imagine on a global site like github, there
are hardly off-hours?

~~~
dyeje
Just because it's a global site doesn't mean the traffic is distributed
uniformly across the day. Certain regions are going to have higher traffic
during business hours. I'd guess their off hours are somewhere around 6pm PST
when North / South America has stopped working, Europe / Africa is asleep, and
India is just waking up.

------
starefossen
As a person working for a large software consultancy in Scandinavia I hate to
see so many using type safety as an excuse for not writing tests. At least a
dynamic language forces you to write tests and frankly it is often easier to
write tests in a dynamic language imho.

------
ksec
I am very much looking forward to Rails 6.0 and see what Github / Shopify will
upstream. Actually Instacart has lot of great gems too which I wish would have
been the default solution in Rails.

------
throwaway427
Given its maturity and settled place in the programming landscape it's always
nice to see that Rails can still evoke irrational disdain in HN comments.

~~~
gameswithgo
Why is irrational to hate languages that are orders of magnitude slower than
are necessary?

~~~
goatlover
It's irrational to hate without mentioning tradeoffs. Sure, if performance is
your only metric, then Ruby is a bad choice. But that's rarely the case,
particularly with the web.

------
dwb
Upgrading early(ish) and often, the very obvious preventative measure against
terrible and failure-prone rewrite or upgrade projects, is one of the first
things that falls by the wayside in the mostly short-termist logic that seems
to dominate modern capitalism. It's absolutely infuriating.

------
ryenus
Anyone knows which ruby runtime GitHub uses? Ruby MRI or JRuby etc.?

------
conroy
> The upgrade started out as kind of a hobby; engineers would work on it when
> they had free time. There was no dedicated team.

I’m not sure why this still surprises me. For a company the size of Github,
there should most certainly be a team responsible for these type of upgrades.

------
nautilus12
Why in the world is github still on rails?

~~~
stephenhuey
Maybe because their codebase still serves their use cases very well?

And perhaps they have little to gain and possibly much to lose if they ditch
it?

You didn't say much in your question, so I don't know if you feel they ought
to rewrite with a popular SPA framework or use something like Elixir Phoenix,
but if their Rails-based solution handily serves 30 million users, why do you
feel so strongly they should move to something else?

~~~
mrdoops
Nothing wrong with Rails, especially if the team knows it well. Time to
develop is the real cost in software most of the time.

If Github wanted to integrate a lot of real-time features, then Elixir +
Phoenix can't be beat. Depending on what they replace, a 10x in performance
and a fraction of the servers needed is a nice win.

~~~
innocentoldguy
I've seen performance boosts closer to 20x when I've helped companies rewrite
their Rails products in Elixir. I've also seen a reduction in server costs. In
all fairness though, simply rewriting the Rails app in Rails, with the benefit
of hindsight, probably would have resulted in a performance gain too.

~~~
ksec
I would bet $10 the scaling problem with Github would properly have more to
deal with Git than Rails itself. Switching to Elixir wouldn't really help.

------
k__
I had the impression GH switched Ruby for Scala years ago.

~~~
marksomnian
That was Twitter.

~~~
k__
Oh, lol.

Thanks :)

------
gameswithgo
How much money in server costs and how much electricity could be saved if
Github didn't use an interpreted language, but something like Go, C#, F#, Java
etc?

~~~
jb3689
Github would not have been the same in any of those. They really took to some
of the Rails concepts - a lot better than most Rails companies - and it shows
in their product (routing, object structures, etc)

~~~
gameswithgo
So your claim is you could not create the same user experience in any language
that is jitted or compiled?

I can't really take that seriously.

~~~
jb3689
Sure, in retrospect you could create the same thing. It's just various text
processing at the end of the day. What I'm claiming is that their choice of
Rails led to certain choices which were really transparent throughout the
product and are still there. It would've grown to be something totally
different on another technology, so I don't think it's fair to just look at
cost and performance. They did many things "the Rails way"

