

Why Bundler 1.1 will be much faster - thibaut_barrere
http://patshaughnessy.net/2011/10/14/why-bundler-1-1-will-be-much-faster

======
qrush
For the curious, the endpoint itself is fast mostly because it's heavily
implemented with Redis. Are you interested in a writeup of how that works? If
so, leave a comment and I'll consider spinning that up.

~~~
moeffju
Yes, I would be interested in that.

Also, why don't you do the recursive lookup internally instead of having
bundler do several calls? It seems like performance would be strictly better
because it saves network round trips.

~~~
qrush
This was pretty much just the first iteration, I'm fine with experimenting
going deeper _cue inception soundclip_.

I could definitely see future iterations going down more levels and caching
the result as well. Right now each dependency lookup is not cached, so that
might cause issues once 1.1 is actually released :)

------
sunkencity
tldr: A classic brute force ruby solution by downloading the whole list of
gems and parsing it has (finally) been replaced by an intelligent way of
fetching just what's needed, on the new rubygems api that bundler and rubygems
have developed together.

awesome!

~~~
tptacek
Which classic brute force Ruby solution was _the right call_ , seeing as how
Bundler (a) improves ad-hoc packaging dramatically and (b) has gained
widespread adoption.

They did the simplest thing that could possibly work, and ended up delivering
65% solution; now they're iterating. The hard work was getting people to adopt
Bundler in the first place.

~~~
martingordon
This.

I'd rather have waited an extra 25 seconds on every bundle install over the
past year than being forced to manage gem dependencies manually for another
year.

------
sjwright
git (and hg, etc) has already solved the problem of delivering reliable deltas
of large, occasionally modified text files. Why not just replace the gem index
download with that, and save the need for server CPU cycles?

~~~
qrush
I'd love to have git be a part of the solution here, really, any other
solution would be awesome. Hit me up in #rubygems on IRC or email me (nick
[at] quaran.to) and we can talk about it :)

------
icepick
Does ruby's Marshal have the same problems that python's pickle have? Could
you construct a valid Marshal bitsting that when loaded would run malicious
code? Is this exposing folks to a MITM attack on rubygems?

~~~
moeffju
What about the old Marshal problems? I just played with Marshal on my rails
console again and it seems it still encapsulates all sorts of implementation
details. I haven't tried whether it's now compatible across Ruby versions, but
I recall the Marshal format changed a few times, introducing incompatibility.

tl;dr: Why, oh why, Marshal, and not, say, JSON?

~~~
evanphx
Not sure what you mean by Marshal having implementation details in the
bitstream, it doesn't. Marshal has been reimplemented in many different
implementations just fine.

As for why not JSON, because there is no JSON parser as part of the standard
library and rubygems needs to be extremely careful about what dependencies it
has.

------
Vitaly
I dont see why not just download all and _cache_ it for a day

~~~
qrush
The RubyGems index needs a ton of work. It's definitely not ideal right now,
this was the easiest thing we could do to get Bundler happier and speedier.

I would love to see a more apt-get like system where you cache things locally,
but that has its own implications/difficulties as well.

Basically, it's hard to keep the time down from "gem push" to "gem install" if
you go towards more distributed/delayed indexing systems...but I'm willing to
compromise if it's way better.

------
spicyj
Is there a reason rubygems.org can't return all of the dependencies, instead
of one layer at a time?

~~~
pat_shaughnessy
That's a great question. I suppose they split it up into separate requests
since downloading all of the dependencies at the same time might end up being
slow and returning a lot of data, especially for something like Rails with a
lot of dependencies. Or maybe separate requests was fast enough and a good
trade off between speed/data and number of requests.

~~~
moeffju
But bundler cannot determine which versions it needs until it has all
dependencies, right? And in effect, all it does is recursively call for one
"layer" of dependencies after the other, so it seems like the backend might
just as well recursively query itself and return all dependencies at once.

------
mey
Will this release address the issue of developing on windows and deploying on
linux?

See: [http://stackoverflow.com/questions/3642085/make-bundler-
use-...](http://stackoverflow.com/questions/3642085/make-bundler-use-
different-gems-for-different-platforms)

