
JS Git: working git clones for offline development in the browser - bpierre
http://www.kickstarter.com/projects/creationix/js-git
======
vjeux
"Today there are fewer and fewer kids getting into programming because
consumer devices are locked down and don't come with a way to program them."

I honestly doubt this is the case, I would expect more and more kids are
getting into programming because it has become so widespread and easy.

~~~
chjj
I agree with that statement in a way.

Linus Torvalds wrote a kernel because he bought a PC: hardware which allowed
him to do anything he wanted.

Today, smartphones and mobile devices are much more popular among the youth.
Could you imagine if Torvalds had been born in 1993? By this point, I'm sure
he would have made something much less spectacular than Linux. Maybe he would
make some pretty cool Facebook or smartphone games, and maybe a few cool web
apps, but the freedom to tinker with hardware directly had been slowly
slipping away over the past ~10 years. Since that avenue was becoming less
available, there's a chance it wouldn't have grabbed his attention in the
first place.

We still have laptops/desktops which aren't locked down. Hopefully that's what
the young/future version of Torvalds would have gotten interested in. But how
long until you buy a laptop only to find out that it doesn't grant you root
access?

~~~
mattstreet
To be fair, it's easier these days to afford both a smartphone AND a piece of
much more hackable hardware. At least as far as prices go.

One of the main real reasons for Linus's success (other than say his talent)
is that there wasn't a good alternative to Linux so even version 0.2 which
could barely do anything by today's standards got interest and community
involvement.

------
drrotmos
I'm a bit intrigued. As the main developer of a Git library[1] myself (in pure
PHP, not depending on the git binary or esoteric extensions), I have no doubt
that the actual parsing of the git objects and packfiles will be very much
possible to achieve in a browser environment.

[1]: <https://github.com/e-butik/Gittern/>

However, there are technical considerations. For example, pack files can get
pretty large (indeed version 2 packfiles supports files > 4 GB), and I doubt
browsers are optimized for random access to large files (so you wouldn't have
to load the entire file in memory). Another potential problem would be pushing
and pulling. The dumb HTTP transport is obviously possible to use (but rarely
supported anymore), and the smart HTTP transport should be able to get
working, but for implementing the git protocol, or git+ssh you'd need regular
sockets, which AFAIK browsers don't provide.

I do however wish the author the best of luck with this project, and since I
see that he's here, I will leave with links to a few resources I found useful:

* <http://git-scm.com/book/en/Git-Internals>

* [http://opensource.apple.com/source/Git/Git-26/src/git-htmldo...](http://opensource.apple.com/source/Git/Git-26/src/git-htmldocs/technical/index-format.txt)

* [http://www.opensource.apple.com/source/Git/Git-17/src/git-ht...](http://www.opensource.apple.com/source/Git/Git-17/src/git-htmldocs/technical/pack-format.txt)

* <http://git.rsbx.net/Documents/Git_Data_Formats.txt>

~~~
creationix
Thanks for the links. You are correct that I probably won't be able to support
all git repos because of memory constraints and unless I get special APIs from
the browser I won't be able to do git protocol or git over ssh. I do feel
pretty confident that I can implement dumb and smart HTTP though.

Even if this library can't be used for everything, it will allow a whole lot
more use cases that are currently possible.

~~~
maksimlin
I'd say smart http is more than enough, especially since Github for example is
pushing it as the preferred protocol for most users:
[https://github.com/blog/1104-credential-caching-for-wrist-
fr...](https://github.com/blog/1104-credential-caching-for-wrist-friendly-git-
usage)

The main issue is that Github do not have CORS support for http access to
repos (though they do support CORS for access to repos via their proprietary
API) and I don't think the other big public hosters like Bitbucket or Google
Code support CORS either.

On the subject of big pack files, sure you'd want to stream them to storage
instead of trying to store in memory (for instance JGit has this as an option)
but I agree Tim that to start with, even supporting small-medium size repos
would be a huge win.

------
artagnon
A pure Javascript implementation of git?

0\. Why is this a good idea? Have all the desktops and laptops died, and are
people forced to use Mozilla OS for development? I'd vastly prefer native
applications on iPad, Windows RT, Android and other such platforms (if I
suddenly went crazy and decided to do development on them). And a PNaCl
application on Chromebook, if that's what you're specifically targeting.

1\. Why won't it die like git.js did? [1] Why aren't you working on extending
git.js?

2\. Where can I see a technical roadmap? You're planning to deflate, applying
deltas, huge packfiles with 4G objects in Javascript? What performance
guarantees can you make?

3\. Git is not a dumb book-keeper where you just checkout, make changes, and
checkin. How are you going to replicate what makes git so awesome: `rebase
-i`, `filter-branch`, `rerere`, pickaxe, and so many other features?

[1]: <https://github.com/danlucraft/git.js>

~~~
creationix
I want this, I know many people who want this. That's good enough in my book
to create it. If enough people believe in the idea enough to fund me, then
it's a good idea.

I know this is a hard project, that's why I haven't done it yet in my free
time. I realize that this will require my full-time attention. But click
through the links of my other projects in the kickstarter. I'm not new to hard
problems and low-level coding in JavaScript.

I don't have a detailed roadmap and I can't make performance guarantees
because I don't know yet. But I do know that I'll do whatever is possible
within the constraints of a web browser. Again, I'm not new to these
constraints. I believe the web is finally mature enough to pull something like
this off.

------
nwienert
I don't think this page explains well the use cases for this. It alludes to
kids programming, but then switches over to explaining how things are
sandboxed, why the developer is experienced, etc.

My question is what is the net benefit of having this? What does it enable us
to do? Why should I back it?

~~~
creationix
The main thing it allows is developing offline en environments like
chromebooks (like the new Pixel). I worked on Cloud9IDE for a year and it was
a great experience as long as you were online with a fast connection. With
this library, html5 apps will finally be able to do the full developer
lifecycle. They can clone from github to the browser's local file storage when
online, work offline using an editor like ace or codemirror, and then when
they are online again, they can push their changes back to github. I'll
implement branching, merging, diffing, and as many other awesome common tasks
from git as possible.

I've wanted this for years and I'm super excited to finally be able to create
it when this kickstarter finishes!

~~~
city41
When working in this way, how do you run the code? I imagine anything
involving a backend would be tough, but would it be possible to run purely
client side code when offline?

~~~
creationix
Not all web apps require constant server interaction to develop. Also you
could probably mock a lot of the server interaction as well.

~~~
tmzt
Possibly even instant offline mocking, somewhat like meteor but without the
need for a server.

------
goldfeld
I decided I like this guy a lot by the projects he has under his belt. Always
nice to see a prolific open source contributor getting financial support out
in the open like this. And this project seems like something I could use in
the future.

~~~
zellyn
Agreed. I'm in for $25. :-)

------
ori_b
I'm confused about what problem this is trying to solve. Can someone enlighten
me?

~~~
MatthewPhillips
Web apps do not have read/write access to the file system. So let's say you
wanted to write a text editor web app. You can't modify a user's local files
so you are forced to store their data on your servers. You are forced to
create some UI for integration with GitHub, Bitbucket, etc. You are forced to
add a lot of stuff Sublime and TextMate don't have to worry about... you are
forced to build an IDE.

This would improve a lot of that kind of thing.

~~~
ldng
IMHO, that's definitely something Github should have been working on for its
paying customers.

------
nicholassmith
I've just read that a few times and I'm still none the wiser.

You want a working git implementation in the browser, okay, cool. Why? What
for? I don't want to just disparage, I'm genuinely in favour of anything that
gets good tools to more people, but I'm completely at a loss at the why from
reading the Kickstarter page.

~~~
creationix
So I can develop offline using nothing more than a browser. I clone a repo
from github to my browser when I'm online. Then I'm offline in a bus or on a
plane, but I can still work because browsers have file storage now. When I'm
done working, I can go online and push my changes back to github.

And all this will work anywhere that has a browser, ChromeBook, Surface RT,
iPad, etc..

Obviously, it's mostly of use to web developers since JS is the only runtime,
but I also have other later projects attacking that problem. But for now, JS
only is a pretty big market.

~~~
nicholassmith
Hey, thanks for the reply!

Couple of things, repos can get pretty big, how are you planning on storing
things? Using the HTML5 storage? Kicking it down the drive? If it's with HTML5
storage after that guy found a work around allowing sites to completely burn
through a drive browser devs might start putting a cap onto the limit of
storage that you can use.

I don't know, I'm a big fan of getting tools to everyone, but I'm still
wondering if this _sounds_ like a great idea, without actually being a great
idea.

EDIT: that last line might be a little overly harsh looking back on it, it's
not that I don't think it's a good idea (definitely is!) but I'm wondering if
it's a _great_ idea that's worthy of a Kickstarter.

~~~
creationix
Like I said, my main use case is mine. I want to develop on my new ChromeBook
Pixel (if it ever gets here) without being forced to use an Ubuntu chroot.
Packaged apps have much larger size quotas and most my code repos are small
anyway.

As far as the education aspect, I have future plans to create a new
programming language aimed at learning to program that compiles to javascript.
Combined with a proper in-browser IDE, this would allow any kid to learn to
program on any machine that has a browser.

It obviously won't replace native dev tools, but it will make web-based stuff
a whole lot more powerful and that's something I and many others feel is worth
paying for.

~~~
nicholassmith
You know what, more power to you. It doesn't fit a use-case for me, but damn
do I love seeing tools made more accessible. Although I'm really interested in
hearing about how you're going to port the incredibly 'Linus' C code into
Javascript, there's a huge amount going on and it's often not, well, that
awesome to read.

------
tjholowaychuk
Good luck man! Sounds like a fun project

------
AaronO
Nice project. I've been working on Dulwich (a pure python implementation of
the Git protocol) as well as a higher level abstraction. I would definitely be
interested in contributing to JS Git.

------
shurcooL
While I'm not interested in this at the moment (I choose to work with native
tools), I can see the value for many other ppl so I've backed your project on
Kickstarter and you on Gittip.

~~~
creationix
Thanks!

------
meryn
Tim, do you know about <https://github.com/danlucraft/git.js> ?

------
rektide
File API + node.js filesystem shim + node-git. It's been a weekend project
somewhere on my todo list for a year, and it'd work just fine as a impl, and
requires naught but working some glue into the cracks between these already
made pieces.

That said, support away, would love to see Tim get this initiative kickstarted
& see what solid work he kicks out.

~~~
MatthewPhillips
node-git just calls the git binary, which you won't have in the browser. This
project is definitely needed.

~~~
rektide
Hi MP-

<https://github.com/christkv/node-git> is the node-git I had in mind: it's a
native (js native) impl modeled strongly after the Grit library used heavily
by a little company called Github, and I believe it's in pretty OK state
although my first hand experience hasn't made it past the touring stages.

Sorry for the confusion! I knew Tim had a node-git as well, but I didn't know
it relied on the binary or other native type stuff (such as a a library
binding), so I didn't think the naming collision I let happen would cause any
confusion.

Is there anything else I'm missing in my equation?

~~~
MatthewPhillips
Thanks, I wasn't aware of this library. I might take a stab at it this weekend
myself, if I have some free time ;)

------
jackmaney
Now that GitHub has the ability to move files within a repo
([https://github.com/blog/1436-moving-and-renaming-files-on-
gi...](https://github.com/blog/1436-moving-and-renaming-files-on-github)) what
will JS Git be able to do that can't be done by creating, editing, and moving
files around in a GitHub repo?

~~~
67726e
It allows you to work offline.

Additionally, not everyone uses GitHub.

------
MatthewPhillips
Will Git source compile with clang? If so emscripten is a possibility.

~~~
TheMakeA
In the past I investigated compiling libgit2 with emscripten. Didn't spend a
lot of time on it but I wonder if that could work.

~~~
creationix
I will explore this as one of the possibilities. I'm not dead set of
reimplementing everything myself. If using an existing JS library and
finishing it or compiling an existing C library using emscripten is the best,
then I'll do that..

The point is I'll spend the proper time to evaluate all the options and fill
in any gaps needed to have a solid solution.

------
espadrine
How will it work on non-Filesystem API browsers (that is, all of them except
Google Chrome and… BB10)?

Are there plans?

<http://caniuse.com/#feat=filesystem>

~~~
MatthewPhillips
IndexedDB can store blobs. There's even a FileSystem API polyfill that uses
IndexedDB: [http://ericbidelman.tumblr.com/post/21649963613/idb-
filesyst...](http://ericbidelman.tumblr.com/post/21649963613/idb-filesystem-
js-bringing-the-html5-filesystem-api)

~~~
creationix
Thanks Matthew, that's the plan. Also since the target audience is developers,
I don't have to support ancient browsers. This isn't a news site for the
general public, it's a developer library.

~~~
MatthewPhillips
I wrote a library that brings Node's fs module to the browser. It
transparently uses the File System API in Chrome and IndexedDB in Firefox (and
presumable IE10, haven't tested it there). Feel free to use/contribute ;)

<https://github.com/matthewp/fs>

------
tlrobinson
I wonder how hard it would be to just compile the real git implementation with
Emscripten? Isn't it mostly C?

