suggestion: use libgit2 under the hoods

[sbeaupre]

Your ideas behind isomorphic-git are very good and very useful in many projects. I think it might become very discouraging (not to say a lot of work) when you have to re-implement all git functionality itself from scratch.

Have you thought about using emscripten in combination with libgit2 and then write isomorphic-git as a shield around it? You would be able to focus on pure functionality without getting into the details of git.

Here you can find an example of somebody who successfully translated libgit2 into a js version and using it in a web git browser:
https://github.com/fintechneo/angular-git-filebrowser
and how he built the js version with emscripten:
https://github.com/fintechneo/libgit2/blob/master/emscripten_hacks/README.md

This would be an amazing boost for your project.

regards,

Sven

[TomasHubelbauer]

For me, a large part of the appeal of this project is it being written in (good!) JavaScript. First, writing it from scratch means Will is free to choose a better API, a more JavaScript-y and less CLI-y one, so it actually feels nice to use, but also, fixing bugs and contributing (either directly or through plugins) to IsoGit is now very straightforward, no need to go and figure out contributing to libgit2 (which is cool, but let's be real, JS is a much lower barrier to entry than C), one can just check the readable JS code out here.

So all in all I think writing Git from scratch in JS is a deliberate and very smart decision. It will allow for Git to be used in kinds of deployments it couldn't have been used in before (like P2P experiments in the browser), but in a way which makes it easy to inspect and extend for most devs.

[jcubic]

Agree, also debugging code that is compiled C in browser is not fun experience. Debugging IsoGit where you have source maps is much better, you see exact code that developer used to produce the library.

[sbeaupre]

@TomasHubelbauer that is the same reason why I do not want libgit2 as such in the first place 😉
libgit2 just has a huge community (including commercial entities who use it in their products), chances that new features need to be implemented any time soon for IsoGit is quite small. So wrapping it gives us the nice IsoGit API just as you mention and at the same time a bootstrap towards all thoroughly tested git functionlity. libgit2 is amongst many others used for nodegit also...

@jcubic totally agree, could be a big disadvantage. On the other hand, the library as such is tested thoroughly and the main issues could be because of compilation with emscripten. Emscripten also became very popular and has also quite a large community behind it, which means that it they can help with issues on this. Btw, correct and validated compilation with emscripten is on the roadmap of libgit2 also: libgit2/libgit2#4376 The example I gave was just a first successful case and it probably won't be long before they have an official one and put it on their homepage.

Just giving some arguments which could help the IsoGit project a boost and more popular. Of course IsoGit is open source and if you guys want to try to write another git implementation, I would say go for it... I'll come by and check now and then what the status us, but I'll probably be testing further libgit.js first since it has all the git features already out of the box (even if it means an ugly API 😞 )

[billiegoose]

@sbeaupre I chose this route because I felt it had a greater chance of succeeding. Emscripten has been around for years. I searched for projects that ported git or libgit2 using emscripten before I began, and it didn't look promising. Whereas js-git had successfully demonstrated the feasibility of this approach.

I see that @petersalomonsen got git clone successfully in September of last year. Well, I started the project in July and had a working git clone in August. So that example didn't exist back when I started. ¯_(ツ)_/¯

It's quite possible that a working libgit2 port would kick isomorphic-git's ass. But I also know people like @CMCDragonkai have been trying for quite a while to achieve that and... well, you have to debug someone else's C code, compiled to a platform with third party tools.

So basically that's why. It's another road, not taken by this project. Other people are taking that road. Their road might be shorter. Or it might have lots of potholes! On our road, we at least can see where we're going, even though we need to pave the road ourselves the whole way.

Really flattered that you like the API by the way! I'd be happy to support you if you wanted to build a wrapper around libgit2 that has the same API as isomorphic-git. I could do something like abstract-pull-git-repo did and make the API and tests their own spec.

[CMCDragonkai]

Yep. Compilation with emscripten works. But there are lots of edgecases that don't. Anything that touches mmap syscall fails. The emscripten fs is a spaghetti and too deeply embedded (and a requirement for me to be able to interop with other js code that uses fs). Ultimately I abandoned it. Having multiple implementations is a good thing as well. And this code has a alternative license.

[petersalomonsen]

You can test cloning the isomorphic-git repository using libgit 2 in the browser from here.

https://fintechneo.github.io/libgit2/emscripten_hacks/index.html

You have to open your webconsole and switch to the worker context where you can type in the following commands:

Module.jsgithost = 'https://cors-buster-tbgktfqyku.now.sh';
jsgitinit();
jsgitclone('https://cors-buster-tbgktfqyku.now.sh/github.com/isomorphic-git/isomorphic-git.git','test');
FS.readdir('test');

The last command should give you the contents of the checked out dir.

As for me I'm very happy using libgit2. Once the emscripten specifics were resolved it has been very stable and performant, and you have access to all the development made to libgit2 over several years now.

Screenshot on how it looks when cloning the isomorphic-git repo:

[sbeaupre]

Thnx all for the constructive feedback. I completely understand the choices that were made and the road you are paving.

[billiegoose]

@petersalomonsen That's friggin awesome! Are you releasing this as its own project or trying to get the libgit2 project to officially support WASM? That would be super awesome.

[billiegoose]

And just to be clear to anyone reading this, my adoration of @petersalomonsen 's accomplishment does not mean I will be switching isomorphic-git to a libgit2 backend anytime soon. As has been pointed out, each approach has its own advantages, and multiple implementations is a GOOD thing. Particularly multiple implementations of git since that hopefully will become a fundamental piece of infrastructure of the Web 3.0 movement in 2018, and having multiple implementations will provide greater redundancy and system reliability for the web as a whole.

[billiegoose]

Interesting observation - your libgit2.wasm is 314KB and my bundle.umd.min.js is 327KB.

Not that that's a meaningful comparison by any means - you've got merge implemented it looks like! - while mine has tree-shaking so builds smaller when bundled with the rest of the app - and yours uses the emscripten file system - and mine uses the BrowserFS file system. But yours uses a WebWorker, while I'm still working on official WebWorker support. So they're not exactly interchangeable. :-/

Oh this is so cool! We're gonna rewrite the Internet using git together!

[petersalomonsen]

@wmhilton I've made a fork and a pull request so I hope that they want to support it officially in libgit2 eventually. Agree with you that multiple implementations is a good thing. Using git is great for syncing and merging data in web apps that need to be available offline, so many variations and implementations on this subject is just great!

As for me I needed this for one of my startup projects ( https://www.orgbrain.ai/ ) where we need offline support (using PWA with serviceworkers and git repo for storing data). Most of my time goes into that project, so the functions exposed from libgit2 have been limited to the use cases there. However it would be great to have an API like you've made for isomorphic-git, and what I hope by sharing this is to contribute to get more people involved, and use git for storing data in offline-capable web apps.

I really share your vision about rewriting the internet using git. I think we're onto something here that could be a new chapter in what web apps can do!

[billiegoose]

@petersalomonsen Oh that's fantastic! You hit all the right keywords - I think we have a similar vision, although you left out P2P. I find it fascinating that all of your work on libgit2.js has been for orgbrain.ai. I think a common misconception (e.g. from the Hacker News pundits etc) is that we do these things just because we can, not realizing that "make git work in the browser" is simply a step along the way towards other goals. (For me, it's become quite a large step that's consumed me and will likely continue to consume me for a while.) Ultimately it's about the things you could DO by using git repos for syncing data between offline-capable PWAs. Stay in touch! I like the way you're thinking. 😁

[alexciesielski]

Sorry for commenting on an already closed discussion but I was wondering if there was any progress done in regards to getting isomorphic-git running with wasm-git under the hood?

[iliakan]

Citing @CMCDragonkai 5 years ago and hoping that things did change ;)

Yep. Compilation with emscripten works. But there are lots of edgecases that don't. Anything that touches mmap syscall fails.

Does that mean that some libgit2 features won't work?

E.g., take wasm-git approach by @petersalomonsen, is it inherently flawed?

[jcubic]

@iliakan this is a completely different library you can't expect that feature from one, not related library will work for someone completely different. The only similarity is that they both have git in the name.

But if you want to add a feature that is in libgit2 that is missing in isomorphic-git you're welcome to contribute. But let's discuss it first before you start coding.

[CMCDragonkai]

Citing @CMCDragonkai 5 years ago and hoping that things did change ;)

Yep. Compilation with emscripten works. But there are lots of edgecases that don't. Anything that touches mmap syscall fails.

Does that mean that some libgit2 features won't work?

E.g., take wasm-git approach by @petersalomonsen, is it inherently flawed?

Probably yes.

[petersalomonsen]

Hi :)

mmap is not an issue anymore. I did several Pull Requests to Emscripten some years ago to fix this:

• fix: handle mmap offset emscripten-core/emscripten#10095
• fix(mmap): handle over-allocated files emscripten-core/emscripten#10526
• fix(mmap): support read-only emscripten-core/emscripten#10782

And also this for using NODEFS: emscripten-core/emscripten#10669

As you can see these are all merged, and also wasm-git is using them which you can read more about here: https://github.com/petersalomonsen/wasm-git/blob/master/README.md#building

[strogonoff]

Having experience with isomorphic-git, and now a little bit of experience with wasm-git, I’d say wasm-git is much more limited at this time (at least until CLI support?) for someone who‘s coming from JS/TS background and wants to be productive with some project quickly.

For example, let’s say I want to list branches and tags. I know how to do it using IsoGit’s straightforward API. Now, if I run lg.callWithOutput(['tag', '--list']) it seems to work, but ['branch'] will give “Command not found”. I assume libgit2 makes it possible through some lower level API[0], but even reading and understanding its reference is not an easy endeavour compared to IsoGit‘s—not to mention jumping into the source in order to understand or patch it!

[0] Which may or may not be made available through wasm-git—there’s no way to tell, since no relevant documentation seems to exist.