Add the packaging metadata to build the bigchaindb snap

[come-maiz]

This package will let you publish the latest bigchaindb in the Ubuntu store, and from there reach many users on all the supported Ubuntu versions, and more Linux distributions in progress.

[come-maiz]

Hello!

I recently found about resonate.is, and looking around how to help then I found bigchaindb. I liked very much the ideas that you are implementing. Here at Ubuntu we are working on snapcraft, a new packaging format and delivery mechanism that will let you put your software directly in the store. We no longer need maintainers nor the slow 6 months release cycle, because every snap is confined and secure. You work on your own timeline, with tools that adjust to your process and automate your releases. I thought you might find it useful to get more people trying your releases at this early stage, and expose it to all the Ubuntu users out there. I wrote about a similar experience with IPFS.

To test this in an Ubuntu 16.04 machine:

$ sudo apt install git snapcraft
$ git clone https://github.com/elopio/bigchaindb
$ cd bigchaindb
$ git checkout snapcraft
$ snapcraft
$ sudo snap install *.snap --dangerous --devmode

(dangerous because the snap you build doesn't come signed from the store. Once you push it to the store, your users will just have to run: $ sudo snap install bigchaindb. devmode because of canonical/snapd#2749, which should land soon)

If you have any questions, please let me know. This is just a quick way to get started, but there are many more things that we can discuss, like bundling the backend in the same snap so it is ready to use after install; or continuous delivery to the edge channel every time you land something to the master branch to get early adopters testing your latest stuff.

pura vida.

[codecov-io]

Codecov Report

Merging #1415 into master will increase coverage by 0.05%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1415      +/-   ##
==========================================
+ Coverage   98.37%   98.43%   +0.05%     
==========================================
  Files          56       56              
  Lines        2588     2744     +156     
==========================================
+ Hits         2546     2701     +155     
- Misses         42       43       +1

[ttmc]

Thanks @ElOpio! We got the email when you agreed to the contributor license agreement and I sent you an email explaining the next step.

[sbellem]

Super cool! Thanks a lot @ElOpio!

[sbellem]

note: this resolves #1426

[krish7919]

@ElOpio : Thanks for the PR. Snap looks good, although I must confess that I am very new to this.

@sbellem @ttmc

This snap only deals with Bigchaindb.
There is no way to talk with either RethinkDB or MongoDB configured here, or maybe I do not understand it.

I am not worried much about performance as of now though, as we can run some benchmark tests later.

I was wondering if we do a sudo to install under /snap, can we then run the app as a separate bigchaindb user (with a bigchaindb group) in the system?
Can the snap.yaml create users/groups when installing the package?

Another question is that if we have 2 snaps - one for the backend database and one for bigchaindb - how will one interface with the other?
Should we bundle both of them together or have them as separate snaps?
What if the bigchaindb snap wants to talk with a non-snapped app?

My last question is if we run the backend database within a snap (bundled with bigchaindb or separately), will it have any performance implications or new constraints?

In summary:
If snap is a sandboxed/containerized application, will it change networking inside this sandbox?
Will the backend db will be a snap or not?
Will it be a problem if bigchaindb runs as sudo?

I am OK if we decide to approve this, but this needs to be productized.
We can have a look once we decide to go down this path.

[sbellem]

we should probably match the current version that is in version.py: 0.11.dev

[come-maiz]

I almost missed this comment. We have a new feature to let you use git or a script to get the version:
https://forum.snapcraft.io/t/snapcraft-version-scriptlets/106
I'm testing this new release as we speak, and I expect it to be in the archive and in the store on Monday. Later I can make a PR to see how you like the new feature, so you don't have to keep the version up to date in two places.

[sbellem]

Good! Created an issue for it: #1436

[sbellem]

I suggest we merge this PR as is as a first step and refine it as needed. Thanks again @ElOpio! We had been wanting to do this for a while (see #1426) but never got to it, so your help is much appreciated!

[ttmc]

@sbellem @krish7919 Please don't merge this PR before @ElOpio has copied the special block of CLA text to these comments.

[ttmc]

Could you write a short snap/README.md file (in Markdown format) to explain how to use this?

[come-maiz]

This is to confirm that I agreed to and accepted the BigchainDB Individual
Contributor Agreement at https://www.bigchaindb.com/cla/ and to represent
and warrant that I have authority to do so.

XjvFwtDmFLJfIra7oCfYC5lFkSCXLit1SHgkQHmOYK0Kzd8wuDFX43IlzvPGdvI

[come-maiz]

Hello!

Thanks for the reviews :D

First, I totally agree to land this so you can explore the features in the store, try the things that snaps offer and see if your users would like it. We can improve it as a side task of your main release, focusing on things that will make your delivery faster. I'm very interested in blockchain tools, so I'll stay here to help you.

For some of the comments we have solutions in place and it's just a matter of preference. Some others are work in progress in our side and still need some work. So the feedback you can give us will be very valuable to assign priorities on our backlog. Let me try to reply to all the points.

This snap only has the bigchaindb binary. You can communicate to the backend through the network, as usual. The section plugs: [network, network-bind] is where you declare that your snap needs additional permissions to talk be a network client and server.

You could do many more things here, like bundling your default backend. Or you could even include both backends so your snap has both rethinkdb and mongodb, and then write a start script where the user can select the one to start. That would increase the size of the snap, but we have some nice features in place to only upload and download deltas between releases. The only painful transfer will be the first one. I suggest to start with no backend, because it's the closer to your current install instructions. You just have to make the final decision before putting the snap in the stable channel, but you can play as much as you want in the edge channel. In edge, the snap won't be visible for user searches, so only your early adopters who know about the snap and want to help testing it will be able to install it.

Now, snaps can talk through the network just by declaring the plug. By default, they can't call a binary that's not inside the snap confinement. So if you bundle mongodb, in your scripts you can call mongod. But if mongodb is an independent snap, you can't. There are ways to share things between snaps, but I think in this case it would just make a mess. Still, if you want a communication between processes in different snaps that's not through the network, we can try to find a good solution depending on the use case.

About users, currently we only have two options. Daemons run as root, but still under strict confinement. And normal apps run as the user that executes the command. Fancier user handling is work in progress, being discussed here: https://forum.snapcraft.io/t/snappy-and-users-and-groups/331
As I said, feedback from a real world project is very important for us, so if you could comment there about your preferences and ideal scenario, that would be very nice.

About performance, there is a little hit with everything that has to go through apparmor and seccomp. But well, the only way to avoid it is to remove all the security of the OS. Also, we have to set up some stuff the first time the application is called, that in an uncontained environment you wouldn't have. We try to keep that to a minimum, and we will obviously consider it a bug if it starts to affect the apps.

I think I replied to all your questions, but if I missed something or you have new doubts, just tell me.

pura vida.

[ttmc]

@ElOpio Re: #1415 (review) You've already written most of what could go in a snap/README.md file (in the comments on this pull request, above). It would just be more handy to have those alongside the YAML file for easy reference, rather than hiding out in the comments of some pull request.

[sbellem]

@ElOpio Thanks again! As @ttmc pointed out it would be nice to have a bit of documentation alongside the yaml file, and then we can definitely merge this PR and move forward with future improvements.

I like the idea of packaging the database backends, so we could explore this in the future.

[come-maiz]

Done. @sbellem @ttmc: I added a very simple README. As we start defining more details we can put instructions there to start the daemon, or if it's autostarted, details about how to connect. Also, The instructions are for building in Ubuntu 16.04 with the snapcraft deb. We are currently testing the snapcraft snap, so soon you will be able to build it in any linux distro too. I'll update the README as we release new versions.

Thank you!

[krish7919]

LGTM.

@ElOpio : Thanks for the elaborate clarification.

I am also inclined towards packaging backend DB with BigchainDB snap for now.

[sbellem]

@ElOpio @ttmc @krish7919 not sure why but the build on travis sometimes fail for Python 3.4, and just restarting the build usually works. So I just did this now.