Skip to content

Commit

Permalink
Update "Developers" section re recent build changes rockstor#319
Browse files Browse the repository at this point in the history 
Due to a change/update in authentication mechanism re pypi we have had
to alter our build procedure a little. Reflect these changes in our
developer contributor section.

Includes improvements re:
- Brevity.
- Readability.
- Consistency.
- Update to V4 "Built on openSUSE" base os.
- Preference for using a rockstor-installer derived instance.
- Removed dead and distracting YouTube links.
- Removed redundant and completely neglected AUTHORS link to
rockstor-core.
  • Loading branch information
@phillxnet
phillxnet committed Nov 20, 2021
1 parent f4c8437 commit fe77ae5
Showing 1 changed file with 88 additions and 81 deletions.
169 changes: 88 additions & 81 deletions contribute/contribute.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -87,38 +87,27 @@ Since we rely on github services, you need to create a profile on `github.com
Go to `rockstor-core repo <https://github.com/rockstor/rockstor-core>`_ and
click on the "Fork" button. This will fork the repository into your profile
which serves as your private git remote called origin. The next few git steps
are demonstrated on a Linux terminal. They work the same on a Mac too. Your IDE
may have ways to completely avoid the terminal, but using the terminal makes
you look cool:
are demonstrated on a Linux terminal. They work the same on a Mac too.

See YouTube `Trinity in the UNIX shell..... <https://www.youtube.com/watch?v=51lGCTgqE_w>`_.
Clone your fork of rockstor-core repo onto your local machine.
You can do this in any directory; the following commands assume a ~/dev directory.::

Your development machine can be a Linux system or a Mac. In my case, it's a
Lenovo laptop running Ubuntu. There are other contributors using Mac, so this
process works the same in both cases.

Clone your fork of rockstor-core repo onto your local development
environment. You can do this in any directory, but chances are you have a
project directory appropriate for this. Let's assume you have a projects
directory in your home(~/projects) for the repo. ::

[you@your_laptop ~/projects]# git clone git@github.com:your_github_username/rockstor-core.git
[you@laptop ~/dev]# git clone git@github.com:your_github_username/rockstor-core.git

The above command creates a local rockstor-core git repo in a new directory by
the same name(rockstor-core). Change into it::
the same name: "rockstor-core". Change into this new directory/repo clone::

[you@your_laptop ~/projects]# cd rockstor-core
[you@laptop ~/dev]# cd rockstor-core

Configure this new git repo with your name and email address. This is required
to accurately record collaboration::

[you@your_laptop rockstor-core]# git config user.name "Firstname Lastname"
[you@your_laptop rockstor-core]# git config user.email your_email_address
[you@laptop rockstor-core]# git config user.name "Firstname Lastname"
[you@laptop rockstor-core]# git config user.email your_email_address

Add a remote called upstream to periodically rebase your local repository with
changes in the upstream made by other contributors::
Add a remote called **upstream** to periodically rebase your local repository with upstream changes by other contributors::

[you@your_laptop rockstor-core]# git remote add upstream https://github.com/rockstor/rockstor-core.git
[you@laptop rockstor-core]# git remote add upstream https://github.com/rockstor/rockstor-core.git

.. _makechanges:

Expand All @@ -130,12 +119,12 @@ We'll assume you have identified an issue(eg: #1234) from the `github issue trac
with the latest code by rebasing your local repo's master branch with the
upstream::

[you@your_laptop rockstor-core]# git checkout master
[you@your_laptop rockstor-core]# git pull --rebase upstream master
[you@laptop rockstor-core]# git checkout master
[you@laptop rockstor-core]# git pull --rebase upstream master

Checkout a new/separate branch for your issue. For example::

[you@your_laptop rockstor-core]# git checkout -b issue#1234_brief_label
[you@laptop rockstor-core]# git checkout -b 1234_issue_title_text

You can then start making changes in this branch.

Expand All @@ -150,7 +139,7 @@ describe the change in more detail in plain text format where each line is no
more than 80 characters. This description should be in present tense. Below is
a fictional example::

foobar functionality for rockstor
foobar functionality for rockstor #1234

Add a new file to implement the algorithm called recursive transaction
launcher to generate transactional foobars recursively during runtime
Expand All @@ -165,86 +154,104 @@ a fictional example::
# new file: foobar.py
#

If you'd like credit for your patch or if you are a frequent contributor, you
should add your name to the `rockstor-core AUTHORS
<https://github.com/rockstor/rockstor-core/blob/master/AUTHORS>`_ file.

Build VM
--------

You need a Virtual Machine (VM) to build and test your changes. An easy
solution is to create a Rockstor VM using either Oracle's `VirtualBox
<https://www.virtualbox.org/>`_:

See YouTube `Rockstor 3 0 installation demo <https://www.youtube.com/watch?v=00k_RwwC5Ms>`_.

or if you are using a Linux desktop then `Virtual Machine Manager (VMM) <https://virt-manager.org>`_ is a more native option.
You need a Virtual Machine (VM) to build and test your changes.
An easy solution is to create a Rockstor VM using either Oracle's
`VirtualBox <https://www.virtualbox.org/>`_
or if you are using a Linux desktop then the
`Virtual Machine Manager (VMM) <https://virt-manager.org>`_
is a more native option.
VMM is used in our :ref:`kvmsetup` howto.
It need not be a VM, but using a physical machine just for this purpose could
be overkill.


Note that when you first create the build VM, Rockstor rpm package will already
be installed. The package files are located in /opt/rockstor. Further more, the
Rockstor service should be running. We don't want that as it interferes with
our development activity. Further down in this document, there is a buildout
step. When that is run for the first time, the rpm package and it's effects are
removed. Please note that this will destroy the existing Rockstor install and
it's associated database details / settings.
A 'real' machine is another option and may be required in some scenarios,
i.e. where a hardware compatibility feature is being developed.

It is suggested that a fresh Rockstor install be used for the target development machine.
I.e. the resulting install from a
`rockstor-installer <https://github.com/rockstor/rockstor-installer>`_
such as are available on our `Downloads page <https://rockstor.com/dls.html>`
Only v4 "Built on openSUSE" and newer bases are considered.
Our v3 and older (CentOS based) are now legacy and no longer developed.
A prior Rockstor v4+ instance is suggested as it already contains many of the dependencies.
And little else.
But note that this prior install will be destroyed, hence the 'fresh' suggestion.

Alternatively an upstream (openSUSE) JeOS instance is also an option:
but only if:

- It's root filesystem is btrfs.
- It uses NetworkManager not wicked for it's network configuration.
- Shellinabox is installed and enabled/running under systemd.
- Apparmour is disabled if installed: "systemctl disable apparmor".

Remove the Existing RPM Rockstor install
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The prior existing rpm install must be removed as it will otherwise interfere.
The following does this, updates the os, and installs the dev dependencies.
Around 30 packages will already be installed and a similar number will be added:
if you are using the suggested rockstor-installer derived install.
This change will be around 60 MB download and 250 MB installed.::

zypper --non-interactive remove rockstor
rm -rf /opt/rockstor # remove dangling rockstor rpm related files.
zypper refresh
zypper up --no-recommends
zypper --non-interactive install avahi samba docker nut nginx nfs-kernel-server \
at gcc python-devel gcc-c++ postgresql10-server libblkid-devel \
dnf-yum dnf-plugins-core python2-distro python3-distro firewalld python2-setuptools \
python2-requests python2-chardet python-xml cryptsetup which dnf-yum dnf-plugins-core \
python3-python-dateutil python2-six python2-psycopg2 python2-pyzmq python2-pyzmq-devel \
sssd sssd-tools sssd-ad sssd-ldap sssd-dbus python-dbus-python dmraid

Helpful terms
-------------

In the following sections we use some terms in the commands; this is a short
explanation of these terms:-

1. **laptop**: This is your laptop or desktop computer.
In the following sections we make these assumptions:

2. **rockstor-core**: This is a directory on your laptop containing your local
rockstor-core repo. In my case, it's ~/Learnix/rockstor-core
1. **laptop**: This is your local laptop or desktop computer.

2. **build_vm**: IP address of your build VM. In my case, I use Virtualbox
with host-only adapter and get an ip in 192.168.56.101-254 range.
2. **~/dev/rockstor-core**: You local (laptop/desktop) git clone & dev branch.

3. **build_dir**: The directory on the build VM where you like to copy the code
to
and build. In my case, I picked /opt/build/.
2. **build_vm**: The Rockstor target machine's (VM or real) IP address.

Build VM initial setup
----------------------

Transfer the code from your laptop to the build VM ::

[you@laptop ]# rsync -avz --exclude=.git /path/to/rockstor-core/ root@build_vm:/path/to/build_dir/
[you@laptop ]# rsync -avz --exclude=.git ~/dev/rockstor-core/ root@build_vm:/opt/rockstor-dev/

If you are building for the first time or like a clean build, execute the
following command in your deploy directory on the VM ::
And via your ssh or console on the target machine::

[root@build_vm ]# python /path/to/build_dir/bootstrap.py -c /path/to/build_dir/buildout.cfg
[root@build_vm ]# zypper --non-interactive install python2-pip
[root@build_vm ]# pip2 install zc.buildout
[root@build_vm ]# cd /opt/rockstor-dev/
[root@build_vm ]# buildout bootstrap

The next step is to build Rockstor with your new changes. This takes a long
time for a clean build, but subsequent builds finish quickly ::

[root@build_vm ]# /path/to/build_dir/bin/buildout -N -c /path/to/build_dir/buildout.cfg

This comment has been minimized.

Sign in to view

Copy link
@FroggyFlox

FroggyFlox Nov 20, 2021

@phillxnet , unless I'm misreading the new organization you laid out, I think we still need that step here; isn't it?

This comment has been minimized.

Sign in to view

Copy link
@phillxnet

phillxnet Nov 21, 2021

Author Owner

@FroggyFlox No worries, it was a last minute edit/save point and I'm proofing it all today hopefully against our existing buildbot build system; via a by-hand walk through. Pr soon hopefully as almost there now.

Thanks for the heads up.

Once the buildout step above succeeds, rockstor services are automatically
started and managed by systemd. You should now be able to login to the WebUI
and verify your changes.
The buildout process, towards the end, enables and starts the rockstor systemd services.
You should now be able to login to the Rockstor Web-UI and verify your changes.
It is also very important to ensure that your code changes service a reboot.
Sometimes, especially when db changes are made, this can be an issue.
So be sure to check that the resulting build behaves as expected over several reboots.

Change -> Test cycle
--------------------

Changes fall into two categories. (1) Backend changes involving python coding
and (2) Frontend changes involving javascript, html and css.
Changes fall into two main categories.

1. Backend changes involving python coding
2. Frontend changes involving javascript, html and css.

To test any change, you need to transfer files from your laptop to the VM::

[you@laptop ]# rsync -avz --exclude=.git /path/to/rockstor-core/ root@build_vm:/path/to/build_dir/
[you@laptop ]# rsync -avz --exclude=.git ~/dev/rockstor-core/ root@build_vm:/opt/rockstor-dev/

If you made any javascript, html or css changes, you need to collect static
files with this command::

[root@build_vm ]# /path/to/build_dir/bin/buildout -c /path/to/build_dir/buildout.cfg install collectstatic
[root@build_vm ]# /opt/rockstor-dev/bin/buildout -c /opt/rockstor-dev/buildout.cfg install collectstatic

Then, refresh the browser to test new changes in the WebUI. It's best to have
aliases setup for above commands and have it all integrated into your
Expand All @@ -256,7 +263,7 @@ When making backend changes, you may want to see debug logs and errors.
Everything that you or any rockstor service logs goes into the following
directory on your VM::

[root@build_vm ]# ls -l /path/to/build_dir/var/log
[root@build_vm ]# ls -la /opt/rockstor-dev/var/log
total 280
-rw-r--r-- 1 root root 106912 Jun 23 19:49 gunicorn.log
-rw-r--r-- 1 root root 119533 Jun 23 19:49 rockstor.log
Expand Down Expand Up @@ -294,7 +301,7 @@ bootstrap, d3 and many others. These are not part of the rockstor-core
repository but are dynamically generated during the buildout step. They are
placed in the below directory on your build VM::

[root@build_vm ]# ls /path/to/build_dir/static/js/lib
[root@build_vm ]# ls /opt/rockstor-dev/static/js/lib
backbone-0.9.2.js cocktail.js humanize.js jquery.flot.stack.js jquery.sparkline.min.js json2.js socket.io.min.js
backbone.routefilter.min.js cron jquery-1.9.1.min.js jquery.flot.stackpercent.js jquery.tablesorter.js jsonform.js underscore-1.3.2.js
bootstrap-datepicker.js cubism.v1.js jquery.flot.axislabels.js jquery.flot.time.js jquery.tools.min.js later.min.js
Expand Down Expand Up @@ -330,23 +337,23 @@ are satisfied.
For model changes in storageadmin application, create a migration file using
::

[root@build_vm ]# /path/to/build_dir/bin/django makemigrations storageadmin
[root@build_vm ]# /opt/rockstor-dev/bin/django makemigrations storageadmin

The above command generates a migration file in
:code:`/path/to/build_dir/src/rockstor/storageadmin/migrations/`. Apply the
:code:`/opt/rockstor-dev/src/rockstor/storageadmin/migrations/`. Apply the
migration with::

[root@build_vm ]# /path/to/build_dir/bin/django migrate storageadmin
[root@build_vm ]# /opt/rockstor-dev/bin/django migrate storageadmin

For model changes in the smart_manager application, create a migration file
using::

[root@build_vm ]# /path/to/build_dir/bin/django makemigrations smart_manager
[root@build_vm ]# /opt/rockstor-dev/bin/django makemigrations smart_manager

Run the migration with
::

[root@build_vm ]# /path/to/build_dir/bin/django migrate --database=smart_manager smart_manager
[root@build_vm ]# /opt/rockstor-dev/bin/django migrate --database=smart_manager smart_manager

.. _shipchanges:

Expand All @@ -357,7 +364,7 @@ As you continue to work on an issue, commit and push changes to the issue
branch of your fork. You can periodically push your changes to github with the
following command::

[you@laptop ]# cd /path/to/rockstor-core; git push origin your_branch_name
[you@laptop ]# cd ~/dev/rockstor-core; git push origin your_branch_name

When you finish work for the issue and are ready to submit, create a pull
request by clicking on the "pull request" button on github. This notifies the
Expand Down

0 comments on commit fe77ae5

Please sign in to comment.