Merge pull request #11 from ConnectEverything/ngs_nsc

Ngs nsc

README.md

@@ -1,8 +1,8 @@
 # NGS
 
+The `ngs` command line tool is used to manage your billing account on Synadia's global messaging service built with NATS. Access to the service relies on the use of Account and User JWTs. These JWTs can be configured and set up locally using the open source tool [nsc](https://github.com/nats-io/nsc). Installing `ngs` using the instructions below will also install the open source `nsc` tool.
 
-A tool for creating an NGS account and users for accessing the NGS global system operated by Synadia.
-
+See [the Synadia site](https://synadia.com/ngs/signup) for more information on how to sign up for Synadia's service and the [NATS documentation site](https://nats-io.github.io/docs/nats_tools/nsc/) for information about using `nsc` to manage your NATS account.
 
 ## Install
 
@@ -18,4 +18,8 @@ Download your platform binary from [here.](https://github.com/connecteverything/
 
 ## Updates are easy
 
-`ngs update` will download and install the latest version.
+`ngs update` will download and install the latest version.
+
+## Release Notes
+
+With 0.9 we are simplifying the NGS tool and relying heavily on the open source `nsc` tool for all JWT editing and creation. More information about migrating to this new version can be found [here](release_notes/migrating_to_0_9.md).

install.py

@@ -31,25 +31,34 @@
 except ImportError:
     from urllib2 import urlopen
 
-REPO_URL = "https://github.com/connecteverything/ngs-cli"
-LATEST_RELEASE_URL = REPO_URL + "/releases/latest"
-TAG_URL = REPO_URL + "/releases/tag/"
-FILENAME_LOOKUP = {
+NGS_REPO_URL = "https://github.com/connecteverything/ngs-cli"
+NGS_LATEST_RELEASE_URL = NGS_REPO_URL + "/releases/latest"
+NGS_TAG_URL = NGS_REPO_URL + "/releases/tag/"
+NGS_FILENAME_LOOKUP = {
     "darwin": "ngs-darwin-amd64.zip",
     "linux": "ngs-linux-amd64.zip",
     "linux2": "ngs-linux-amd64.zip",
     "win32": "ngs-windows-amd64.zip"
 }
 
 
-def release_url(platform, tag):
+NSC_REPO_URL = "https://github.com/nats-io/nsc"
+NSC_LATEST_RELEASE_URL = NSC_REPO_URL + "/releases/latest"
+NSC_TAG_URL = NSC_REPO_URL + "/releases/tag/"
+NSC_FILENAME_LOOKUP = {
+    "darwin": "nsc-darwin-amd64.zip",
+    "linux": "nsc-linux-amd64.zip",
+    "win32": "nsc-windows-amd64.zip"
+}
+
+def ngs_release_url(platform, tag):
     try:
-        filename = FILENAME_LOOKUP[platform]
+        filename = NGS_FILENAME_LOOKUP[platform]
     except KeyError:
         print("Unable to locate appropriate filename for", platform)
         sys.exit(1)
 
-    url = TAG_URL + tag if tag else LATEST_RELEASE_URL
+    url = NGS_TAG_URL + tag if tag else NGS_LATEST_RELEASE_URL
 
     try:
         html = urlopen(url).read().decode('utf-8')
@@ -66,9 +75,32 @@ def release_url(platform, tag):
 
     return "https://github.com" + matching[0]
 
+def nsc_release_url(platform, tag):
+    try:
+        filename = NSC_FILENAME_LOOKUP[platform]
+    except KeyError:
+        print("Unable to locate appropriate filename for", platform)
+        sys.exit(1)
+
+    url = NSC_TAG_URL + tag if tag else NSC_LATEST_RELEASE_URL
 
-def download_with_progress(url):
-    print("Downloading NGS installer: ", url)
+    try:
+        html = urlopen(url).read().decode('utf-8')
+    except:
+        print("Unable to find release page for", tag)
+        sys.exit(1)
+
+    urls = re.findall(r'href=[\'"]?([^\'" >]+)', html)
+    matching = [u for u in urls if filename in u]
+
+    if len(matching) != 1:
+        print("Unable to find download url for", filename)
+        sys.exit(1)
+
+    return "https://github.com" + matching[0]
+
+def download_with_progress(msg, url):
+    print(msg, url)
 
     remote_file = urlopen(url)
     total_size = int(remote_file.headers['Content-Length'].strip())
@@ -90,74 +122,109 @@ def download_with_progress(url):
 
     return b''.join(data)
 
+def mkdir(d):
+    if not os.path.exists(d):
+        os.mkdir(d)
+
+def add_env(bin_dir):
+    home = os.path.expanduser("~")
+    ngs_home = os.path.join(home, ".nsc")
+    env = os.path.join(ngs_home, "env")
+    env_cmd = 'export PATH=' + bin_dir + ':$PATH  #Add NGS utility to the path'
+    with open(env, 'w+') as env_file:
+        env_file.write(env_cmd + '\n')
+    os.chmod(env, 0o744)
+
+def make_bin_dir():
+    home = os.path.expanduser("~")
+    toolhome = os.path.join(home, ".nsc")
+    mkdir(toolhome)
+    bin_dir = os.path.join(toolhome, "bin")
+    mkdir(bin_dir)
+    return bin_dir
 
 def main():
+
+    platform = sys.platform
+    if "linux" in platform:
+            # convert any linux regardless of version reported to "linux"
+            platform = "linux"
+
     print()
-    print("Installing NGS tools for platform: " + sys.platform)
+    print("Installing NGS tools for platform: " + platform)
+
+    url = ngs_release_url(platform, sys.argv[1] if len(sys.argv) > 1 else None)
+    bin_dir = make_bin_dir()
 
-    url = release_url(sys.platform, sys.argv[1] if len(sys.argv) > 1 else None)
-    bin_dir = ngs_bin_dir()
-    exe_fn = os.path.join(bin_dir, "ngs")
+    ngs_exe_path = os.path.join(bin_dir, "ngs")
     if "windows" in url:
-        exe_fn = os.path.join(bin_dir, "ngs.exe")
+        ngs_exe_path = os.path.join(bin_dir, "ngs.exe")
 
-    compressed = download_with_progress(url)
+    compressed = download_with_progress("Downloading NGS installer: ", url)
 
     if url.endswith(".zip"):
         with zipfile.ZipFile(io.BytesIO(compressed), 'r') as z:
-            with open(exe_fn, 'wb+') as exe:
+            with open(ngs_exe_path, 'wb+') as exe:
                 if "windows" not in url:
                     exe.write(z.read('ngs'))
                 else:
                     exe.write(z.read('ngs.exe'))
     else:
         # Note: gzip.decompress is not available in python2.
         content = zlib.decompress(compressed, 15 + 32)
-        with open(exe_fn, 'wb+') as exe:
+        with open(ngs_exe_path, 'wb+') as exe:
+            exe.write(content)
+    os.chmod(ngs_exe_path, 0o744)
+
+    nsc_exe_path = os.path.join(bin_dir, "nsc")
+    if "windows" in url:
+        nsc_exe_path = os.path.join(bin_dir, "nsc.exe")
+
+    url = nsc_release_url(platform, sys.argv[1] if len(sys.argv) > 1 else None)
+    compressed = download_with_progress("Downloading NSC installer: ", url)
+
+    if url.endswith(".zip"):
+        with zipfile.ZipFile(io.BytesIO(compressed), 'r') as z:
+            with open(nsc_exe_path, 'wb+') as exe:
+                if "windows" not in url:
+                    exe.write(z.read('nsc'))
+                else:
+                    exe.write(z.read('nsc.exe'))
+    else:
+        # Note: gzip.decompress is not available in python2.
+        content = zlib.decompress(compressed, 15 + 32)
+        with open(nsc_exe_path, 'wb+') as exe:
             exe.write(content)
-    os.chmod(exe_fn, 0o744)
+    os.chmod(nsc_exe_path, 0o744)
 
     # Add the env helper.
-    ngs_add_env()
+    add_env(bin_dir)
 
-    p_exe_dir = os.path.join('$HOME','.ngs','bin','ngs')
-    env_tool = os.path.join('$HOME', '.ngs', 'env')
+    env_tool = os.path.join('$HOME', '.nsc', 'env')
 
     print()
-    print("The NGS tool has been installed: " + p_exe_dir)
+    print("The NSC and NGS tools have been installed.")
     print()
-    print("You will need to extend your $PATH. Place the ")
-    print("contents of "+ env_tool + " in your shell setup of choice.")
-    print("e.g. 'cat " + env_tool + " >> " + os.path.join('$HOME', '.bashrc') + "'")
+    print(nsc_exe_path + " is used to edit, view and deploy NATS security JWTS.")
+    print(ngs_exe_path + " can be used to signup for the Synadia global service and manage your billing plan.")
     print()
-    print("To get started, try 'source " + env_tool + "'")
-    print("If successful, 'ngs -h' will show the help options.")
+    print("To add these commands to your $PATH")
     print()
-    print("Signup for a  free account using 'ngs signup --free'.")
-    print("When complete, use 'ngs demo echo <msg>' to send your first secure message to the NGS global system.")
+    print("Bash:")
+    print("  echo 'export PATH=\"$PATH:%s\"' >> $HOME/.bash_profile" % bin_dir)
+    print("  source $HOME/.bash_profile")
+    print()
+    print("zsh:")
+    print("  echo 'export PATH=\"$PATH:%s\"' >> $HOME/.zshrc" % bin_dir)
+    print("  source $HOME/.zshrc")
+    print()
+    print("windows:")
+    print("  setx path %%path;\"%s\"" % bin_dir)
+    print()
+    print("If successful, 'ngs -h' and 'nsc -h' will show the help options.")
+    print()
+    print("Signup for a free account by creating a folder, cd-ing into it and using 'nsc init -r synadia'.")
     print()
-
-def mkdir(d):
-    if not os.path.exists(d):
-        os.mkdir(d)
-
-def ngs_add_env():
-    home = os.path.expanduser("~")
-    ngs_home = os.path.join(home, ".ngs")
-    env = os.path.join(ngs_home, "env")
-    bin_dir = os.path.join('$HOME', '.ngs','bin')
-    env_cmd = 'export PATH=' + bin_dir + ':$PATH  #Add NGS utility to the path'
-    with open(env, 'w+') as env_file:
-        env_file.write(env_cmd + '\n')
-    os.chmod(env, 0o744)
-
-def ngs_bin_dir():
-    home = os.path.expanduser("~")
-    toolhome = os.path.join(home, ".ngs")
-    mkdir(toolhome)
-    bin_dir = os.path.join(toolhome, "bin")
-    mkdir(bin_dir)
-    return bin_dir
 
 
 if __name__ == '__main__':

release_notes/migrating_to_0_9.md

@@ -0,0 +1,48 @@
+# Simplifying Account Management for NGS with NSC
+
+With version 0.9 we are simplifying the NATS administrative experience by moving all account JWT management to `nsc` and simplifying the `ngs` tool to only manage your billing account with Synadia. The open sourced `nsc` tool can now be used to edit, view and deploy your account JWTs locally, to a nats-account-server or Synadia’s global service.
+
+To assist in this simplification we need to migrate your current `~/.ngs` folder to `~/.nsc`. The old folder will remain in place, but will no longer be used.
+
+Your keys should already be stored in a central location, `~/.nkeys`, or separately if you choose to manage them elsewhere.
+
+While we won’t delete your old data, the new tools will not be able to use it in the old format. Please treat this as a one-way migration if at all possible.
+
+While this is a fairly major change, we think that it will minimize the context switching between local and NGS accounts, and unify the tools required to manage your account across all nats-servers. Moving forward you will use `nsc` to add imports and exports, or create new users. `ngs` will only be used to edit your billing plan.
+
+Migration is a two part process. First, you will tell `nsc` to download the current JWT for the Synadia operator using the standard `add operator` command. We have extended `nsc` to understand special named operators. You can create these in your enterprise as well, check out the [tools documentation](https://nats-io.github.io/docs/nats_tools/nsc/) to learn how. Adding the Synadia operator will set the default operator to Synadia. Using that default, we can use a hidden command in `nsc` just for this transition to migrate your existing `ngs` accounts and users to the new folder. This process will copy the JWTs for your accounts and upload them on the NGS servers. It will also copy the user JWTs. Once the migration succeeds you can archive the old folder, as you won't need it any longer.
+
+```bash
+nsc add operator -u synadia
+nsc migrate --operator-dir ~/.ngs/nats/synadia
+```
+
+If you don’t have `nsc` installed, you can download the installer at [https://github.com/nats-io/nsc](https://github.com/nats-io/nsc). The latest [ngs installer](https://github.com/ConnectEverything/ngs-cli) will install `nsc` as well.
+
+Once you migrate, you can use `nsc` to add users with `nsc add user -n steve`, while `ngs` has a small set of commands like `ngs status` and `ngs edit` to manage your Synadia billing account.
+
+By default, Synadia provides three demo services (echo, usage, active) to new accounts through a set of automated imports added to your account JWT when you first upload it. These services can now be accessed with the `nsc` tool. In fact, `nsc` has added tools to publish, subscribe, request, reply and check round trip message times with your operator's server. These tools work with NGS but also with other operators if they are configured to include a connect URL.
+
+The echo service returns what you send it, a standard NATS request.
+
+```bash
+% nsc tool req ngs.echo "hello world"
+```
+
+The usage service shows an approximation of your account-wide data usage.
+
+```bash
+% nsc tool req ngs.usage ""
+```
+
+The active service sends out a message stream you can subscribe to. Messages indicate known servers based on various locations in the NGS cluster.
+
+```bash
+% nsc tool sub ngs.active
+```
+
+In all three cases, the operator you see when you type `nsc env` should be Synadia, and the account and user should be the ones you want to use to run the tool.
+
+As always `ngs --help` and `nsc --help` can be used to learn more about what the commands provide.
+
+Thank you for using Synadia’s global messaging service.