Add CLI as a golang project

- Add cli/ folder containing golang code
- Add script to compile code to executable in build/cli/build-cli.sh
- Add entrypoint script for development in bin/cli.sh
- Add build profile to build the CLI as part of tha maven build via `-PgoCli`
- Add `conf` service, consisting of
  - `bin/cli.sh conf get` as an example command, equivalent to `bin/alluxio getConf`
  - `bin/cli.sh conf log --name fully.qualified.class.path`, equivalent to `bin/alluxio fsadmin logLevel --logName fully.qualified.class.path`

See #17522 for more background info

			pr-link: #17532

dev docs for defining conventions for the new alluxio cli in golang

			pr-link: #17530

- Add commands to start/stop individual processes, ex. `bin/cli.sh process start master`
- Define process interface and registry
  - process specific java opts are defined with the process and are dynamically added to env
- Add journal format command as part of starting master process
- Mount options for worker are not ported because they will be deprecated in the near future

			pr-link: #17561

Adds `info` subcommand, which contains:
- `cache`: worker capacity information, calls the existing `bin/alluxio fsadmin report capacity` command
- `collect`: collects cluster information into a single tarball, calls the existing `bin/alluxio collectInfo` command
- `master`: master quorum information, calls the existing `bin/alluxio fs masterInfo` command
- `report`: cluster information, calls the existing `bin/alluxio fsadmin report` command, excluding the `capacity` category

			pr-link: #17566

Add journal commands to CLI as part of #17522
			pr-link: #17569

Add quorum/HA related commands as part of #17522
			pr-link: #17570

Add fs commands to golang CLI as part of #17522

unlike other CLI commands, utilize arguments to maintain the existing structure of filesystem commands (ex. cp, du, ls, mv, rm, etc)

			pr-link: #17580

Add generate commands to golang CLI as part of #17522

bin/alluxio docGen -> bin/cli generate docs
bin/alluxio bootstrapConf -> bin/cli generate template

to read ALLUXIO_CONF_DIR, also conducted a minor change on env variables.

			pr-link: #17790

Add exec commands to golang CLI as part of #17522

`bin/alluxio runClass` -> `bin/cli exec class`
`bin/alluxio runTests` -> `bin/cli exec testRun`
`bin/alluxio runUfsTests` -> `bin/cli exec testUfs`
`bin/alluxio runUfsIOTest` -> `bin/cli exec testUfsIO`
`bin/alluxio runHdfsMountTests` -> `bin/cli exec testHdfsMount`
`bin/alluxio runHmsTests` -> `bin/cli exec testHms`
`bin/alluxio runJournalCrashTest` -> `bin/cli exec testJournalCrash`

For command `exec test`, using different `--name` flags can lead to huge difference in other flags and options.
To manage flags and options, currently using different commands for different test types.
			pr-link: #17797

Prepare golang cli for tarball build and replace bin/alluxio

allow java 8 or 11

add cli binaries to other tarball profiles

Add `process` commands with multiple nodes to golang CLI as part of #17522

`bin/alluxio-start.sh masters` -> `bin/cli.sh process start masters`
`bin/alluxio-start.sh job_masters` -> `bin/cli.sh process start job_masters`
`bin/alluxio-start.sh workers` -> `bin/cli.sh process start workers`
`bin/alluxio-start.sh job_workers` -> `bin/cli.sh process start job_workers`
`bin/alluxio-start.sh proxies` -> `bin/cli.sh process start proxies`
`bin/alluxio-start.sh all` -> `bin/cli.sh process start all`
`bin/alluxio-start.sh local` -> `bin/cli.sh process start local`

`bin/alluxio-stop.sh masters` -> `bin/cli.sh process stop masters`
`bin/alluxio-stop.sh job_masters` -> `bin/cli.sh process stop job_masters`
`bin/alluxio-stop.sh workers` -> `bin/cli.sh process stop workers`
`bin/alluxio-stop.sh job_workers` -> `bin/cli.sh process stop job_workers`
`bin/alluxio-stop.sh proxies` -> `bin/cli.sh process stop proxies`
`bin/alluxio-stop.sh all` -> `bin/cli.sh process stop all`
`bin/alluxio-stop.sh local` -> `bin/cli.sh process stop local`

For command `process start/stop` on multiple nodes, using crypto's `ssh` package to create an SSH session, connect to masters, workers or all nodes, then send according subcommand on single nodes to these nodes.
			pr-link: #17887

Add job commands to golang CLI as part of #17522

`bin/alluxio-bash job cancel id` -> `bin/alluxio job cancel --id`
`bin/alluxio-bash job leader` -> `bin/alluxio job leader`
`bin/alluxio-bash job ls` -> `bin/alluxio job list`
`bin/alluxio-bash job getCmdStatus jobControlId` -> `bin/alluxio job cmdStatus --id`
`bin/alluxio-bash job stat [-v] id` -> `bin/alluxio job jobStatus [-v] --id`
`bin/alluxio-bash fs distributedCp src dst` -> `bin/alluxio job submit --type cp --src --dst`
`bin/alluxio-bash fs distributedMv src dst` -> `bin/alluxio job submit --type mv --src --dst`
`bin/alluxio-bash fs load path --submit` -> `bin/alluxio job load --path`

			pr-link: #17931

Clean up multiprocess code

- refactor commands launching the same process on multiple hosts into a single struct
- simplify the logic to ssh into different hosts with more straightforward handling of error channel
- move names.go under its own package to avoid import cycles
			pr-link: #17947

Add an `info version` subcommand to golang CLI as part of #17522

`bin/alluxio-bash version` -> `bin/alluxio info version`
			pr-link: #17943

update calls to old bash scripts to refer to alluxio-bash

Add `cache` commands to golang CLI as part of #17522
`bin/alluxio-bash formatWorker` -> `bin/cli cache format`
`bin/alluxio-bash fs freeWorker` -> `bin/cli cache free --worker`
`bin/alluxio-bash fs free` -> `bin/cli cache free [-f] --path`
`bin/alluxio-bash fs persist` -> `bin/cli cache persist --path`
`bin/alluxio-bash fs setTtl` -> `bin/cli cache ttl --set --duration --action`
`bin/alluxio-bash fs unsetTtl` -> `bin/cli cache ttl --unset`
			pr-link: #17937

backcompat support for start/stop scripts

skip mount args in start script backcompat

remove unsupported cmds

bin/alluxio

@@ -10,8 +10,20 @@
 # See the NOTICE file distributed with this work for information regarding copyright ownership.
 #
 
+set -eu
 
-BIN=$(cd "$( dirname "$( readlink "$0" || echo "$0" )" )"; pwd)
+BIN_DIR=$(cd "$( dirname "$( readlink "$0" || echo "$0" )" )"; pwd)
+ROOT_PATH="${BIN_DIR}/.."
+# run the compiled go binary according to the system OS and arch
+CLI_PATH="${ROOT_PATH}/cli/src/alluxio.org/cli/bin/alluxioCli-$(uname)-$(uname -m)"
 
-# temporary placeholder that redirects to alluxio-bash to prepare for golangCli branch merge
-"${BIN}/alluxio-bash" "$@"
+# if pom.xml exists, assume development environment and compile before running
+if [[ -f "${ROOT_PATH}/pom.xml" ]]; then
+  "${ROOT_PATH}/build/cli/build-cli.sh"
+  "${CLI_PATH}" --rootPath="${ROOT_PATH}" "$@"
+  exit
+fi
+
+# otherwise, assume deployed environment which will have binaries precompiled
+# run with the flag to point to assembly jars in their deployed locations
+"${CLI_PATH}" --rootPath="${ROOT_PATH}" --deployed-env "$@"

bin/alluxio-start.sh

@@ -12,5 +12,16 @@
 
 BIN=$(cd "$( dirname "$( readlink "$0" || echo "$0" )" )"; pwd)
 
-# temporary placeholder that redirects to alluxio-start-bash.sh to prepare for golangCli branch merge
-"${BIN}/alluxio-start-bash.sh" "$@"
+echo "The alluxio-start.sh script is deprecated. Use the \"bin/alluxio process start\" command to start processes."
+
+# while the leading argument starts with a -, continue to parse flags
+# this will skip the mount related arguments that are no longer relevant and would cause an error to the new start command
+startArgs=()
+for arg in "$@"; do
+  startArgs+=("${arg}")
+  if [[ "${arg}" != -* ]]; then
+    break
+  fi
+done
+
+"${BIN}"/alluxio process start "${startArgs[@]}"

bin/alluxio-stop.sh

@@ -12,5 +12,6 @@
 
 BIN=$(cd "$( dirname "$( readlink "$0" || echo "$0" )" )"; pwd)
 
-# temporary placeholder that redirects to alluxio-stop-bash.sh to prepare for golangCli branch merge
-"${BIN}/alluxio-stop-bash.sh" "$@"
+echo "The alluxio-stop.sh script is deprecated. Use the \"bin/alluxio process stop\" command to stop processes."
+
+"${BIN}"/alluxio process stop "$@"

build/cli/build-cli.sh

@@ -0,0 +1,61 @@
+#!/usr/bin/env bash
+#
+# The Alluxio Open Foundation licenses this work under the Apache License, version 2.0
+# (the "License"). You may not use this work except in compliance with the License, which is
+# available at www.apache.org/licenses/LICENSE-2.0
+#
+# This software is distributed on an "AS IS" basis, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
+# either express or implied, as more fully set forth in the License.
+#
+# See the NOTICE file distributed with this work for information regarding copyright ownership.
+#
+
+set -eu
+
+CLI_DIR=$(cd "$( dirname "$( readlink "$0" || echo "$0" )" )"; pwd)
+
+# mapping of GOOS/GOARCH value to uname value
+declare -A ALL_OS
+ALL_OS["linux"]="Linux"
+ALL_OS["darwin"]="Darwin"
+declare -A ALL_ARCH
+ALL_ARCH["amd64"]="x86_64"
+ALL_ARCH["arm64"]="aarch64"
+
+MAIN_PATH="cli/main.go"
+USAGE="Usage: build-cli.sh [-a]
+-a   Build executables for all OS and architecture combinations
+"
+
+main() {
+  build_all="false"
+  while getopts "a" opt; do
+    case "${opt}" in
+      a)
+        build_all="true"
+        ;;
+      *)
+        echo -e "${USAGE}" >&2
+        exit 1
+        ;;
+    esac
+  done
+
+  cliBinDir="${CLI_DIR}/../../cli/src/alluxio.org/cli/bin"
+  mkdir -p "${cliBinDir}"
+
+  cd "${CLI_DIR}/../../cli/src/alluxio.org/"
+
+  if [[ ${build_all} == "false" ]]; then
+    GO111MODULE=on go build -o "${cliBinDir}/alluxioCli-$(uname)-$(uname -m)" "${MAIN_PATH}"
+  else
+    for osKey in "${!ALL_OS[@]}"; do
+      for archKey in "${!ALL_ARCH[@]}"; do
+        echo "Building executable for ${osKey} ${archKey}"
+        GO111MODULE=on GOOS="${osKey}" GOARCH="${archKey}" go build -o "${cliBinDir}/alluxioCli-${ALL_OS[$osKey]}-${ALL_ARCH[$archKey]}" "${MAIN_PATH}"
+      done
+    done
+  fi
+}
+
+main "$@"

cli/README.md

@@ -0,0 +1,58 @@
+# Alluxio CLI
+
+The Alluxio command line interface is the single entrypoint for users to:
+- Initialize and configure the Alluxio cluster
+- Start and stop processes
+- Expose information about the running cluster
+- Interact with the filesystem, running commands such as `ls` and `cp`
+- Perform administrator level actions such as format or backup
+
+The CLI is invoked through the shell script at `bin/cli.sh`.
+Commands follow the format of:
+```console
+bin/cli.sh <service> <operation> [--<flag>[=<value>]] [<args>]
+```
+
+Add the `-h` flag to view more details regarding a service or operation.
+
+## Layout and naming conventions
+
+The choice of names for services, operations, and flags should be succinct: short and unambiguous.
+Use of a single word is strongly preferred, but otherwise the name parts should be delimited by a dash such as `foo-bar`.
+
+For example, let's assume there is a `mark` operation as part of a `item` service that can be `set` or `unset` on an item `name`.
+The recommended format for a command is
+```console
+bin/cli.sh item mark --set name
+bin/cli.sh item mark --unset name
+```
+where it is expected that either `--set` or `--unset` are specified.
+This is preferred over the alternative of two separate commands with `setMark` and `unsetMark` as the operations.
+
+## User input
+
+### Flags and arguments
+After selecting the desired command, additional user input can be parsed, as a mix of arguments and/or flags:
+- Arguments: `bin/cli.sh command arg1 arg2 ...`
+- Flags: `bin/cli.sh command --flag1 --flag2 val2 ...`
+
+Flags are strictly preferred over arguments.
+- The flag name conveys context; an argument does not
+- Arguments must be ordered; flags can be declared arbitrarily
+- Flags can be designated as required to ensure user input.
+- Repeated flags can be defined to capture an ordered list of inputs, ex. `--target target1 --target target2`
+
+### Input validation
+
+User inputs should be validated by the CLI command as much as possible as opposed to the resulting invocation.
+
+## Output conventions and java invocation
+
+A majority of commands result in invoking a java class with arguments to execute the expected operation and possibly return some output.
+The output returned from the java invocation should tend towards being plain or machine parseable, such as a JSON formatted string,
+rather than terminal friendly or human readable format.
+When appropriate, the CLI command will default to formatting this output to be terminal friendly, with an option to output in a machine parseable format.
+
+## References
+
+Github CLI guidelines: https://primer.style/design/native/cli/foundations

cli/src/alluxio.org/cli/bin/.gitignore

@@ -0,0 +1 @@
+alluxioCli*

cli/src/alluxio.org/cli/cmd/cache/cache.go

@@ -0,0 +1,25 @@
+/*
+ * The Alluxio Open Foundation licenses this work under the Apache License, version 2.0
+ * (the "License"). You may not use this work except in compliance with the License, which is
+ * available at www.apache.org/licenses/LICENSE-2.0
+ *
+ * This software is distributed on an "AS IS" basis, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
+ * either express or implied, as more fully set forth in the License.
+ *
+ * See the NOTICE file distributed with this work for information regarding copyright ownership.
+ */
+
+package cache
+
+import (
+	"alluxio.org/cli/env"
+)
+
+var Service = &env.Service{
+	Name:        "cache",
+	Description: "Worker-related file system and format operations.",
+	Commands: []env.Command{
+		Format,
+		Free,
+	},
+}

cli/src/alluxio.org/cli/cmd/cache/format.go

@@ -0,0 +1,53 @@
+/*
+ * The Alluxio Open Foundation licenses this work under the Apache License, version 2.0
+ * (the "License"). You may not use this work except in compliance with the License, which is
+ * available at www.apache.org/licenses/LICENSE-2.0
+ *
+ * This software is distributed on an "AS IS" basis, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
+ * either express or implied, as more fully set forth in the License.
+ *
+ * See the NOTICE file distributed with this work for information regarding copyright ownership.
+ */
+
+package cache
+
+import (
+	"fmt"
+
+	"github.com/spf13/cobra"
+
+	"alluxio.org/cli/env"
+)
+
+var Format = &FormatCommand{
+	BaseJavaCommand: &env.BaseJavaCommand{
+		CommandName:   "format",
+		JavaClassName: "alluxio.cli.Format",
+		ShellJavaOpts: fmt.Sprintf(env.JavaOptFormat, env.ConfAlluxioLoggerType, "Console"),
+	},
+}
+
+type FormatCommand struct {
+	*env.BaseJavaCommand
+}
+
+func (c *FormatCommand) Base() *env.BaseJavaCommand {
+	return c.BaseJavaCommand
+}
+
+func (c *FormatCommand) ToCommand() *cobra.Command {
+	cmd := c.Base().InitRunJavaClassCmd(&cobra.Command{
+		Use:   Format.CommandName,
+		Short: "Format Alluxio worker nodes.",
+		Args:  cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, args []string) error {
+			return c.Run(args)
+		},
+	})
+	return cmd
+}
+
+func (c *FormatCommand) Run(args []string) error {
+	javaArgs := []string{"worker"}
+	return c.Base().Run(javaArgs)
+}

cli/src/alluxio.org/cli/cmd/cache/free.go

@@ -0,0 +1,80 @@
+/*
+ * The Alluxio Open Foundation licenses this work under the Apache License, version 2.0
+ * (the "License"). You may not use this work except in compliance with the License, which is
+ * available at www.apache.org/licenses/LICENSE-2.0
+ *
+ * This software is distributed on an "AS IS" basis, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
+ * either express or implied, as more fully set forth in the License.
+ *
+ * See the NOTICE file distributed with this work for information regarding copyright ownership.
+ */
+
+package cache
+
+import (
+	"github.com/palantir/stacktrace"
+	"github.com/spf13/cobra"
+
+	"alluxio.org/cli/cmd/names"
+	"alluxio.org/cli/env"
+)
+
+var Free = &FreeCommand{
+	BaseJavaCommand: &env.BaseJavaCommand{
+		CommandName:   "free",
+		JavaClassName: names.FileSystemShellJavaClass,
+	},
+}
+
+type FreeCommand struct {
+	*env.BaseJavaCommand
+	worker string
+	path   string
+	force  bool
+}
+
+func (c *FreeCommand) Base() *env.BaseJavaCommand {
+	return c.BaseJavaCommand
+}
+
+func (c *FreeCommand) ToCommand() *cobra.Command {
+	cmd := c.Base().InitRunJavaClassCmd(&cobra.Command{
+		Use: Free.CommandName,
+		Short: "Synchronously free all blocks and directories of specific worker, " +
+			"or free the space occupied by a file or a directory in Alluxio",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, args []string) error {
+			return c.Run(args)
+		},
+	})
+	cmd.Flags().StringVar(&c.worker, "worker", "", "The worker to free")
+	cmd.Flags().StringVar(&c.path, "path", "", "The file or directory to free")
+	cmd.Flags().BoolVarP(&c.force, "force", "f", false,
+		"Force freeing pinned files in the directory")
+	cmd.MarkFlagsMutuallyExclusive("worker", "path")
+	return cmd
+}
+
+func (c *FreeCommand) Run(args []string) error {
+	var javaArgs []string
+	if c.worker == "" {
+		if c.path != "" {
+			// free directory
+			javaArgs = append(javaArgs, "free")
+			if c.force {
+				javaArgs = append(javaArgs, "-f")
+			}
+			javaArgs = append(javaArgs, c.path)
+		} else {
+			return stacktrace.NewError("neither worker nor path to free specified")
+		}
+	} else {
+		if c.path == "" {
+			// free workers
+			javaArgs = append(javaArgs, "freeWorker", c.worker)
+		} else {
+			return stacktrace.NewError("both worker and path to free specified")
+		}
+	}
+	return c.Base().Run(javaArgs)
+}

cli/src/alluxio.org/cli/cmd/conf/conf.go

@@ -0,0 +1,25 @@
+/*
+ * The Alluxio Open Foundation licenses this work under the Apache License, version 2.0
+ * (the "License"). You may not use this work except in compliance with the License, which is
+ * available at www.apache.org/licenses/LICENSE-2.0
+ *
+ * This software is distributed on an "AS IS" basis, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
+ * either express or implied, as more fully set forth in the License.
+ *
+ * See the NOTICE file distributed with this work for information regarding copyright ownership.
+ */
+
+package conf
+
+import (
+	"alluxio.org/cli/env"
+)
+
+var Service = &env.Service{
+	Name:        "conf",
+	Description: "Get, set, and validate configuration settings, primarily those defined in conf/alluxio-site.properties",
+	Commands: []env.Command{
+		Get,
+		Log,
+	},
+}