Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update eco tools #823

Merged
merged 1 commit into from
Oct 22, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs-2.0/1.introduction/1.what-is-nebula-graph.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ Nebula Graph supports strict role-based access control and external authenticati

### Diversified ecosystem

More and more native tools of Nebula Graph have been released, such as [Nebula Graph Studio](https://github.com/vesoft-inc/nebula-web-docker), [Nebula Console](https://github.com/vesoft-inc/nebula-console), and [Nebula Exchange](https://github.com/vesoft-inc/nebula-spark-utils/tree/v2.0.0/nebula-exchange).
More and more native tools of Nebula Graph have been released, such as [Nebula Graph Studio](https://github.com/vesoft-inc/nebula-web-docker), [Nebula Console](https://github.com/vesoft-inc/nebula-console), and [Nebula Exchange](https://github.com/vesoft-inc/nebula-exchange). For more ecosystem tools, see [Ecosystem tools overview](../20.appendix/6.eco-tool-version.md).

Besides, Nebula Graph has the ability to be integrated with many cutting-edge technologies, such as Spark, Flink, and HBase, for the purpose of mutual strengthening in a world of increasing challenges and chances. For more information, see [Ecosystem development](../20.appendix/6.eco-tool-version.md).

Expand Down
22 changes: 15 additions & 7 deletions docs-2.0/20.appendix/6.eco-tool-version.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,47 +58,55 @@ Nebula Explorer (Explorer for short) is a graph exploration visualization tool t

Nebula Exchange (Exchange for short) is an Apache Spark&trade application for batch migration of data in a cluster to Nebula Graph in a distributed environment. It can support the migration of batch data and streaming data in a variety of different formats. For details, see [What is Nebula Exchange](../nebula-exchange/about-exchange/ex-ug-what-is-exchange.md).

|Nebula Graph version|[Exchange](https://github.com/vesoft-inc/nebula-spark-utils/tree/{{exchange.branch}}/nebula-exchange) version (commit id)|
|Nebula Graph version|Exchange version (commit id)|
|:---|:---|
| {{ nebula.release }} | {{exchange.release}}(2c61ca5) |

## Nebula Operator

Nebula Operator (Operator for short) is a tool to automate the deployment, operation, and maintenance of Nebula Graph clusters on Kubernetes. Building upon the excellent scalability mechanism of Kubernetes, Nebula Graph introduced its operation and maintenance knowledge into the Kubernetes system, which makes Nebula Graph a real cloud-native graph database. For more information, see [What is Nebula Operator](../nebula-operator/1.introduction-to-nebula-operator.md).

|Nebula Graph version|Operator version(commit id)|
|:---|:---|
| {{ nebula.release }} | {{operator.release}}(6d1104e) |

## Nebula Importer

Nebula Importer (Importer for short) is a CSV file import tool for Nebula Graph. The Importer can read the local CSV file, and then import the data into the Nebula Graph database. For details, see [What is Nebula Importer](../nebula-importer/use-importer.md).

|Nebula Graph version|[Importer](https://github.com/vesoft-inc/nebula-importer/tree/{{importer.branch}}) version(commit id)|
|Nebula Graph version|[Importer](https://github.com/vesoft-inc/nebula-importer/tree/{{importer.branch}}) version (commit id)|
|:---|:---|
| {{ nebula.release }} | {{importer.release}}(5c7417d) |

## Nebula Spark Connector

Nebula Spark Connector is a Spark connector that provides the ability to read and write Nebula Graph data in the Spark standard format. Nebula Spark Connector consists of two parts, Reader and Writer. For details, see [What is Nebula Spark Connector](../nebula-spark-connector.md).

|Nebula Graph version|[Spark Connector](https://github.com/vesoft-inc/nebula-spark-utils/tree/{{sparkconnector.branch}}/nebula-spark-connector) version (commit id)|
|Nebula Graph version|Spark Connector version (commit id)|
|:---|:---|
| {{ nebula.release }} | {{sparkconnector.release}}(2c61ca5) |

## Nebula Flink Connector

Nebula Flink Connector is a connector that helps Flink users quickly access Nebula Graph. It supports reading data from the Nebula Graph database or writing data read from other external data sources to the Nebula Graph database. For details, see [What is Nebula Flink Connector](../nebula-flink-connector.md).

|Nebula Graph version|[Flink Connector](https://github.com/vesoft-inc/nebula-flink-connector/tree/{{flinkconnector.branch}}) version (commit id)|
|Nebula Graph version|Flink Connector version (commit id)|
|:---|:---|
| {{ nebula.release }} | {{flinkconnector.release}}(49b8f3d) |

## Nebula Algorithm

Nebula Algorithm (Algorithm for short) is a Spark application based on [GraphX](https://spark.apache.org/graphx/), which uses a complete algorithm tool to analyze data in the Nebula Graph database by submitting a Spark task To perform graph computing, use the algorithm under the lib repository through programming to perform graph computing for DataFrame. For details, see [What is Nebula Algorithm](../nebula-algorithm.md).

|Nebula Graph version|[Algorithm](https://github.com/vesoft-inc/nebula-spark-utils/tree/{{algorithm.branch}}/nebula-algorithm) version (commit id)|
|Nebula Graph version|Algorithm version (commit id)|
|:---|:---|
| {{ nebula.release }} | {{algorithm.release}}(2c61ca5) |

## Nebula Console

Nebula Console is the native CLI client of Nebula Graph. For how to use it, see [Connect Nebula Graph](../2.quick-start/3.connect-to-nebula-graph.md).

|Nebula Graph version|[Console](https://github.com/vesoft-inc/nebula-console) version (commit id)|
|Nebula Graph version|Console version (commit id)|
|:---|:---|
| {{ nebula.release }} | {{console.release}}(3ce5151) |

Expand All @@ -124,7 +132,7 @@ Docker Compose can quickly deploy Nebula Graph clusters. For how to use it, plea

[Nebula Bench](https://github.com/vesoft-inc/nebula-bench) is used to test the baseline performance data of Nebula Graph. It uses the standard data set of LDBC v0.3.3.

|Nebula Graph version|[Nebula Bench](https://github.com/vesoft-inc/nebula-bench) version (commit id)|
|Nebula Graph version|Nebula Bench version (commit id)|
|:---|:---|
| {{ nebula.release }} | {{bench.release}}(661f871) |

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -270,7 +270,7 @@ By default, old versions of Nebula Graph are installed in `/usr/local/nebula/`,

11. Upgrade other Nebula Graph clients.

You must upgrade all other clients to corresponding Nebula Graph v{{nebula.release}}. The clients include but are not limited to [Python](https://github.com/vesoft-inc/nebula-python), [Java](https://github.com/vesoft-inc/nebula-java), [go](https://github.com/vesoft-inc/nebula-go), [C++](https://github.com/vesoft-inc/nebula-cpp), [Flink-connector](https://github.com/vesoft-inc/nebula-flink-connector), [Spark-util](https://github.com/vesoft-inc/nebula-spark-utils), and [Nebula Bench](https://github.com/vesoft-inc/nebula-bench). Find the v{{nebula.release}} branch for each client.
You must upgrade all other clients to corresponding Nebula Graph v{{nebula.release}}. The clients include but are not limited to [Python](https://github.com/vesoft-inc/nebula-python), [Java](https://github.com/vesoft-inc/nebula-java), [go](https://github.com/vesoft-inc/nebula-go), [C++](https://github.com/vesoft-inc/nebula-cpp), [Flink-connector](https://github.com/vesoft-inc/nebula-flink-connector), [Algorithm](https://github.com/vesoft-inc/nebula-algorithm), [Exchange](https://github.com/vesoft-inc/nebula-exchange), [Spark-connector](https://github.com/vesoft-inc/nebula-spark-connector), and [Nebula Bench](https://github.com/vesoft-inc/nebula-bench). Find the v{{nebula.release}} branch for each client.

!!! Note

Expand Down
26 changes: 13 additions & 13 deletions docs-2.0/nebula-algorithm.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Nebula Algorithm

[Nebula Algorithm](https://github.com/vesoft-inc/nebula-spark-utils/tree/master/nebula-algorithm) (Algorithm) is a Spark application based on [GraphX](https://spark.apache.org/graphx/). It uses a complete algorithm tool to perform graph computing on the data in the Nebula Graph database by submitting a Spark task. You can also programmatically use the algorithm under the lib repository to perform graph computing on DataFrame.
[Nebula Algorithm](https://github.com/vesoft-inc/nebula-algorithm) (Algorithm) is a Spark application based on [GraphX](https://spark.apache.org/graphx/). It uses a complete algorithm tool to perform graph computing on the data in the Nebula Graph database by submitting a Spark task. You can also programmatically use the algorithm under the lib repository to perform graph computing on DataFrame.

## Prerequisites

Expand Down Expand Up @@ -35,7 +35,7 @@ The graph computing algorithms supported by Nebula Algorithm are as follows.
| ShortestPath |The shortest path | Path planning, network planning |
| TriangleCount |Triangle counting | Network structure analysis|
| GraphTriangleCount | Graph triangle counting | Network structure and tightness analysis|
| BetweennessCentrality | Intermediate centrality | Key node mining, node influence computing |
| BetweennessCentrality | Intermediate centrality | Key node mining, node influence computing |
| DegreeStatic |Degree of statistical | Graph structure analysis|

## Implementation methods
Expand All @@ -48,22 +48,22 @@ Nebula Algorithm implements the graph calculating as follows:

3. Use graph algorithms provided by GraphX (such as PageRank) or self-implemented algorithms (such as Louvain).

For detailed implementation methods, see [Scala file](https://github.com/vesoft-inc/nebula-spark-utils/tree/master/nebula-algorithm/src/main/scala/com/vesoft/nebula/algorithm/lib).
For detailed implementation methods, see [Scala file](https://github.com/vesoft-inc/nebula-algorithm/tree/master/nebula-algorithm/src/main/scala/com/vesoft/nebula/algorithm/lib).

## Get Nebula Algorithm

### Compile and package

1. Clone the repository `nebula-spark-utils`.
1. Clone the repository `nebula-algorithm`.

```bash
$ git clone -b {{algorithm.branch}} https://github.com/vesoft-inc/nebula-spark-utils.git
$ git clone -b {{algorithm.branch}} https://github.com/vesoft-inc/nebula-algorithm.git
```

2. Enter the directory `nebula-algorithm`.

```bash
$ cd nebula-spark-utils/nebula-algorithm
$ cd nebula-algorithm
```

3. Compile and package.
Expand All @@ -88,13 +88,13 @@ The `lib` repository provides 10 common graph algorithms.

```bash
<dependency>
<groupId>com.vesoft</groupId>
<artifactId>nebula-algorithm</artifactId>
<version>{{algorithm.release}}</version>
<groupId>com.vesoft</groupId>
<artifactId>nebula-algorithm</artifactId>
<version>{{algorithm.release}}</version>
</dependency>
```

2. Use the algorithm (take PageRank as an example) by filling in parameters. For more algorithms, see [Test cases](https://github.com/vesoft-inc/nebula-spark-utils/tree/master/nebula-algorithm/src/test/scala/com/vesoft/nebula/algorithm/lib).
2. Use the algorithm (take PageRank as an example) by filling in parameters. For more algorithms, see [Test cases](https://github.com/vesoft-inc/nebula-algorithm/tree/master/nebula-algorithm/src/test/scala/com/vesoft/nebula/algorithm/lib)).

!!! note
By default, the DataFrame that executes the algorithm sets the first column as the starting vertex, the second column as the destination vertex, and the third column as the edge weights (not the rank in the Nebula Graph).
Expand All @@ -109,7 +109,7 @@ The `lib` repository provides 10 common graph algorithms.
!!! note
There are limitations to use sealed packages. For example, when sinking a repository into Nebula Graph, the property name of the tag created in the sunk graph space must match the preset name in the code. The first method is recommended if the user has development skills.

1. Set the [Configuration file](https://github.com/vesoft-inc/nebula-spark-utils/blob/{{algorithm.branch}}/nebula-algorithm/src/main/resources/application.conf).
1. Set the [Configuration file](https://github.com/vesoft-inc/nebula-algorithm/blob/{{algorithm.branch}}/nebula-algorithm/src/main/resources/application.conf).

```bash
{
Expand Down Expand Up @@ -260,11 +260,11 @@ The `lib` repository provides 10 common graph algorithms.
2. Submit the graph computing task.

```bash
${SPARK_HOME}/bin/spark-submit --master <mode> --class com.vesoft.nebula.algorithm.Main <nebula-algorithm-2.0.0.jar_path> -p <application.conf_path>
${SPARK_HOME}/bin/spark-submit --master <mode> --class com.vesoft.nebula.algorithm.Main <nebula-algorithm-{{algorithm.release}}.jar_path> -p <application.conf_path>
```

Example:

```bash
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.algorithm.Main /root/nebula-spark-utils/nebula-algorithm/target/nebula-algorithm-2.0.0.jar -p /root/nebula-spark-utils/nebula-algorithm/src/main/resources/application.conf
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.algorithm.Main /root/nebula-algorithm/target/nebula-algorithm-{{algorithm.release}}.jar -p /root/nebula-algorithm/src/main/resources/application.conf
```
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# What is Nebula Exchange

[Nebula Exchange](https://github.com/vesoft-inc/nebula-spark-utils/tree/{{exchange.branch}}/nebula-exchange) (Exchange) is an Apache Spark&trade; application for bulk migration of cluster data to Nebula Graph in a distributed environment, supporting batch and streaming data migration in a variety of formats.
[Nebula Exchange](https://github.com/vesoft-inc/nebula-exchange) (Exchange) is an Apache Spark&trade; application for bulk migration of cluster data to Nebula Graph in a distributed environment, supporting batch and streaming data migration in a variety of formats.

Exchange consists of Reader, Processor, and Writer. After Reader reads data from different sources and returns a DataFrame, the Processor iterates through each row of the DataFrame and obtains the corresponding value based on the mapping between `fields` in the configuration file. After iterating through the number of rows in the specified batch, Writer writes the captured data to the Nebula Graph at once. The following figure illustrates the process by which Exchange completes the data conversion and migration.

Expand Down
8 changes: 4 additions & 4 deletions docs-2.0/nebula-exchange/ex-ug-compile.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,16 +11,16 @@ This topic describes how to compile Nebula Exchange. Users can also [download](h

## Steps

1. Clone the repository `nebula-spark-utils` in the `/` directory.
1. Clone the repository `nebula-exchange` in the `/` directory.

```bash
git clone -b {{exchange.branch}} https://github.com/vesoft-inc/nebula-spark-utils.git
git clone -b {{exchange.branch}} https://github.com/vesoft-inc/nebula-exchange.git
```

2. Switch to the directory `nebula-exchange`.

```bash
cd nebula-spark-utils/nebula-exchange
cd nebula-exchange/nebula-exchange
```

3. Package Nebula Exchange.
Expand Down Expand Up @@ -56,7 +56,7 @@ In the `target` directory, users can find the `exchange-2.x.y.jar` file.

The JAR file version changes with the release of the Nebula Java Client. Users can view the latest version on the [Releases page](https://github.com/vesoft-inc/nebula-java/releases).

When migrating data, you can refer to configuration file [`target/classes/application.conf`](https://github.com/vesoft-inc/nebula-spark-utils/blob/master/nebula-exchange/src/main/resources/application.conf).
When migrating data, you can refer to configuration file [`target/classes/application.conf`](https://github.com/vesoft-inc/nebula-exchange/blob/master/nebula-exchange/src/main/resources/application.conf).

## Failed to download the dependency package

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Parameters in the configuration file

This topic describes how to configure the file [`application.conf`](https://github.com/vesoft-inc/nebula-spark-utils/blob/master/nebula-exchange/src/main/resources/application.conf) when users use Nebula Exchange.
This topic describes how to configure the file [`application.conf`](https://github.com/vesoft-inc/nebula-exchange/blob/master/nebula-exchange/src/main/resources/application.conf) when users use Nebula Exchange.

Before configuring the `application.conf` file, it is recommended to copy the file name `application.conf` and then edit the file name according to the file type of a data source. For example, change the file name to `csv_application.conf` if the file type of the data source is CSV.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -287,7 +287,7 @@ ${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchan
For example:

```bash
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchange.Exchange /root/nebula-spark-utils/nebula-exchange/target/nebula-exchange-{{exchange.release}}.jar -c /root/nebula-spark-utils/nebula-exchange/target/classes/clickhouse_application.conf
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchange.Exchange /root/nebula-exchange/nebula-exchange/target/nebula-exchange-{{exchange.release}}.jar -c /root/nebula-exchange/nebula-exchange/target/classes/clickhouse_application.conf
```

You can search for `batchSuccess.<tag_name/edge_name>` in the command output to check the number of successes. For example, `batchSuccess.follow: 300`.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -379,7 +379,7 @@ ${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchan
For example:

```bash
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchange.Exchange /root/nebula-spark-utils/nebula-exchange/target/nebula-exchange-{{exchange.release}}.jar -c /root/nebula-spark-utils/nebula-exchange/target/classes/csv_application.conf
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchange.Exchange /root/nebula-exchange/nebula-exchange/target/nebula-exchange-{{exchange.release}}.jar -c /root/nebula-exchange/nebula-exchange/target/classes/csv_application.conf
```

You can search for `batchSuccess.<tag_name/edge_name>` in the command output to check the number of successes. For example, `batchSuccess.follow: 300`.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -319,7 +319,7 @@ ${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchan
For example:

```bash
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchange.Exchange /root/nebula-spark-utils/nebula-exchange/target/nebula-exchange-{{exchange.release}}.jar -c /root/nebula-spark-utils/nebula-exchange/target/classes/hbase_application.conf
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchange.Exchange /root/nebula-exchange/nebula-exchange/target/nebula-exchange-{{exchange.release}}.jar -c /root/nebula-exchange/nebula-exchange/target/classes/hbase_application.conf
```

You can search for `batchSuccess.<tag_name/edge_name>` in the command output to check the number of successes. For example, `batchSuccess.follow: 300`.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -343,7 +343,7 @@ ${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchan
For example:

```bash
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchange.Exchange /root/nebula-spark-utils/nebula-exchange/target/nebula-exchange-{{exchange.release}}.jar -c /root/nebula-spark-utils/nebula-exchange/target/classes/hive_application.conf -h
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchange.Exchange /root/nebula-exchange/nebula-exchange/target/nebula-exchange-{{exchange.release}}.jar -c /root/nebula-exchange/nebula-exchange/target/classes/hive_application.conf -h
```

You can search for `batchSuccess.<tag_name/edge_name>` in the command output to check the number of successes. For example, `batchSuccess.follow: 300`.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -380,7 +380,7 @@ ${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchan
For example:

```bash
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchange.Exchange /root/nebula-spark-utils/nebula-exchange/target/nebula-exchange-{{exchange.release}}.jar -c /root/nebula-spark-utils/nebula-exchange/target/classes/json_application.conf
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchange.Exchange /root/nebula-echange/nebula-exchange/target/nebula-exchange-{{exchange.release}}.jar -c /root/nebula-exchange/nebula-exchange/target/classes/json_application.conf
```

You can search for `batchSuccess.<tag_name/edge_name>` in the command output to check the number of successes. For example, `batchSuccess.follow: 300`.
Expand Down
Loading