Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

exchange vid support prefix (#2232) #2233

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,8 @@ For different data sources, the vertex configurations are different. There are m
|`tags.vertex.udf.separator`|string|-|No|Support merging multiple columns by custom rules. This parameter specifies the join character.|
|`tags.vertex.udf.oldColNames`|list|-|No|Support merging multiple columns by custom rules. This parameter specifies the names of the columns to be merged. Multiple columns are separated by commas.|
|`tags.vertex.udf.newColName`|string|-|No|Support merging multiple columns by custom rules. This parameter specifies the new column name. |
|`tags.vertex.prefix`|string|-|No|Add the specified prefix to the VID. For example, if the VID is `12345`, adding the prefix `tag1` will result in `tag1_12345`. The underscore cannot be modified.|
|`tags.vertex.policy`|string|-|No|Supports only the value `hash`. Performs hashing operations on VIDs of type string.|
|`tags.batch`|int|`256`|Yes|The maximum number of vertices written into NebulaGraph in a single batch.|
|`tags.partition`|int|`32`|Yes|The number of Spark partitions.|

Expand Down Expand Up @@ -251,7 +253,11 @@ For the specific parameters of different data sources for edge configurations, p
|`edges.fields`|list\[string\]|-|Yes|The header or column name of the column corresponding to properties. If there is a header or column name, please use that name directly. If a CSV file does not have a header, use the form of `[_c0, _c1, _c2]` to represent the first column, the second column, the third column, and so on.|
|`edges.nebula.fields`|list\[string\]|-|Yes|Edge names defined in NebulaGraph, the order of which must correspond to `edges.fields`. For example, `[_c2, _c3]` corresponds to `[start_year, end_year]`, which means that values in the third column are the values of the start year, and values in the fourth column are the values of the end year.|
|`edges.source.field`|string|-|Yes|The column of source vertices of edges. For example, `_c0` indicates a value in the first column that is used as the source vertex of an edge.|
|`edges.source.prefix`|string|-|No|Add the specified prefix to the VID. For example, if the VID is `12345`, adding the prefix `tag1` will result in `tag1_12345`. The underscore cannot be modified.|
|`edges.source.policy`|string|-|No|Supports only the value `hash`. Performs hashing operations on VIDs of type string.|
|`edges.target.field`|string|-|Yes|The column of destination vertices of edges. For example, `_c0` indicates a value in the first column that is used as the destination vertex of an edge.|
|`edges.target.prefix`|string|-|No|Add the specified prefix to the VID. For example, if the VID is `12345`, adding the prefix `tag1` will result in `tag1_12345`. The underscore cannot be modified.|
|`edges.target.policy`|string|-|No|Supports only the value `hash`. Performs hashing operations on VIDs of type string.|
|`edges.ranking`|int|-|No|The column of rank values. If not specified, all rank values are `0` by default.|
|`edges.batch`|int|`256`|Yes|The maximum number of edges written into NebulaGraph in a single batch.|
|`edges.partition`|int|`32`|Yes|The number of Spark partitions.|
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,10 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
# policy:hash
# Add the specified prefix to the VID. For example, if the VID is `12345`, adding the prefix `tag1` will result in `tag1_12345`. The underscore cannot be modified.
# prefix:"tag1"
# Performs hashing operations on VIDs of type string.
# policy:hash
}

# Batch operation types, including INSERT, UPDATE, and DELETE. defaults to INSERT.
Expand Down Expand Up @@ -245,6 +248,10 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
# Add the specified prefix to the VID. For example, if the VID is `12345`, adding the prefix `tag1` will result in `tag1_12345`. The underscore cannot be modified.
# prefix:"tag1"
# Performs hashing operations on VIDs of type string.
# policy:hash
}

# In target, use a column in the follow table as the source of the edge's destination vertexes.
Expand All @@ -255,6 +262,10 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
# Add the specified prefix to the VID. For example, if the VID is `12345`, adding the prefix `tag1` will result in `tag1_12345`. The underscore cannot be modified.
# prefix:"tag1"
# Performs hashing operations on VIDs of type string.
# policy:hash
}

# (Optional) Specify a column as the source of the rank.
Expand Down
71 changes: 12 additions & 59 deletions docs-2.0/nebula-exchange/use-exchange/ex-ug-import-from-csv.md
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,10 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
# policy:hash
# Add the specified prefix to the VID. For example, if the VID is `12345`, adding the prefix `tag1` will result in `tag1_12345`. The underscore cannot be modified.
# prefix:"tag1"
# Performs hashing operations on VIDs of type string.
# policy:hash
}

# The delimiter specified. The default value is comma.
Expand All @@ -217,48 +220,20 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`

# Set the information about the Tag Team.
{
# Specify the Tag name defined in NebulaGraph.
name: team
type: {
# Specify the data source file format to CSV.
source: csv

# Specify how to import the data into NebulaGraph: Client or SST.
sink: client
}

# Specify the path to the CSV file.
# If the file is stored in HDFS, use double quotation marks to enclose the file path, starting with hdfs://. For example: "hdfs://ip:port/xx/xx".
# If the file is stored locally, use double quotation marks to enclose the file path, starting with file://. For example: "file:///tmp/xx.csv".
path: "hdfs://192.168.*.*:9000/data/vertex_team.csv"

# If the CSV file does not have a header, use [_c0, _c1, _c2, ..., _cn] to represent its header and indicate the columns as the source of the property values.
# If the CSV file has headers, use the actual column names.
fields: [_c1]

# Specify the column names in the player table in fields, and their corresponding values are specified as properties in the NebulaGraph.
# The sequence of fields and nebula.fields must correspond to each other.
nebula.fields: [name]

# Specify a column of data in the table as the source of VIDs in the NebulaGraph.
# The value of vertex must be the same as the column names in the above fields or csv.fields.
# Currently, NebulaGraph {{nebula.release}} supports only strings or integers of VID.
vertex: {
field:_c0
# policy:hash
}

# The delimiter specified. The default value is comma.
separator: ","

# If the CSV file has a header, set the header to true.
# If the CSV file does not have a header, set the header to false. The default value is false.
header: false

# The number of data written to NebulaGraph in a single batch.
batch: 256

# The number of Spark partitions.
partition: 32
}

Expand Down Expand Up @@ -302,6 +277,10 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
# Add the specified prefix to the VID. For example, if the VID is `12345`, adding the prefix `tag1` will result in `tag1_12345`. The underscore cannot be modified.
# prefix:"tag1"
# Performs hashing operations on VIDs of type string.
# policy:hash
}
target: {
field: _c1
Expand All @@ -310,6 +289,10 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
# Add the specified prefix to the VID. For example, if the VID is `12345`, adding the prefix `tag1` will result in `tag1_12345`. The underscore cannot be modified.
# prefix:"tag1"
# Performs hashing operations on VIDs of type string.
# policy:hash
}

# The delimiter specified. The default value is comma.
Expand All @@ -335,53 +318,23 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`

# Set the information about the Edge Type serve.
{
# Specify the Edge Type name defined in NebulaGraph.
name: serve
type: {
# Specify the data source file format to CSV.
source: csv

# Specify how to import the data into NebulaGraph: Client or SST.
sink: client
}

# Specify the path to the CSV file.
# If the file is stored in HDFS, use double quotation marks to enclose the file path, starting with hdfs://. For example: "hdfs://ip:port/xx/xx".
# If the file is stored locally, use double quotation marks to enclose the file path, starting with file://. For example: "file:///tmp/xx.csv".
path: "hdfs://192.168.*.*:9000/data/edge_serve.csv"

# If the CSV file does not have a header, use [_c0, _c1, _c2, ..., _cn] to represent its header and indicate the columns as the source of the property values.
# If the CSV file has headers, use the actual column names.
fields: [_c2,_c3]

# Specify the column names in the edge table in fields, and their corresponding values are specified as properties in the NebulaGraph.
# The sequence of fields and nebula.fields must correspond to each other.
nebula.fields: [start_year, end_year]

# Specify a column as the source for the source and destination vertexes.
# The value of vertex must be the same as the column names in the above fields or csv.fields.
# Currently, NebulaGraph {{nebula.release}} supports only strings or integers of VID.
source: {
field: _c0
}
target: {
field: _c1
}

# The delimiter specified. The default value is comma.
separator: ","

# Specify a column as the source of the rank (optional).
#ranking: _c5

# If the CSV file has a header, set the header to true.
# If the CSV file does not have a header, set the header to false. The default value is false.
header: false

# The number of data written to NebulaGraph in a single batch.
batch: 256

# The number of Spark partitions.
partition: 32
}

Expand Down
12 changes: 12 additions & 0 deletions docs-2.0/nebula-exchange/use-exchange/ex-ug-import-from-hbase.md
Original file line number Diff line number Diff line change
Expand Up @@ -201,6 +201,10 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
# Add the specified prefix to the VID. For example, if the VID is `12345`, adding the prefix `tag1` will result in `tag1_12345`. The underscore cannot be modified.
# prefix:"tag1"
# Performs hashing operations on VIDs of type string.
# policy:hash
}

# Batch operation types, including INSERT, UPDATE, and DELETE. defaults to INSERT.
Expand Down Expand Up @@ -273,6 +277,10 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
# Add the specified prefix to the VID. For example, if the VID is `12345`, adding the prefix `tag1` will result in `tag1_12345`. The underscore cannot be modified.
# prefix:"tag1"
# Performs hashing operations on VIDs of type string.
# policy:hash
}


Expand All @@ -283,6 +291,10 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
# Add the specified prefix to the VID. For example, if the VID is `12345`, adding the prefix `tag1` will result in `tag1_12345`. The underscore cannot be modified.
# prefix:"tag1"
# Performs hashing operations on VIDs of type string.
# policy:hash
}

# (Optional) Specify a column as the source of the rank.
Expand Down
12 changes: 12 additions & 0 deletions docs-2.0/nebula-exchange/use-exchange/ex-ug-import-from-hive.md
Original file line number Diff line number Diff line change
Expand Up @@ -243,6 +243,10 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
# Add the specified prefix to the VID. For example, if the VID is `12345`, adding the prefix `tag1` will result in `tag1_12345`. The underscore cannot be modified.
# prefix:"tag1"
# Performs hashing operations on VIDs of type string.
# policy:hash
}

# Batch operation types, including INSERT, UPDATE, and DELETE. defaults to INSERT.
Expand Down Expand Up @@ -310,6 +314,10 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
# Add the specified prefix to the VID. For example, if the VID is `12345`, adding the prefix `tag1` will result in `tag1_12345`. The underscore cannot be modified.
# prefix:"tag1"
# Performs hashing operations on VIDs of type string.
# policy:hash
}

target: {
Expand All @@ -319,6 +327,10 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
# Add the specified prefix to the VID. For example, if the VID is `12345`, adding the prefix `tag1` will result in `tag1_12345`. The underscore cannot be modified.
# prefix:"tag1"
# Performs hashing operations on VIDs of type string.
# policy:hash
}

# (Optional) Specify a column as the source of the rank.
Expand Down
12 changes: 12 additions & 0 deletions docs-2.0/nebula-exchange/use-exchange/ex-ug-import-from-jdbc.md
Original file line number Diff line number Diff line change
Expand Up @@ -232,6 +232,10 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
# Add the specified prefix to the VID. For example, if the VID is `12345`, adding the prefix `tag1` will result in `tag1_12345`. The underscore cannot be modified.
# prefix:"tag1"
# Performs hashing operations on VIDs of type string.
# policy:hash
}

# Batch operation types, including INSERT, UPDATE, and DELETE. defaults to INSERT.
Expand Down Expand Up @@ -329,6 +333,10 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
# Add the specified prefix to the VID. For example, if the VID is `12345`, adding the prefix `tag1` will result in `tag1_12345`. The underscore cannot be modified.
# prefix:"tag1"
# Performs hashing operations on VIDs of type string.
# policy:hash
}

target: {
Expand All @@ -338,6 +346,10 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
# Add the specified prefix to the VID. For example, if the VID is `12345`, adding the prefix `tag1` will result in `tag1_12345`. The underscore cannot be modified.
# prefix:"tag1"
# Performs hashing operations on VIDs of type string.
# policy:hash
}

# (Optional) Specify a column as the source of the rank.
Expand Down
Loading