Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CARBONDATA-865] Remove configurations for Kettle from installation-guide.md #734

Closed
wants to merge 1 commit into from
Closed

[CARBONDATA-865] Remove configurations for Kettle from installation-guide.md #734

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 9 additions & 21 deletions docs/installation-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,29 +50,23 @@ followed by :

4. Copy the `./conf/carbon.properties.template` file from CarbonData repository to `$SPARK_HOME/conf/` folder and rename the file to `carbon.properties`.

5. Copy the `./processing/carbonplugins` folder from CarbonData repository to `$SPARK_HOME/carbonlib/` folder.

**NOTE**: carbonplugins will contain .kettle folder.

6. Repeat Step 2 to Step 5 in all the nodes of the cluster.
5. Repeat Step 2 to Step 5 in all the nodes of the cluster.

7. In Spark node[master], configure the properties mentioned in the following table in `$SPARK_HOME/conf/spark-defaults.conf` file.
6. In Spark node[master], configure the properties mentioned in the following table in `$SPARK_HOME/conf/spark-defaults.conf` file.

| Property | Value | Description |
|---------------------------------|-----------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------|
| carbon.kettle.home | `$SPARK_HOME/carbonlib/carbonplugins` | Path that will be used by CarbonData internally to create graph for loading the data |
| spark.driver.extraJavaOptions | `-Dcarbon.properties.filepath=$SPARK_HOME/conf/carbon.properties` | A string of extra JVM options to pass to the driver. For instance, GC settings or other logging. |
| spark.executor.extraJavaOptions | `-Dcarbon.properties.filepath=$SPARK_HOME/conf/carbon.properties` | A string of extra JVM options to pass to executors. For instance, GC settings or other logging. **NOTE**: You can enter multiple values separated by space. |

8. Add the following properties in `$SPARK_HOME/conf/carbon.properties` file:
7. Add the following properties in `$SPARK_HOME/conf/carbon.properties` file:

| Property | Required | Description | Example | Remark |
|----------------------|----------|----------------------------------------------------------------------------------------|-------------------------------------|---------|
| carbon.storelocation | NO | Location where data CarbonData will create the store and write the data in its own format. | hdfs://HOSTNAME:PORT/Opt/CarbonStore | Propose to set HDFS directory |
| carbon.kettle.home | YES | Path that will be used by CarbonData internally to create graph for loading the data. | `$SPARK_HOME/carbonlib/carbonplugins` | |


9. Verify the installation. For example:
8. Verify the installation. For example:

```
./spark-shell --master spark://HOSTNAME:PORT --total-executor-cores 2
Expand Down Expand Up @@ -100,21 +94,17 @@ To get started with CarbonData : [Quick Start](quick-start-guide.md), [DDL Opera

**NOTE**: Create the carbonlib folder if it does not exists inside `$SPARK_HOME` path.

2. Copy the `./processing/carbonplugins` folder from CarbonData repository to `$SPARK_HOME/carbonlib/` folder.

**NOTE**: carbonplugins will contain .kettle folder.
2. Copy the `./conf/carbon.properties.template` file from CarbonData repository to `$SPARK_HOME/conf/` folder and rename the file to `carbon.properties`.

3. Copy the `./conf/carbon.properties.template` file from CarbonData repository to `$SPARK_HOME/conf/` folder and rename the file to `carbon.properties`.

4. Create `tar,gz` file of carbonlib folder and move it inside the carbonlib folder.
3. Create `tar,gz` file of carbonlib folder and move it inside the carbonlib folder.

```
cd $SPARK_HOME
tar -zcvf carbondata.tar.gz carbonlib/
mv carbondata.tar.gz carbonlib/
```

5. Configure the properties mentioned in the following table in `$SPARK_HOME/conf/spark-defaults.conf` file.
4. Configure the properties mentioned in the following table in `$SPARK_HOME/conf/spark-defaults.conf` file.

| Property | Description | Value |
|---------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------|
Expand All @@ -127,15 +117,13 @@ To get started with CarbonData : [Quick Start](quick-start-guide.md), [DDL Opera
| spark.driver.extraJavaOptions | A string of extra JVM options to pass to the driver. For instance, GC settings or other logging. |`-Dcarbon.properties.filepath=$SPARK_HOME/conf/carbon.properties` |


6. Add the following properties in `$SPARK_HOME/conf/carbon.properties`:
5. Add the following properties in `$SPARK_HOME/conf/carbon.properties`:

| Property | Required | Description | Example | Default Value |
|----------------------|----------|----------------------------------------------------------------------------------------|-------------------------------------|---------------|
| carbon.storelocation | NO | Location where CarbonData will create the store and write the data in its own format. | hdfs://HOSTNAME:PORT/Opt/CarbonStore | Propose to set HDFS directory|
| carbon.kettle.home | YES | Path that will be used by CarbonData internally to create graph for loading the data. | carbondata.tar.gz/carbonlib/carbonplugins | |


7. Verify the installation.
6. Verify the installation.

```
./bin/spark-shell --master yarn-client --driver-memory 1g
Expand Down