-
Notifications
You must be signed in to change notification settings - Fork 16
QAT Codec Deployment Guide (Non CDH or Manually Setup Cluster)
Note that the installation guide does not favour any platforms. It is mostly based on IntelQATCodec 1.0.0 and CentOS 7.5.
It is recommended the platform be updated to the latest available packages
sudo yum update
Kernel-devel version must be the same with the version of your kernel. If the package update installed a new kernel, then the platform must be restarted.
sudo shutdown -r now
Install systemd-devel by running:
sudo yum install systemd-devel
The following steps should be performed on all hosts.
Make sure YUM works.
Create a file named intel-qat.repo in /etc/yum.repos.d with the following contents.
[intel-qat]
name=Intel QAT
baseurl=https://download.01.org/QAT/repo
gpgcheck=0
Clean Yum cache by running:
sudo yum clean all
Install QATzip by running:
sudo yum install QATzip
Configure huge page related setting by running:
sudo cat /etc/sysctl.conf ensure the line vm.nr_hugepages=512 exists in the file
sudo sysctl -p
sudo cat /proc/meminfo | grep -i hugepages_total
HugePages_Total: 512
Ensure QAT services by running the following commands on all hosts:
sudo service qat_service start
Run the following command to check if the QATzip is setup correctly for compressing or decompressing files.
qzip -k $your_input_file
Download QAT Codec release from Releases, extract the release qatcodec-<version>.tar.gz and find the QAT Codec artifacts in qatcodec-<version>\qat-codec-<version> directory. Follow any one of the below procedure to install the QAT Codec in a manually setup Hadoop cluster.
1. Copy the jar file and .so file in the appropriate location of Hadoop installation (for all nodes in the cluster)
Step 1: Copy the hadoop_qat_codec*.jar file to $HADOOP_COMMON_HOME/share/hadoop/common
Step 2: Copy the libqatcodec.so file to $HADOOP_COMMON_HOME/lib/native
Step 3 (For Spark): Copy the spark_qat_codec*.jar to $SPARK_HOME/assembly/target/scala-2.11/jars
Step 4 (optional, required if use Hive): Create soft link for Parquet related jars (parquet-format-[CDH_Version][QAT codec version].jar and parquet-hadoop-bundle-[CDH_version][QAT codec version]) and Hive related jars (hive-exec-[CDH_Version][QAT codec version].jar, hive-shim-0.23-[CDH_version][QAT codec version].jar and hive-shims-common-[CDH_version][QAT codec version].jar) to CDH jars folder ([path/to/CDH/lib/hive/]).
(Or)
Step 1: Copy the hadoop_qat_codec*.jar and libqatcodec.so file to the same location in all the nodes in the cluster
Step 2: Add the location of hadoop_qat_codec*.jar to mapreduce.application.classpath in mapred-site.xml or yarn.application.classpath in yarn-site.xml.
Step 3: Add the location of libqatcodec.so file to mapreduce.admin.user.env in mapred-site.xml.
For Spark,
Step 4: Copy the spark_qat_codec*.jar to the same location in all the nodes in the cluster
Step 5: Add the location of spark_qat_codec*.jar to spark.driver.extraClassPath and spark.executor.extraClassPath in spark-defaults.conf
After performing the above steps using any of the approach, restart the necessary running Hadoop/Spark services to take effect of the classpath and env update.
Make sure the following users are in qat user group in operating system.
hdfs, mapred, yarn, spark
Use the following commands to add these users into qat user group on all hosts.
sudo usermod -a -G qat hdfs
sudo usermod -a -G qat mapred
sudo usermod -a -G qat yarn
sudo usermod -a -G qat spark
Use the class org.apache.hadoop.io.compress.QatCodec for Hadoop compression codec class and org.apache.spark.io.QatCompressionCodec for Spark compression codec like below,
mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.QatCodec
mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.QatCodec
spark.io.compression.codec=org.apache.spark.io.QatCompressionCodec
Please find the detailed doc for QAT Codec internals, additional configurations and deployment information.
[std_err]
[error] SalCtrl_ServiceInit() - : Failed to initialise all service instances
[error] SalCtrl_ServiceEventStart() - : Private data is NULL
[error] SalCtrl_AdfServicesStartedCheck() - : Sal Ctrl failed to start in given time
[std_out]
ADF_UIO_PROXY err: get_bundle_from_dev_cached: failed to open uio dev /dev/uio0
ADF_UIO_PROXY err: adf_user_subsystemInit: Failed to initialise Subservice SAL
ADF_UIO_PROXY err: adf_user_subsystemStart: Failed to start Subservice SAL
ADF_UIO_PROXY err: icp_adf_subsystemUnregister: Failed to shutdown subservice SAL.
ADF_UIO_PROXY err: icp_adf_userProcessToStart: File /dev/qat_dev_processes already opened
Perform the commands below on all hosts:
sudo chmod 666 /dev/uio*
sudo chmod 666 /dev/usdm_drv
[std_out]
ADF_UIO_PROXY err: icp_adf_userProcessToStart: Error reading /dev/qat_dev_processes file
Perform the commands below on all hosts:
sudo chmod 666 /dev/qat_dev_processes
[std_err]
Error no hardware, switch to SW if permitted
g_process.qz_init_status = QZ_NO_HW
Perform the commands below on all hosts:
sudo chmod 777 /dev/qat_adf_ctl
sudo chmod 777 /dev/hugepages