CDH (Cloudera's Distribution including Apache Hadoop) is a popular distribution of Hadoop. Although later versions like CDH 6.0.1 (with Hadoop 3.0.0) natively support OSS, older versions like CDH 5 (with Hadoop 2.6) do not. This topic shows you how to configure a CDH 5 cluster to read data from and write data to OSS.
Prerequisites
You have a running CDH 5 cluster. This topic uses CDH 5.14.4 as an example. For instructions on how to set up a CDH 5 cluster, see the official documentation.Background information
CDH 5 uses an older version of the httpclient and httpcore libraries (version 4.2.5), as required by its Resource Manager. However, the OSS SDK requires newer versions of these libraries. This topic provides a workaround to resolve this dependency conflict.
Step 1: Add OSS configurations
Perform the following steps on all nodes in your CDH cluster:
- Check the directory structure of your CDH 5 installation, located at
${CDH_HOME}:[root@cdh-master CDH-5.14.4-1.cdh5.14.4.p0.3]# ls -lh total 100K drwxr-xr-x 2 root root 4.0K Jun 12 21:03 bin drwxr-xr-x 27 root root 4.0K Jun 12 20:57 etc drwxr-xr-x 5 root root 4.0K Jun 12 20:57 include drwxr-xr-x 2 root root 68K Jun 12 21:09 jars drwxr-xr-x 38 root root 4.0K Jun 12 21:03 lib drwxr-xr-x 3 root root 4.0K Jun 12 20:57 lib64 drwxr-xr-x 3 root root 4.0K Jun 12 20:51 libexec drwxr-xr-x 2 root root 4.0K Jun 12 21:02 meta drwxr-xr-x 4 root root 4.0K Jun 12 21:03 shareNote In this topic, items enclosed in${}are environment variables. Replace them with the appropriate values for your environment. - Download the OSS support package for CDH 5.14.4 to the
jarsfolder of your CDH 5 installation directory.This package was compiled based on the Hadoop version in CDH 5.14.4 and includes the necessary patch for OSS support. You can download packages for other CDH versions from the following links:- CDH 5.8.5
- CDH 5.4.4
- CDH 6.3.2
Note For CDH 6.3.2, copy the package files to the
jarsfolder of your CDH 6 installation directory. Then, follow the next steps to deploy the package. This involves updating thealiyun-sdk-oss-3.4.1.jarfile and creating symbolic links for thealiyun-java-sdk-*.jarfiles to their corresponding locations.
- Extract the package.
[root@cdh-master CDH-5.14.4-1.cdh5.14.4.p0.3]# tar -tvf hadoop-oss-cdh-5.14.4.tar.gz drwxr-xr-x root/root 0 2018-10-08 18:16 hadoop-oss-cdh-5.14.4/ -rw-r--r-- root/root 13277 2018-10-08 17:36 hadoop-oss-cdh-5.14.4/aliyun-java-sdk-sts-3.0.0.jar -rw-r--r-- root/root 326724 2018-10-08 18:16 hadoop-oss-cdh-5.14.4/httpcore-4.4.4.jar -rw-r--r-- root/root 524927 2018-10-08 17:36 hadoop-oss-cdh-5.14.4/aliyun-sdk-oss-3.4.1.jar -rw-r--r-- root/root 116337 2018-10-08 17:36 hadoop-oss-cdh-5.14.4/aliyun-java-sdk-core-3.4.0.jar -rw-r--r-- root/root 215492 2018-10-08 17:36 hadoop-oss-cdh-5.14.4/aliyun-java-sdk-ram-3.0.0.jar -rw-r--r-- root/root 788137 2018-10-08 17:36 hadoop-oss-cdh-5.14.4/aliyun-java-sdk-ecs-4.2.0.jar -rw-r--r-- root/root 70017 2018-10-08 17:36 hadoop-oss-cdh-5.14.4/hadoop-aliyun-2.6.0-cdh5.14.4.jar -rw-r--r-- root/root 736658 2018-10-08 18:16 hadoop-oss-cdh-5.14.4/httpclient-4.5.2.jar - Navigate to the
${CDH_HOME}/lib/hadoopdirectory and run the following commands:[root@cdh-master hadoop]# rm -f lib/httpclient-4.2.5.jar [root@cdh-master hadoop]# rm -f lib/httpcore-4.2.5.jar [root@cdh-master hadoop]# ln -s ../../jars/hadoop-aliyun-2.6.0-cdh5.14.4.jar hadoop-aliyun-2.6.0-cdh5.14.4.jar [root@cdh-master hadoop]# ln -s hadoop-aliyun-2.6.0-cdh5.14.4.jar hadoop-aliyun.jar [root@cdh-master hadoop]# cd lib [root@cdh-master lib]# ln -s ../../../jars/aliyun-java-sdk-core-3.4.0.jar aliyun-java-sdk-core-3.4.0.jar [root@cdh-master lib]# ln -s ../../../jars/aliyun-java-sdk-ecs-4.2.0.jar aliyun-java-sdk-ecs-4.2.0.jar [root@cdh-master lib]# ln -s ../../../jars/aliyun-java-sdk-ram-3.0.0.jar aliyun-java-sdk-ram-3.0.0.jar [root@cdh-master lib]# ln -s ../../../jars/aliyun-java-sdk-sts-3.0.0.jar aliyun-java-sdk-sts-3.0.0.jar [root@cdh-master lib]# ln -s ../../../jars/aliyun-sdk-oss-3.4.1.jar aliyun-sdk-oss-3.4.1.jar [root@cdh-master lib]# ln -s ../../../jars/httpclient-4.5.2.jar httpclient-4.5.2.jar [root@cdh-master lib]# ln -s ../../../jars/httpcore-4.4.4.jar httpcore-4.4.4.jar [root@cdh-master lib]# ln -s ../../../jars/jdom-1.1.jar jdom-1.1.jar - ${CDH_HOME}/lib/hadoop-yarn/bin/yarn
CLASSPATH=${CLASSPATH}:$HADOOP_YARN_HOME/${YARN_DIR}/* CLASSPATH=${CLASSPATH}:$HADOOP_YARN_HOME/${YARN_LIB_JARS_DIR}/*CLASSPATH=$HADOOP_YARN_HOME/${YARN_DIR}/*:${CLASSPATH}CLASSPATH=$HADOOP_YARN_HOME/${YARN_LIB_JARS_DIR}/*:${CLASSPATH}On the node where the Resource Manager is deployed, navigate to the directory. Open the script and replace with . - On the node where the Resource Manager is deployed, navigate to the
${CDH_HOME}/lib/hadoop-yarn/libdirectory and run the following commands:[root@cdh-master lib]# ln -s ../../../jars/httpclient-4.2.5.jar httpclient-4.2.5.jar [root@cdh-master lib]# ln -s ../../../jars/httpcore-4.2.5.jar httpcore-4.2.5.jar - Add the required configurations using Cloudera Manager.
If your cluster is not managed by Cloudera Manager, you can modify the
core-site.xmlfile directly. In Cloudera Manager, navigate to the HDFS service's Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml page and add the OSS-related parameters.Parameter Value fs.oss.endpoint The connection endpoint for OSS. For example, the endpoint for the China (Hangzhou) region is
oss-cn-hangzhou.aliyuncs.comfs.oss.accessKeyId Your AccessKey ID. To learn how to obtain an AccessKey pair, see Create an AccessKey pair. fs.oss.accessKeySecret Your AccessKey Secret. To learn how to obtain an AccessKey pair, see Create an AccessKey pair. fs.oss.impl The implementation class for the Hadoop OSS file system. Set this to org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemfs.oss.buffer.dir The temporary file directory. Recommended value:
/tmp/ossfs.oss.connection.secure.enabled Specifies whether to enable HTTPS. Enabling HTTPS can affect performance. Recommended value: false
fs.oss.connection.maximum The maximum number of connections to OSS. Recommended value: 2048
For more information about the parameters, see Hadoop-Aliyun module.
- Restart the cluster as prompted by Cloudera Manager.
- Test reading from and writing to OSS.
- Test reading data:
hadoop fs -ls oss://${your-bucket-name}/ - Test writing data:
hadoop fs -mkdir oss://${your-bucket-name}/hadoop-testIf you can read from and write to OSS, the setup is successful. Otherwise, review your configuration.
- Test reading data:
Step 2: Configure Impala to support OSS
Impala can directly query data stored in HDFS. After you enable OSS support for your CDH 5 cluster, you can also use Impala to query data in OSS. Because the OSS SDK requires newer versions of the httpclient and httpcore libraries, follow these steps on all nodes where Impala is deployed:
- Navigate to the
${CDH_HOME}/lib/impala/libdirectory and run the following commands:[root@cdh-master lib]# rm -f httpclient-4.2.5.jar httpcore-4.2.5.jar [root@cdh-master lib]# ln -s ../../../jars/httpclient-4.5.2.jar httpclient-4.5.2.jar [root@cdh-master lib]# ln -s ../../../jars/httpcore-4.4.4.jar httpcore-4.4.4.jar [root@cdh-master lib]# ln -s ../../../jars/hadoop-aliyun-2.6.0-cdh5.14.4.jar hadoop-aliyun.jar [root@cdh-master lib]# ln -s ../../../jars/aliyun-java-sdk-core-3.4.0.jar aliyun-java-sdk-core-3.4.0.jar [root@cdh-master lib]# ln -s ../../../jars/aliyun-java-sdk-ecs-4.2.0.jar aliyun-java-sdk-ecs-4.2.0.jar [root@cdh-master lib]# ln -s ../../../jars/aliyun-java-sdk-ram-3.0.0.jar aliyun-java-sdk-ram-3.0.0.jar [root@cdh-master lib]# ln -s ../../../jars/aliyun-java-sdk-sts-3.0.0.jar aliyun-java-sdk-sts-3.0.0.jar [root@cdh-master lib]# ln -s ../../../jars/aliyun-sdk-oss-3.4.1.jar aliyun-sdk-oss-3.4.1.jar [root@cdh-master lib]# ln -s ../../../jars/jdom-1.1.jar jdom-1.1.jar - Navigate to the
${CDH_HOME}/bindirectory. Modify theimpalad,statestored, andcatalogdscripts. In each file, add the following line before the finalexeccommand:export CLASSPATH=$CLASSPATH:${IMPALA_HOME}/lib/httpclient-4.5.2.jar:${IMPALA_HOME}/lib/httpcore-4.4.4.jar:${IMPALA_HOME}/lib/hadoop-aliyun.jar:${IMPALA_HOME}/lib/aliyun-java-sdk-core-3.4.0.jar:${IMPALA_HOME}/lib/aliyun-java-sdk-ecs-4.2.0.jar:${IMPALA_HOME}/lib/aliyun-java-sdk-ram-3.0.0.jar:${IMPALA_HOME}/lib/aliyun-java-sdk-sts-3.0.0.jar:${IMPALA_HOME}/lib/aliyun-sdk-oss-3.4.1.jar:${IMPALA_HOME}/lib/jdom-1.1.jar - Restart all Impala-related processes on all nodes.
After the restart, you can use Impala to query data in OSS.
Verify configuration
For example, consider a TPC-DS benchmark table named
call_center stored in OSS. You can create an external table pointing to this data and then run a query to count the records grouped by cc_country. [root@cdh-master ~]# impala-shell -i cdh-slave01:21000
Starting Impala Shell without Kerberos authentication
Connected to cdh-slave01:21000
Server version: impalad version 2.11.0-cdh5.14.4 RELEASE (build20e635646a13347800fad36a7d0b1da25ab32404)
***********************************************************************************
Welcome to the Impala shell.
(Impala Shell v2.11.0-cdh5.14.4 (20e6356) built on Tue Jun 1203:43:08 PDT 2018)
The HISTORY command lists all shell commands in chronological order.
***********************************************************************************
[cdh-slave01:21000] > droptableifexists call_center;
Query: droptableifexists call_center
[cdh-slave01:21000] >
[cdh-slave01:21000] > createexternaltable call_center(
> cc_call_center_sk bigint
> , cc_call_center_id string
> , cc_rec_start_date string
> , cc_rec_end_date string
> , cc_closed_date_sk bigint
> , cc_open_date_sk bigint
> , cc_name string
> , cc_class string
> , cc_employees int
> , cc_sq_ft int
> , cc_hours string
> , cc_manager string
> , cc_mkt_id int
> , cc_mkt_class string
> , cc_mkt_desc string
> , cc_market_manager string
> , cc_division int
> , cc_division_name string
> , cc_company int
> , cc_company_name string
> , cc_street_number string
> , cc_street_name string
> , cc_street_type string
> , cc_suite_number string
> , cc_city string
> , cc_county string
> , cc_state string
> , cc_zip string
> , cc_country string
> , cc_gmt_offset double
> , cc_tax_percentage double
> )
> rowformatdelimitedfieldsterminatedby'|'
> location 'oss://${your-bucket-name}/call_center';
Query: createexternaltable call_center(
cc_call_center_sk bigint
, cc_call_center_id string
, cc_rec_start_date string
, cc_rec_end_date string
, cc_closed_date_sk bigint
, cc_open_date_sk bigint
, cc_name string
, cc_class string
, cc_employees int
, cc_sq_ft int
, cc_hours string
, cc_manager string
, cc_mkt_id int
, cc_mkt_class string
, cc_mkt_desc string
, cc_market_manager string
, cc_division int
, cc_division_name string
, cc_company int
, cc_company_name string
, cc_street_number string
, cc_street_name string
, cc_street_type string
, cc_suite_number string
, cc_city string
, cc_county string
, cc_state string
, cc_zip string
, cc_country string
, cc_gmt_offset double
, cc_tax_percentage double
)
rowformatdelimitedfieldsterminatedby'|'
location 'oss://${your-bucket-name}/call_center'
Fetched 0row(s) in0.07s
[cdh-slave01:21000] > select cc_country, count(*) from call_center groupby cc_country;
Query: select cc_country, count(*) from call_center groupby cc_country
Query submitted at: 2018-10-2816:21:13 (Coordinator: http://cdh-slave01:25000)
Query progress can be monitored at: http://cdh-slave01:25000/query_plan?query_id=fb4e09977145f367:3bdfe4d600000000
+---------------+----------+
| cc_country | count(*) |
+---------------+----------+
| United States | 30 |
+---------------+----------+
Fetched 1 row(s) in 4.71sMore information
For more information about OSS support in Hadoop, see Hadoop support for OSS.