E-MapReduce:Upgrade and roll back JindoSDK on an EMR cluster (x86)

Step 1: Prepare the package and upgrade scripts

1. Log on to the master node of the EMR cluster. For more information, see Log on to a cluster.

2. Download the patch package to the home directory of emr-user and decompress the package.

   su - emr-user
   cd /home/emr-user/
   wget https://jindodata-binary.oss-cn-shanghai.aliyuncs.com/resources/emr-taihao/jindosdk-patches.tar.gz
   tar zxf jindosdk-patches.tar.gz

3. Download the JindoSDK package jindosdk-{VERSION}.tar.gz and place it in the decompressed directory.

   This example upgrades JindoSDK in the cluster to version 6.8.2.

   cd jindosdk-patches
   
   wget https://jindodata-binary.oss-cn-shanghai.aliyuncs.com/release/6.8.2/jindosdk-6.8.2-linux.tar.gz
   
   ls -l

   The jindosdk-patches directory contains the following files:

   -rwxrwxr-x 1 emr-user emr-user      2439 May 01 00:00 apply_all.sh
   -rwxrwxr-x 1 emr-user emr-user      7315 May 01 00:00 apply.sh
   -rw-rw-r-- 1 emr-user emr-user        40 May 01 00:00 hosts
   -rw-r----- 1 emr-user emr-user xxxxxxxxx May 01 00:00 jindosdk-6.8.2-linux.tar.gz
   -rwxrwxr-x 1 emr-user emr-user      1112 May 01 00:00 revert_all.sh
   -rwxrwxr-x 1 emr-user emr-user      2042 May 01 00:00 revert.sh

Step 1: Create a bootstrap upgrade package

1. Run the following commands to download jindosdk-patches.tar.gz, jindosdk-{VERSION}-{PLATFORM}.tar.gz, and bootstrap_jindosdk.sh.

   This example upgrades JindoSDK in the cluster to version 6.8.2.

   mkdir jindo-patch
   
   cd jindo-patch
   
   wget https://jindodata-binary.oss-cn-shanghai.aliyuncs.com/resources/emr-taihao/jindosdk-patches.tar.gz
   
   wget https://jindodata-binary.oss-cn-shanghai.aliyuncs.com/release/6.8.2/jindosdk-6.8.2-linux.tar.gz
   
   wget https://jindodata-binary.oss-cn-shanghai.aliyuncs.com/resources/emr-taihao/bootstrap_jindosdk.sh
   
   ls -l

   The following output is returned:

   -rw-r----- 1 hadoop hadoop      xxxx May 01 00:00 bootstrap_jindosdk.sh
   -rw-r----- 1 hadoop hadoop xxxxxxxxx May 01 00:00 jindosdk-6.8.2-linux.tar.gz
   -rw-r----- 1 hadoop hadoop      xxxx May 01 00:00 jindosdk-patches.tar.gz

2. Run the following command to create the upgrade package.

   bash bootstrap_jindosdk.sh -gen $NEW_JINDOSDK_VERSION  # Run the bootstrap_jindosdk.sh script with the specified $NEW_JINDOSDK_VERSION to upgrade JindoSDK to that version.

   Note

   • To scale out an existing cluster, use the -gen option to generate a lightweight upgrade package.

   • To create a new cluster, use the -gen-full option to generate an upgrade package that contains all the required content.

   For example, to upgrade JindoSDK to version 6.8.2:

   bash bootstrap_jindosdk.sh -gen 6.8.2

   After the upgrade package is created, the following message is displayed.

   Generated patch at /home/emr-user/jindo-patch/jindosdk-bootstrap-patches.tar.gz

   When the build is complete, the patch package jindosdk-bootstrap-patches.tar.gz is generated.

Step 2: Upload the bootstrap package

Upload the patch package and the bootstrap script to OSS. You can use Hadoop commands from within the EMR cluster, or use tools such as the Alibaba Cloud OSS console, ossutil, or OSS Browser.

For example, the OSS paths are oss://<bucket-name>/path/to/bootstrap_jindosdk.sh and oss://<bucket-name>/path/to/jindosdk-bootstrap-patches.tar.gz.

hadoop dfs -mkdir -p oss://<bucket-name>/path/to/patch/

cd /home/hadoop/patch/
hadoop dfs -put jindosdk-bootstrap-patches.tar.gz oss://<bucket-name>/path/to/patch/
hadoop dfs -put bootstrap_jindosdk.sh oss://<bucket-name>/path/to/patch/

hadoop dfs -ls oss://<bucket-name>/path/to/patch/

The following output is returned:

Found 2 items
-rw-rw-rw-   1       2634 2022-05-13 14:07 oss://<bucket-name>/.../bootstrap_jindosdk.sh
-rw-rw-rw-   1  597342992 2022-05-13 13:41 oss://<bucket-name>/.../jindosdk-bootstrap-patches.tar.gz

Step 3: Add a bootstrap action

Add a bootstrap action in the EMR console. For more information, see Manage bootstrap actions.

Configure the bootstrap action with the settings in the following table.

Step 4: Handle special nodes

1. Upgrade Gateway nodes created using the EMR CLI.

   • If a Gateway node was created in the EMR console, it is already covered by the preceding steps.

   • If you created a Gateway node using the EMR CLI, it exists independently and you must run the upgrade script on it manually. In addition, you must upgrade elastic nodes in advance when you initialize them.

2. Replace the JindoSDK for services such as Trino, Presto, and Impala.

   For EMR clusters running versions earlier than EMR v3.53.0 or v5.19.0, the JindoSDK for services such as Trino, Presto, and Impala does not upgrade automatically. You must manually replace the JindoSDK JAR file in the plugins path of these services with the target version and restart the services to apply the changes.

Step 5: Restart services

Restart the relevant services to ensure they load the latest fixes.

• If you are creating a new cluster, restart services such as Hive, Presto, Impala, Flink, Ranger, Spark, and Zeppelin.

• If you are scaling out by adding new nodes, restart the services on those nodes.

Step 1: Prepare the rollback scripts

1. Log on to the master node of the EMR cluster. For more information, see Log on to a cluster.

2. Download the patch package to the home directory of emr-user and decompress the package.

   su - emr-user
   cd /home/emr-user/
   wget https://jindodata-binary.oss-cn-shanghai.aliyuncs.com/resources/emr-taihao/jindosdk-patches.tar.gz
   tar zxf jindosdk-patches.tar.gz
   cd jindosdk-patches
   ls -l

   The following output is returned:

   -rwxrwxr-x 1 emr-user emr-user      2439 May 01 00:00 apply_all.sh
   -rwxrwxr-x 1 emr-user emr-user      7315 May 01 00:00 apply.sh
   -rw-rw-r-- 1 emr-user emr-user        40 May 01 00:00 hosts
   -rwxrwxr-x 1 emr-user emr-user      1112 May 01 00:00 revert_all.sh
   -rwxrwxr-x 1 emr-user emr-user      2042 May 01 00:00 revert.sh

Step 2: Configure node information

• Manually configure node information

  1. Edit the hosts file in the patch package.

     vim hosts

  2. Add the hostnames of all nodes in the cluster, such as master-1-1 or core-1-1, with one hostname per line.

     For example, the hosts file contains the following content:

     master-1-1
     core-1-1
     core-1-2

• Automatically populate node information

  You can also run the following command to retrieve all node information. If the command fails to populate the hosts file, you must add the node information manually.

  cat  /usr/local/taihao-executor-all/data/cache/.cluster_context | jq --raw-output '.nodes[].hostname.alias[]' > hosts

Step 3: Run the rollback

Run the following command to roll back all changes.

./revert_all.sh

The script is finished when the output contains ### DONE.

>>> updating ...  master-1-1
>>> updating ...  core-1-1
>>> updating ...  core-1-2
### DONE

Step 4: Verify the rollback

ls -l /opt/apps/JINDOSDK/jindosdk-current/lib

The following example shows the output after rolling back to version 6.2.0.

-rw-r--r-- 1 emr-user emr-user  1253740 Apr 24 17:40 jindo-core-6.2.0.jar
-rw-r--r-- 1 emr-user emr-user 13110547 Apr 24 17:40 jindo-core-linux-el7-aarch64-6.2.0.jar
-rw-r--r-- 1 emr-user emr-user  4432227 Apr 24 17:40 jindo-sdk-6.2.0.jar
drwxr-xr-x 2 emr-user emr-user     4096 Apr 24 17:40 native

Step 5: Handle special nodes

1. Upgrade Gateway nodes created using the EMR CLI.

   • If a Gateway node was created in the EMR console, it is already covered by the preceding steps.

   • If you created a Gateway node using the EMR CLI, it exists independently and you must run the upgrade script on it manually. In addition, you must upgrade elastic nodes in advance when you initialize them.

2. Replace the JindoSDK for services such as Trino, Presto, and Impala.

   For EMR clusters running versions earlier than EMR v3.53.0 or v5.19.0, the JindoSDK for services such as Trino, Presto, and Impala does not roll back automatically. Manually replace the JindoSDK JAR file in the plugins path of these services with the default version, and restart the services to apply the changes.

Step 6: Restart services

To apply the rollback, restart services such as Hive, Presto, Impala, Flink, Ranger, Spark, and Zeppelin.

For example, to restart the Hive service, go to the Hive service page of your EMR cluster and choose More > Restart in the upper-right corner.