E-MapReduce:What do I do if the Reached timeout error occurs when I use JindoSDK to access OSS?

Timeout message

[ErrorCode] : 25201 , [ErrorType]: OSS Op Error. [ErrorMessage]: [E1008]Reached timeout=30000ms

Methods to identify the cause

rename appears in the error message of the exception stack

• Sample error message

  java.io.IOException: rename src:oss://bucket.oss-cn-xxxx-internal.aliyuncs.com/user/hive/warehouse/tmp/hive/xxxx/c185ce78-f843-4104-8dca-f5b96fc9****/hive_xxxx_00-07-51_265_5593904247532586093-47957/_tmp.-mr-10006, dst:oss://bucket.oss-cn-xxxx-internal.aliyuncs.com/user/hive/warehouse/tmp/hive/xxx/c185ce78-f843-4104-8dca-f5b96fc9****/hive_xxxx-07-21_00-07-51_265_5593904247532586093-47957/_tmp.-mr-10006.moved
  ,java.io.IOException: [ErrorCode] : 25201 , [ErrorType]: OSS Op Error. [ErrorMessage]: [E1008]Reached timeout=30000ms @100.118.xx.xx:80 ERROR_CODE : 1008

• Cause

  The Rename operation of JindoSDK is implemented based on the CopyObject operation of OSS. The error occurs because OSS fastcopy optimization is not complete for the CopyObject operation. As a result, the Rename operation of JindoSDK times out.

• Solution

  You can contact technical support for OSS to confirm the configuration requirements for bucket.oss-cn-xxx-internal.aliyuncs.com and troubleshoot the cause of the internal fastcopy optimization failure.

  You can perform fastcopy optimization only on most common objects whose storage class is Standard. If the storage class of your OSS objects is not Standard or the OSS objects are of special types, fastcopy optimization may fail. As a result, the rename operation may time out. We recommend that you use JindoSDK to upload only common objects to OSS and use the Standard storage class. Fastcopy optimization may fail on the following objects:

  • Objects whose storage class is not Standard, such as objects whose storage class is IA, Archive, or Cold Archive

  • Objects of the AppendableObject type in OSS

  • Small objects that are uploaded by calling an operation other than MultiPart

  • Objects that use features such as server-side encryption

  • Special objects, such as objects of the LINK, SYMLINK, and DELETEMARKER types

InputStream or Read appears in the error message of the exception stack

• Sample error message

  Read from oss://xxxx with error message:  [HostId]: oss-cn-zhangjiakou-internal.aliyuncs.com [ErrorMessage]: [E1008]Reached timeout=30000ms

• Cause and solution

  • The SeekRead operation of JindoData 4.0.0 has a defect that can cause the Reached timeout error. If you use JindoData 4.0.0, upgrade JindoData to 4.3.X or a later version. For more information, see Upgrade JindoSDK in EMR clusters in the new EMR console.

  • OSS bandwidth throttling is triggered. Contact OSS technical support to identify the reason for the throttling.

OutputStream, write, or Close appears in the error message of the exception stack

• Sample error message

  com.aliyun.emr.fs.oss.commit.magic.JindoOssMagicOutputStream.write(JindoOssMagicOutputStream.java:146)
          ... 14 more
  Caused by: java.io.IOException: ErrorCode : 25201 , ErrorMsg: OSS Op Error.  [HostId]: oss-cn-beijing-internal.aliyuncs.com [ErrorMessage]: [E1008]Reached timeout=30000ms @100.118.xx.xx:80 ERROR_CODE : 1008
          at com.alibaba.jboot.JbootFuture.get(JbootFuture.java:179)
          at com.alibaba.jboot.JbootOssWriter.write(JbootOssWriter.java:85)

  Or

  Caused by: java.io.IOException: Close stream oss://xxx error java.io.IOException: [ErrorCode] : 25201 , [ErrorType]: OSS Op Error.  [HostId]: xxx [ErrorMessage]: [E1008]Reached timeout=30000ms @xxx ERROR_CODE : 1008

• Cause and solution

  This error usually occurs because OSS bandwidth throttling is triggered. Contact OSS technical support to identify the reason for the throttling.

getFileStatus appears in the error message of the exception stack

• Cause

  Versioning is enabled for the OSS bucket.

• Solution

  1. Confirm whether versioning is or was ever enabled for the bucket. In the OSS Management Console, go to the target bucket. In the left-side navigation pane, choose Data Security and click Versioning to view the versioning status. If the status is Enabled or Suspended, it indicates that versioning has been activated for the bucket.

  2. Contact OSS technical support to check whether more than 100,000 delete markers exist in the bucket or under the specified path. If so, clear these markers.

rm command

• rm command with skipTrash

  The rm command that contains the skipTrash option is run by performing two steps. The getFileStatus operation is called, and then the delete operation is called. If a timeout error occurs when you run the rm command that contains the skipTrash option, the timeout error occurs when the getFileStatus operation is called. To fix the timeout error, follow the instructions that are described in getFileStatus appears in the error message of the exception stack.

• rm command without skipTrash

  The rm command that does not contain the skipTrash option is run by performing two steps. The getFileStatus operation is called, and then the Rename operation is called. If a timeout error occurs when you run the rm command that does not contain the skipTrash option, follow the instructions that are described in getFileStatus appears in the error message of the exception stack and rename appears in the error message of the exception stack to fix the error.

ls command

The Is command is run by performing two steps. The getFileStatus operation is called, and then the listDirectory operation is called. If a timeout error occurs when you run the Is command, the timeout error occurs when the getFileStatus operation is called. To fix the timeout error, follow the instructions that are described in getFileStatus appears in the error message of the exception stack.