This topic describes the causes of and solutions to the Reached timeout error that may occur when you use JindoSDK to access Object Storage Service (OSS).
Timeout message
[ErrorCode] : 25201 , [ErrorType]: OSS Op Error. [ErrorMessage]: [E1008]Reached timeout=30000ms
Methods to identify the cause
You cannot identify the cause based on only the timeout message. If the OSS server does not return a response within the specified timeout period, you need to check the context logs or the exception stack.
Check the context logs based on the exception stack:
If you run the following commands to access OSS, the Reached timeout error may occur:
rename appears in the error message of the exception stack
Sample error message
java.io.IOException: rename src:oss://bucket.oss-cn-xxxx-internal.aliyuncs.com/user/hive/warehouse/tmp/hive/xxxx/c185ce78-f843-4104-8dca-f5b96fc9****/hive_xxxx_00-07-51_265_5593904247532586093-47957/_tmp.-mr-10006, dst:oss://bucket.oss-cn-xxxx-internal.aliyuncs.com/user/hive/warehouse/tmp/hive/xxx/c185ce78-f843-4104-8dca-f5b96fc9****/hive_xxxx-07-21_00-07-51_265_5593904247532586093-47957/_tmp.-mr-10006.moved ,java.io.IOException: [ErrorCode] : 25201 , [ErrorType]: OSS Op Error. [ErrorMessage]: [E1008]Reached timeout=30000ms @100.118.xx.xx:80 ERROR_CODE : 1008Cause
The Rename operation of JindoSDK is implemented based on the CopyObject operation of OSS. The error occurs because OSS fastcopy optimization is not complete for the CopyObject operation. As a result, the Rename operation of JindoSDK times out.
Solution
Contact technical support of OSS to confirm the setting requirements on
bucket.oss-cn-xxx-internal.aliyuncs.com. Then, check the causes for the failure of OSS fastcopy optimization.You can perform fastcopy optimization only on most common objects whose storage class is Standard. If the storage class of your OSS objects is not Standard or the OSS objects are of special types, fastcopy optimization may fail. As a result, the rename operation may time out. We recommend that you use JindoSDK to upload only common objects to OSS and use the Standard storage class. Fastcopy optimization may fail on the following objects:
Objects whose storage class is not Standard, such as objects whose storage class is IA, Archive, or Cold Archive
Objects of the AppendableObject type in OSS
Small objects that are uploaded by calling an operation other than MultiPart
Objects that use features such as server-side encryption
Special objects, such as objects of the LINK, SYMLINK, and DELETEMARKER types
InputStream or Read appears in the error message of the exception stack
Sample error message
Read from oss://xxxx with error message: [HostId]: oss-cn-zhangjiakou-internal.aliyuncs.com [ErrorMessage]: [E1008]Reached timeout=30000msCause and solution
The SeekRead operation of JindoData 4.0.0 has a defect that can cause the Reached timeout error. If you use JindoData 4.0.0, upgrade JindoData to 4.3.X or a later version. For more information, see Upgrade JindoSDK in EMR clusters in the new EMR console.
Bandwidth throttling of OSS is triggered. In this case, contact technical support of OSS to check the reason why bandwidth throttling is triggered.
OutputStream, write, or Close appears in the error message of the exception stack
Sample error message
com.aliyun.emr.fs.oss.commit.magic.JindoOssMagicOutputStream.write(JindoOssMagicOutputStream.java:146) ... 14 more Caused by: java.io.IOException: ErrorCode : 25201 , ErrorMsg: OSS Op Error. [HostId]: oss-cn-beijing-internal.aliyuncs.com [ErrorMessage]: [E1008]Reached timeout=30000ms @100.118.xx.xx:80 ERROR_CODE : 1008 at com.alibaba.jboot.JbootFuture.get(JbootFuture.java:179) at com.alibaba.jboot.JbootOssWriter.write(JbootOssWriter.java:85)Or
Caused by: java.io.IOException: Close stream oss://xxx error java.io.IOException: [ErrorCode] : 25201 , [ErrorType]: OSS Op Error. [HostId]: xxx [ErrorMessage]: [E1008]Reached timeout=30000ms @xxx ERROR_CODE : 1008Cause and solution
In most cases, this error is caused because bandwidth throttling of OSS is triggered. Contact OSS technical support to check the reason why bandwidth throttling is triggered.
getFileStatus appears in the error message of the exception stack
Cause
Versioning is enabled for the OSS bucket.
Solution
Check whether versioning is enabled or whether you previously enabled versioning.
Contact technical support of OSS to check whether more than 100,000 delete markers exist in the OSS bucket or path. If more than 100,000 delete markers exist, remove the additional delete markers.
rm command
rm command with skipTrash
The rm command that contains the skipTrash option is run by performing two steps. The getFileStatus operation is called, and then the delete operation is called. If a timeout error occurs when you run the rm command that contains the skipTrash option, the timeout error occurs when the getFileStatus operation is called. To fix the timeout error, follow the instructions that are described in getFileStatus appears in the error message of the exception stack.
rm command without skipTrash
The rm command that does not contain the skipTrash option is run by performing two steps. The getFileStatus operation is called, and then the Rename operation is called. If a timeout error occurs when you run the rm command that does not contain the skipTrash option, follow the instructions that are described in getFileStatus appears in the error message of the exception stack and rename appears in the error message of the exception stack to fix the error.
skipTrash is an option of the rm command. For example, you can run the hadoop fs -rm -skipTrash oss://bucket/path command to delete OSS objects. Deleted objects are not moved to the recycle bin.
ls command
The Is command is run by performing two steps. The getFileStatus operation is called, and then the listDirectory operation is called. If a timeout error occurs when you run the Is command, the timeout error occurs when the getFileStatus operation is called. To fix the timeout error, follow the instructions that are described in getFileStatus appears in the error message of the exception stack.