This topic describes the causes of and solutions to common errors that may occur when you use ossimport.

Introduction

All ossimport commands mentioned in this topic are shortened. The complete form must be used in actual scenarios.

  • Add console.bat to Windows commands. For example, change submit to console.bat submit.
  • Add bash console.sh to Linux commands. For example, change submit to bash console.sh submit.

Common problems about migration failures

If a migration job fails, we recommend that you view the migration failure log to identify the cause. After you resolve the issues, you can run the retry command to migrate the files again. The path of the migration failure log is master/jobs/${JobName}/failed_tasks/${TaskName}/audit.log.

The job state is displayed as failed when you run the stat command.

Solution: Run the stat command to view the job state. If JobState is failed, the migration job fails. After the migration job is complete, run the retry command to migrate the files again.

Some files fail to be migrated and retrying the migration also fails.

Solution:

  1. View the list of failed files in master/jobs/${JobName}/failed_tasks/${TaskName}/error.list to obtain their relative paths.
  2. Check whether you are authorized to access these files, whether these files are deleted, whether these files are symbolic links, and whether the file names are garbled.
  3. After you resolve the preceding issues, run the retry command to migrate the files again.

The bucket you are attempting to access must be addressed using the specified endpoint. is displayed in the migration failure log.

Exception:com.aliyun.oss.OSSException: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
<Error>
  <Code>AccessDenied</Code>
  <Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message>
  <RequestId>56EA98DE815804**21B23EE6</RequestId>
  <HostId>my-oss-bucket.oss-cn-qingdao.aliyuncs.com</HostId>
  <Bucket>my-oss-bucket</Bucket>
  <Endpoint>oss-cn-hangzhou.aliyuncs.com</Endpoint>
</Error>
  • Cause: The value of srcDomain or destDomain is invalid.
  • Solution: Enter valid endpoints based on the endpoints listed in Regions and endpoints.

The request signature we calculated does not match the signature you provided. is displayed in the migration failure log.

Exception:com.aliyun.oss.OSSException: The request signature we calculated does not match the signature you provided. Check your key and signing method.
[ErrorCode]: SignatureDoesNotMatch
[RequestId]: xxxxxxx
[HostId]: xxx.oss-cn-shanghai.aliyuncs.com
  • Cause: The values of destAccessKey and destSecretKey are invalid.
  • Solution: Enter a valid AccessKey pair.

The bucket name "xxx/xx" is invalid. is displayed in the migration failure log.

java.lang.IllegalArgumentException: The bucket name "xxx/xx" is invalid. A bucket name must: 1) be comprised of lower-case characters, numbers or dash(-); 2) start with lower case or numbers; 3) be between 3-63 characters long.
  • Cause: The value of destBucket is invalid.
  • Solution: Enter a valid bucket name. For more information about the naming conventions, see Bucket naming conventions.

Connect to xxx.oss-cn-beijing-internal.aliyuncs.com:80 timed out. is displayed in the migration failure log.

Unable to execute HTTP request: Connect to xxx.oss-cn-beijing-internal.aliyuncs.com:80 timed out
[ErrorCode]: ConnectionTimeout
[RequestId]: Unknown
  • Cause: The connection timeout error is returned because the configuration file uses the internal endpoint of OSS, but the device that was used to migrate data is not an ECS instance or is not an ECS instance that is located in the same region as OSS instances. The internal endpoint allows access only from ECS instances that are located in the same region as OSS instances.
  • Solution:
    • Set the domain name to a public endpoint in the configuration file. Delete the migration job and submit the job again.
    • Use an ECS instance that is located in the same region as OSS instances to perform the migration job.

The specified bucket is not valid. is displayed in the migration failure log.

com.aliyun.oss.OSSException: The specified bucket is not valid.
[ErrorCode]: InvalidBucketName
[RequestId]: 57906B4DD0EBAB0FF553D661
[HostId]: you-bucket.you-bucketoss-cn-hangzhou-internal.aliyuncs.com
  • Cause: The destDomian parameter is incorrectly configured.
  • Solution: The value of destDomain must be the endpoint of the region where the bucket is located and not the second-level domain name that contains the bucket name. For example, if the bucket is located in the China (Beijing) region, enter oss-cn-beijing.aliyuncs.com. For more information, see Configuration file examples.

Unable to execute HTTP request: The Difference between ... is too large. is displayed in the migration failure log.

Unable to execute HTTP request: The Difference between the request time and the current time is too large.
[ErrorCode]: RequestTimeTooSkewed
[RequestId]: xxxxxxx
  • Cause
    • The difference between the system time of the machine or device that sends the request and the time that is adopted by OSS exceeds 15 minutes. This is the most common situation.
    • A large number of requests are sent at the same time, which results in a high CPU utilization and slow concurrent uploads.
  • Solution
    • Set the system time of the machine or device that sends the request to the time that is adopted by OSS.
    • If the number of requests sent at the same time is too large, you can reduce the workerTaskThreadNum parameter of the sys.properties file.

No route to host. is displayed in the migration failure log.

  • Cause: This error may be returned when the network connection fails because of local firewalls or iptables.
  • Solution: Run the ping command to check whether the network connection between the source and destination instances is normal.
    • If the network connection is normal, you can check if restrictions are configured by the computer firewalls and local firewalls. You can also try disabling the firewall.
    • If the network connection is abnormal, troubleshoot the problem and try again.

Unknown http list file format. is displayed in the migration failure log when you migrate files over HTTP.

  • Cause: This error is returned when the format of the specified HTTP list file or its content is invalid.
  • Solution:
    • You can convert the format of files that are copied from a different operating system by running commands such as mac2unix and doc2unix in Linux and tools such as notepad in Windows.
    • If the format of the HTTP list file is invalid, change the format to the valid format. For more information about the format of the HTTP list file , see Configuration files.

The object key "/xxxxx.jpg" is invalid is displayed in the migration failure log.

Exception:java.lang.IllegalArgumentException: The object key "/xxxxx.jpg" is invalid. An object name should be between 1 - 1023 bytes long when encoded as UTF-8 and cannot contain LF or CR os unsupported chars in XML1.0, and cannot begin with "/" or "\".
  • Cause: The srcPrefix or destPrefix parameter is incorrectly configured.
  • Solution
    • Check whether srcPrefix is a directory. If you set the value to a directory, the value must end with a forward slash (/).
    • Check whether the value of destPrefix starts with a forward slash (/) or a backslash (\). Delete the forward slash (/) or backslash (\).

Common problems when running a migration job

If a problem occurs during data migration, you can view the running log.

  • Standalone deployment: To view the running information about jobs, view logs/ossimport2.log.
  • Distributed deployment: To view the running information about jobs, view logs/import.log.

UnsupportedClassVersionError. is returned when a command is run.

Exception in thread "main" java.lang.UnsupportedClassVersionError: com/aliyun/ossimport2/OSSImport2 : Unsupported major.minor version 51.0
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
        at com.simontuffs.onejar.JarClassLoader.defineClass(JarClassLoader.java:693)
        at com.simontuffs.onejar.JarClassLoader.findClass(JarClassLoader.java:599)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at com.simontuffs.onejar.Boot.run(Boot.java:300)
        at com.simontuffs.onejar.Boot.main(Boot.java:159)
  • Cause: The Java version is earlier than the required version.
  • Solution: Upgrade the Java version to 1.7 or 1.8.

InvocationTargetException. is returned when you run the submit command to submit a job.

Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at com.simontuffs.onejar.Boot.run(Boot.java:306)
        at com.simontuffs.onejar.Boot.main(Boot.java:159)
Caused by: java.lang.NullPointerException
        at com.aliyun.ossimport2.config.JobConfig.load(JobConfig.java:44)
        at com.aliyun.ossimport2.OSSImport2.doSubmitJob(OSSImport2.java:289)
        at com.aliyun.ossimport2.OSSImport2.main(OSSImport2.java:120)
        ... 6 more
  • Cause: The system detects that some configuration items are deleted or commented out from the configuration file.
  • Solution: Restore the configuration items that are deleted or commented out from the configuration file. Leave configurations items unspecified after the equal sign (=) and delete them. For more information, see Configuration file examples.

com.aliyun.oss.ClientException: Unknown. is displayed in the running log.

com.aliyun.oss.ClientException: Unknown
[ErrorCode]: NonRepeatableRequest
[RequestId]: Cannot retry request with a non-repeatable request entity.  The cause lists the reason the original request failed.
  • Cause: The com.aliyun.oss.ClientException: Unknown. or SocketTimeoutException error is returned when the server consumes full bandwidth.
  • Solution: ossimport automatically retries the migration job. If the job still fails, run the retry command to migrate the files again.

too many open files. is displayed in the running log in Linux.

Solution: Run the ulimit -n command to view the limit of handles in Linux.

  • If the handle limit is smaller than 10,000, run the ulimit -n 65536 command to increase the value and restart the process.
  • If the handle limit is greater than 10,000, run the sudo lsof -n command to check which processes have enabled the handles. You can evaluate these processes and stop the unnecessary processes to release the handles.

The migration job exits in seconds after starting in Windows.

  • Cause
    • Java is not installed or a Java version earlier than 1.7 is used.
    • Configuration file error.
  • Solution

After you submit a job by running the submit command, no jobs is running or finished. is displayed when you run the stat command to view the job status.

bash console.sh stat
[WARN]   List files dir not exist : /home/<user>/ossimport/workdir/master/jobs/
no jobs is running or finished.
  • Cause: The service is not started when you run the job. You can run the stat command to view the running status of the task only after the job is submitted and the service is started.
  • Solution
    • Run the start command to start the service.
    • If the service is started and the job has just been submitted, Master must scan the file list first. Because the job has not been generated and distributed at this time, this error is displayed in the logs.
    • If the error still occurs after the service has been started and the job has been submitted for a period of time, check whether the process exits unexpectedly after it starts. To view the running information about jobs for standalone deployment, view logs/ossimport2.log. To view the running information about jobs for distributed deployment, view logs/ossimport2.log. Identify the cause of the error and fix the error, and then start the service process.

scanFinished: false. is displayed when you run the stat command to check the job status.

Solution: Check whether the total number of tasks is increasing.

  • If the total number of tasks is increasing, the file list of the migration job is listing new files. It is normal that this error is returned in this case.
  • If the incremental data migration mode is enabled, scanFinished is not true, and the number of tasks does not change, ossimport scans the file list at specified intervals to check for new or modified files.
  • If the incremental data migration mode is disabled and the number of tasks does not increase, check the running log for exceptions. To view the running information about jobs for standalone deployment, view logs/ossimport2.log. To view the running information about jobs for distributed deployment, view logs/ossimport2.log. Identify the cause of the error and fix the error, and then start the service process.

The process in Linux is abnormal, but no exception is reported in logs.

  • Cause: If the system has less than 2 GB of available memory, process exception may occur due to insufficient memory.
  • Solution: Check whether the dmesg log contains records about process exceptions because of insufficient memory.

Operations to perform when the service is restarted after a process exception occurs.

Solution: All submitted jobs have breakpoint records. If you do not clear the original jobs that have the same name by running the clean command, you can directly run the start command to start the service without submitting the job again.

How do I upload files whose names are garbled to OSS in Linux?

Solution:
  1. Check the encoding format of the garbled names.
  2. Run the export LANG="<your file name encode>" command to parse these encoded names.
  3. Run the clean command to clear the original job, and then run the submit command to submit the job again.

java.nio.file.AccessDeniedException. is returned when the service is started.

  • Cause: You are not authorized to access the configuration file.
  • Solution
    • Set the access permission of the configuration file to public read.
    • Log on to Linux as an administrator to start the service.

The value of Task Counts is 0, but JobState is displayed as succeed.

[2015-12-28 16:12:35]   [INFO]  JobName:dir_data
[2015-12-28 16:12:35]   [INFO]  Pending Task Count:0
[2015-12-28 16:12:35]   [INFO]  Dispatched Task Count:0
[2015-12-28 16:12:35]   [INFO]  Succeed Task Count:0
[2015-12-28 16:12:35]   [INFO]  Failed Task Count:0
[2015-12-28 16:12:35]   [INFO]  Is Scan Finished:true
[2015-12-28 16:12:35]   [INFO]  JobState:SUCCEED
  • Cause
    • Files cannot be listed because the value of srcPrefix is invalid.
    • Directories specified by srcPrefix do not contain files. Directories cannot be uploaded to OSS because the concept of directories is simulated by OSS.
  • Solution: Specify a valid value for the srcPrefix parameter and make sure there are available files in the directory specified by srcPrefix.

InvocationTargetException. is returned when you submit the job.

submit job:/disk2/ossimport2/local_job.cfg
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at com.simontuffs.onejar.Boot.run(Boot.java:306)
        at com.simontuffs.onejar.Boot.main(Boot.java:159)
Caused by: java.lang.NullPointerException
        at com.aliyun.ossimport2.OSSImport2.doSubmitJob(OSSImport2.java:289)
        at com.aliyun.ossimport2.OSSImport2.main(OSSImport2.java:120)
        ... 6 more
  • Cause: This error is returned because the configuration file or configuration file path is invalid.
  • Solution
    • Configure the workingDir configuration item in the conf/sys.properties file properly.
    • Check the configuration file path.

Source files for synchronization do not exist.

Cause: When Master performs a migration job, Master first lists the files and then migrates data based on the list. If files are deleted from the source after listing, the files cannot be migrated. In this case, Master skips the deleted source files and reports such files in the error list.

The job configuration file is valid, but the job running status is inconsistent with the job configuration file during the migration.

  • Cause: If a migration job has been submitted, modifications to the job configuration file do not take effect after the job is suspended.
  • Solution: Clear the previous job by running the clean command. After the job configuration file is modified, submit the job again.

The file is transferred normally, but the error NullPointerException. is returned.

  • Cause: The file statistics feature is added to ossimport 2.3.5. This feature requires cpt files to be loaded. However, the cpt loading format of HTTP jobs is different from that of other sources. As a result, program mismatch occurs.
  • Solution
    • Roll ossimport to version 2.3.4.
    • Ignore the error. This issue is only a display error that does not affect the migration job.

Common problems about migrating data from UPYUN Storage Service (USS) to OSS

The number of jobs is always 0.

Solution: View the job running log.

[2016-07-21 10:21:46] [INFO] [name=YoupaiList, totalRequest=1729925, avgLatency=38,
          recentLatency=300000]
  • If the value of recentLatency in the running log is smaller than or equal to 30,000, the files are listed properly. It can take more than 30 seconds (timeout) for files to be listed on USS in most cases. The number of returned files varies based on the number of files in the list within 30 seconds. Wait until the listing is complete.
  • If the value of recentLatency is small, this error occurs because the account password is invalid. If a USS SDK has an error, only null is returned, and no error result is returned. In this case, capture packets to obtain the error code returned by USS for troubleshooting.

How to specify srcAccessKey and srcSecretKey?

Solution: Enter the operator account and password of USS.

HTTP status code 429 keeps appearing.

  • Cause: Bandwidth throttling is configured for the access interval of the USS SDK.
  • Solution: Contact USS to remove the bandwidth throttling.

After the migration job is complete, the data volume displayed in the OSS console is smaller than the source data volume.

  • Description: The bucket size does not change in the OSS console after all migration jobs are complete, but the data sizes calculated by running the du command vary greatly in Linux.
  • Cause
    • Bucket sizes in the OSS console are updated with a delay of one to two hours. Check whether the bucket size changes after one to two hours.
    • The du command in Linux calculates the block size, which is larger than the file size. We recommend that you run the ls -lR <absolute directory path> | grep "\-rw" | awk '{sum+=$5}END{print sum}' command to calculate the actual size of the local directory.

The error message such as unknown command "java" or unknown command "nohup" is returned when you run commands in Linux.

  • Cause: The command that you want to run is not installed.
  • Solution: Run the yum, apt-get, or zypper command to install the corresponding command based on the operating system.

Can I use srcPrefix in the configuration file to specify files?

No. You can only use srcPrefix to specify directories or prefixes.

Can I configure proxies for ossimport?

No.

Why am I charged for migrating data within OSS?

If you migrate data by using an internal endpoint, you are charged based on the number of requests. No traffic fee is charged.

Are the files that are deleted from the local directory deleted from OSS if the incremental data migration mode is enabled?

No. Local delete operations are not synchronized to OSS.

Why are new local files not synchronized to OSS when the incremental data migration mode is enabled?

In incremental data migration mode, the last modification time of a file is used to determine whether the file is new. The Linux mv command and Windows commands such as cp, mv, and rsync with -t or -a parameters do not modify the last modification time of files. Files modified by running these commands are not scanned for and therefore not synchronized to OSS during migration.

Can file permissions be migrated when files are migrated to OSS?

No. After the migration is complete, you can run the set-meta command of ossutil to modify the permissions. For more information, see set-meta (manage object metadata).