All Products
Search
Document Center

Data Online Migration:FAQ

Last Updated:Mar 18, 2024

This topic provides answers to some frequently asked questions and common errors that may occur when you use ossimport.

Notes

All ossimport commands mentioned in this topic are shortened. The complete form of the commands must be used in actual scenarios.

  • Add console.bat for Windows commands. For example, change submit to console.bat submit.

  • Add bash console.sh for Linux commands. For example, change submit to sudo bash console.sh submit.

How do I check whether data is migrated by using ossimport?

After full data migration is complete, you can run the following command to view the task state.

  • On Windows, run console.bat stat in Command Prompt.

  • On Linux, run sudo bash console.sh stat.

If the migration task is in the Succeed state, the migration is successful.

If you set isIncremental to true for the job, ossimport periodically scans the source directory at specified intervals, checks for new or modified files, and synchronizes incremental data to Object Storage Service (OSS). You can view the incremental objects in the corresponding OSS bucket.

Important

ossimport does not verify data after migration and therefore does not guarantee data consistency and integrity. After a migration task is complete, make sure to verify data consistency between the migration source and destination.

If you delete the source data without verifying data consistency between the source and destination, you are responsible for any losses and consequences that arise.

Common migration failures

If a migration job fails, we recommend that you view the migration failure log to identify the cause. After you resolve migration issues, you can run the retry command to migrate the files again. The path of the failure log file is master/jobs/${JobName}/failed_tasks/${TaskName}/audit.log.

The stat command returns the "failed" job state.

The following command is run to check the status of a job:

sudo bash console.sh stat

If the value of the JobState parameter is failed, the migration job failed.

状态

Solution: After the migration job is complete, run the retry command to retry the job.

Several files fail to be migrated, and retrying the migration also fails.

Solutions:

  1. Check the master/jobs/${JobName}/failed_tasks/${TaskName}/error.list file for the relative paths to the failed files.

  2. Check whether you are authorized to access the files, whether the files are deleted, whether the files are symbolic links, and whether the file names are garbled.

  3. After you resolve the preceding issues, run the retry command to migrate the files again.

The "The bucket you are attempting to access must be addressed using the specified endpoint." error message is contained in the migration failure log.

Exception:com.aliyun.oss.OSSException: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
<Error>
  <Code>AccessDenied</Code>
  <Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message>
  <RequestId>56EA98DE815804**21B23EE6</RequestId>
  <HostId>my-oss-bucket.oss-cn-qingdao.aliyuncs.com</HostId>
  <Bucket>my-oss-bucket</Bucket>
  <Endpoint>oss-cn-hangzhou.aliyuncs.com</Endpoint>
</Error>
  • Cause: The value of the srcDomain or destDomain parameter is invalid.

  • Solution: Specify a valid endpoint. For more information, see Regions and endpoints.

The "The request signature we calculated does not match the signature you provided." error message is contained in the migration failure log.

Exception:com.aliyun.oss.OSSException: The request signature we calculated does not match the signature you provided. Check your key and signing method.
[ErrorCode]: SignatureDoesNotMatch
[RequestId]: xxxxxxx
[HostId]: xxx.oss-cn-shanghai.aliyuncs.com
  • Cause: The value of the destAccessKey or destSecretKey parameter is invalid.

  • Solution: Enter a valid AccessKey pair.

The The bucket name "xxx/xx" is invalid. error message is contained in the migration failure log.

java.lang.IllegalArgumentException: The bucket name "xxx/xx" is invalid. A bucket name must: 1) be comprised of lower-case characters, numbers or dash(-); 2) start with lower case or numbers; 3) be between 3-63 characters long.
  • Cause: The value of the destBucket parameter is invalid.

  • Solution: Enter a valid bucket name. For more information about bucket naming, see Bucket naming conventions.

The "Connect to xxx.oss-cn-beijing-internal.aliyuncs.com:80 timed out." error message is contained in the migration failure log.

Unable to execute HTTP request: Connect to xxx.oss-cn-beijing-internal.aliyuncs.com:80 timed out
[ErrorCode]: ConnectionTimeout
[RequestId]: Unknown
  • Cause: The connection timeout error is returned because the configuration file uses the internal endpoint of OSS, but the device that is used to migrate data is not an Elastic Compute Service (ECS) instance or is not an ECS instance that is located in the same region as the OSS bucket. The internal endpoint of a bucket allows access only from ECS instances that are located in the same region as the OSS bucket.

  • Solutions:

    • Set the domain name to a public endpoint in the configuration file. Delete the migration job and resubmit the job.

    • Use an ECS instance that is located in the same region as the bucket to perform the migration job.

The "The specified bucket is not valid." error message is contained in the migration failure log.

com.aliyun.oss.OSSException: The specified bucket is not valid.
[ErrorCode]: InvalidBucketName
[RequestId]: 57906B4DD0EBAB0FF553D661
[HostId]: you-bucket.you-bucketoss-cn-hangzhou-internal.aliyuncs.com
  • Cause: The value of the destDomian parameter is invalid.

  • Solution: The value of the destDomain parameter must be the endpoint of the region in which the bucket is located and not a subdomain that contains the bucket name. For example, if the bucket is located in the China (Beijing) region, enter oss-cn-beijing.aliyuncs.com. For more information, see Configuration file examples.

The "Unable to execute HTTP request: The Difference between ... is too large." error message is contained in the migration failure log.

Unable to execute HTTP request: The Difference between the request time and the current time is too large.
[ErrorCode]: RequestTimeTooSkewed
[RequestId]: xxxxxxx
  • Causes:

    • In most cases, the difference between the system time on the on-premises machine or device and the time on the OSS server exceeds 15 minutes.

    • An excessively large number of requests are sent at the same time. This results in high CPU utilization and slow concurrent uploads.

  • Solutions:

    • Synchronize the local system time with the time on the OSS server.

    • If an excessively large number of requests are sent at the same time, set the workerTaskThreadNum parameter in the sys.properties file to a smaller value.

The "No route to host." error message is contained in the migration failure log.

  • Cause: This error may be returned when the network connection fails because of local firewalls or iptables.

  • Solution: Run the ping command to check the network connection between the source and destination of the migration.

    • If the network connection is normal, you can check whether restrictions are configured on the computer firewalls and local firewalls. You can also try to disable firewalls and then test network connectivity.

    • If the network connection is abnormal, troubleshoot the issue and try again.

The "Unknown http list file format." error message is contained in the migration failure log when an HTTP URL list is used to migrate data.

  • Cause: This error is returned when the format or the content of the specified HTTP list file is invalid.

  • Solutions:

    • If the files are copied from a different operating system, convert the files to a valid format for migration. For example, you can run mac2unix and doc2unix on Linux and use tools such as notepad on Windows to convert the file format.

    • If the format of the HTTP list file is invalid, change the format to a valid format. For more information about the formats of the HTTP list file, see Configuration files.

The The object key "/xxxxx.jpg" is invalid error message is contained in the migration failure log.

Exception:java.lang.IllegalArgumentException: The object key "/xxxxx.jpg" is invalid. An object name should be between 1 - 1023 bytes long when encoded as UTF-8 and cannot contain LF or CR os unsupported chars in XML1.0, and cannot begin with "/" or "\".
  • Cause: The value of the srcPrefix or destPrefix parameter is invalid.

  • Solutions:

    • Check whether the srcPrefix parameter specifies a directory. If yes, the value of the parameter must end with a forward slash (/).

    • Check whether the value of the destPrefix parameter starts with a forward slash (/) or a backslash (\). If yes, delete the forward slash (/) or backslash (\).

Common job execution issues

If an error occurs during data migration, you can view the job execution log.

  • Standalone deployment: The path of the job execution log file is logs/ossimport2.log.

  • Distributed deployment: The path of the job execution log file is logs/import.log.

UnsupportedClassVersionError. is returned when I run a command.

Exception in thread "main" java.lang.UnsupportedClassVersionError: com/aliyun/ossimport2/OSSImport2 : Unsupported major.minor version 51.0
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
        at com.simontuffs.onejar.JarClassLoader.defineClass(JarClassLoader.java:693)
        at com.simontuffs.onejar.JarClassLoader.findClass(JarClassLoader.java:599)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at com.simontuffs.onejar.Boot.run(Boot.java:300)
        at com.simontuffs.onejar.Boot.main(Boot.java:159)
  • Cause: The Java version is earlier than the required version.

  • Solution: Upgrade the Java version to 1.7 or 1.8.

InvocationTargetException. is returned when I run the submit command to submit a job.

Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at com.simontuffs.onejar.Boot.run(Boot.java:306)
        at com.simontuffs.onejar.Boot.main(Boot.java:159)
Caused by: java.lang.NullPointerException
        at com.aliyun.ossimport2.config.JobConfig.load(JobConfig.java:44)
        at com.aliyun.ossimport2.OSSImport2.doSubmitJob(OSSImport2.java:289)
        at com.aliyun.ossimport2.OSSImport2.main(OSSImport2.java:120)
        ... 6 more
  • Cause: The configuration file is incomplete. For example, some configuration items are deleted or commented out from the configuration file.

  • Solution: Restore the configuration items that are deleted or commented out from the configuration file. If you do not need a configuration item, leave the value on the right side of the equal sign (=) empty or delete the configuration item. For more information about configuration examples, see Configuration file examples.

The "com.aliyun.oss.ClientException: Unknown." error message is contained in the job execution log.

com.aliyun.oss.ClientException: Unknown
[ErrorCode]: NonRepeatableRequest
[RequestId]: Cannot retry request with a non-repeatable request entity.  The cause lists the reason the original request failed.
  • Cause: The com.aliyun.oss.ClientException: Unknown. or SocketTimeoutException error is returned when the server consumes full bandwidth.

  • Solution: ossimport automatically retries the migration job. If the job still fails, run the retry command to migrate the files again.

The "too many open files." error message is contained in the job execution log in Linux.

Solution: Run the ulimit -n command to view the limit of handles on Linux.

  • If the handle limit is smaller than 10,000, run the ulimit -n 65536 command to increase the value and restart the process.

  • If the handle limit is greater than 10,000, run the sudo lsof -n command to check which processes have enabled the handles. You can evaluate the processes and stop the unnecessary processes to release the handles. If you no longer need a handle, we recommend that you release the handle.

The migration job exits unexpectedly after the job is started in Windows.

  • Causes:

    • Java is not installed or a Java version earlier than 1.7 is used.

    • The configuration file is invalid.

  • Solutions:

    • Install Java 1.8.

    • Modify the configuration file based on sample configurations. For more information about configuration examples, see Configuration file examples.

After I run the submit command to submit a job, no jobs is running or finished. is displayed when I run the stat command to view the job status.

sudo bash console.sh stat
[WARN]   List files dir not exist : /home/<user>/ossimport/workdir/master/jobs/
no jobs is running or finished.
  • Cause: The job or service is not started when you run the stat command. The stat command returns status of a job only after the job is submitted and the service is started.

  • Solutions:

    • Run the start command to start the service.

    • If the service is started and the job has just been submitted, the master needs to scan the file list first. This error is returned if tasks have not been created and dispatched.

    • If the error still occurs if a long period of time has elapsed after the service start and job submission, check whether the process exits unexpectedly after it starts. If ossimport is deployed in standalone mode, the path of the log file is logs/ossimport2.log. If ossimport is deployed in distributed mode, the path of the job file is logs/ossimport.log. Identify the cause of the error, fix the error, and then start the service process.

scanFinished: false. is displayed when I run the stat command to check the job status.

Solution: Check whether the total number of jobs increases.

  • If the total number of tasks increases, the file list of the migration job is listing new files. It is normal that this error is returned in this case.

  • If the incremental data migration mode is enabled, scanFinished is not true, and the number of tasks does not change, ossimport scans the source directory at specified intervals to check for new or modified files.

  • If the incremental data migration mode is disabled and the number of tasks does not increase, check the job execution log for exceptions. If ossimport is deployed in standalone mode, the path of the log file is logs/ossimport2.log. If ossimport is deployed in distributed mode, the path of the job file is logs/ossimport.log. Identify the cause of the error, fix the error, and then start the service process.

The process in Linux is abnormal, but no exception is reported in the log file.

  • Cause: If the system has less than 2 GB of available memory, process exceptions may occur due to insufficient memory.

  • Solution: Check whether the dmesg log contains records about process exceptions because of insufficient memory.

What operations do I need to perform when the service is restarted after a process exception occurs?

Solution: If you do not delete the jobs that have the same name by running the clean command, all submitted jobs have breakpoint records. In this case, you can run the start command to directly start the service without submitting the job again.

How do I upload files whose names are garbled to OSS on Linux?

Solutions:

  1. Check the encoding format of the garbled names.

  2. Run the export LANG="<your file name encode>" command to parse the encoding format of the garbled file names.

  3. Run the clean command to clear the original job, and then run the submit command to submit the job again.

java.nio.file.AccessDeniedException. is returned when the service is started.

  • Cause: You are not authorized to access the configuration file.

  • Solutions:

    • Add the read permission for all users.

    • Log on to Linux as an administrator to start the service.

The value of Task Counts is 0, but JobState is SUCCEED.

[2015-12-28 16:12:35]   [INFO]  JobName:dir_data
[2015-12-28 16:12:35]   [INFO]  Pending Task Count:0
[2015-12-28 16:12:35]   [INFO]  Dispatched Task Count:0
[2015-12-28 16:12:35]   [INFO]  Succeed Task Count:0
[2015-12-28 16:12:35]   [INFO]  Failed Task Count:0
[2015-12-28 16:12:35]   [INFO]  Is Scan Finished:true
[2015-12-28 16:12:35]   [INFO]  JobState:SUCCEED
  • Cause

    • Files cannot be listed because the value of srcPrefix is invalid.

    • Directories specified by srcPrefix do not contain objects. Directories cannot be uploaded to OSS because the concept of directory is simulated by OSS.

  • Solution: Specify a valid value for the srcPrefix parameter and make sure that there are available files in the directory specified by srcPrefix.

InvocationTargetException. is returned when I submit the job.

sudo submit job:/disk2/ossimport2/local_job.cfg
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at com.simontuffs.onejar.Boot.run(Boot.java:306)
        at com.simontuffs.onejar.Boot.main(Boot.java:159)
Caused by: java.lang.NullPointerException
        at com.aliyun.ossimport2.OSSImport2.doSubmitJob(OSSImport2.java:289)
        at com.aliyun.ossimport2.OSSImport2.main(OSSImport2.java:120)
        ... 6 more
  • Cause: This error is returned because the configuration file or configuration file path is invalid.

  • Solutions:

    • Specify a valid value for the workingDir item in the conf/sys.properties file.

    • Check the configuration file path.

Source files for synchronization do not exist.

Cause: When the master performs a migration job, the master first lists the files and then migrates the files in the list. If files are deleted from the source after listing, the files cannot be migrated. In this case, the master skips the deleted source files and reports the deleted files in the error list.

The job configuration file is valid, but the job execution status is inconsistent with the settings in the job configuration file during the migration.

  • Cause: Modifications made to the configuration file of a job after job submission do not take effect for the job. For example, after you pause a submitted job and modify the configuration file, modifications do not apply to the job.

  • Solution: Delete the previous job by running the clean command. After the job configuration file is modified, submit the job again.

The file is transferred normally, but the error NullPointerException. is returned.

  • Cause: The file statistics feature is added to ossimport 2.3.5. This feature requires cpt files to be loaded. However, the cpt loading format of HTTP jobs is different from that of other sources. As a result, program mismatch occurs.

  • Solutions:

    • Roll back ossimport to version 2.3.4.

    • Ignore the error. This error does not affect the migration job.

Common issues about migrating data from UPYUN Storage Service (USS)

The number of jobs is always 0.

Solution: View the job execution log.

[2016-07-21 10:21:46] [INFO] [name=YoupaiList, totalRequest=1729925, avgLatency=38,
          recentLatency=300000]
  • If the value of recentLatency in the job execution log is smaller than or equal to 30,000, the files are listed. It can take more than 30 seconds (timeout) for files to be listed on USS in most cases. The number of returned files varies based on the number of files that is listed in 30 seconds. Wait until the listing is complete.

  • If the value of recentLatency is small, this error occurs because the account password is invalid. If an error is reported from USS SDK, only null is returned, and no error result is returned. In this case, capture packets to obtain the error code returned by USS for troubleshooting.

How do I specify srcAccessKey and srcSecretKey when migrating data from USS?

Solution: Enter the operator account and password of USS.

HTTP 429 status code keeps appearing when I migrate data from USS.

  • Cause: The number of requests within the period of time exceeds the request throttling threshold configured for USS SDK.

  • Solution: Contact USS to remove request throttling.

After the migration job is complete, the data volume displayed in the OSS console is smaller than the source data volume.

  • Description: The bucket size does not change in the OSS console after all migration jobs are complete, but the data size returned by the du command in Linux is significantly different.

  • Causes:

    • The bucket size in the OSS console are updated with a delay of 1 to 2 hours. Check whether the bucket size changes after 1 to 2 hours.

    • The du command in Linux calculates the block size, which is larger than the file size. We recommend that you run the ls -lR <absolute directory path> | grep "\-rw" | awk '{sum+=$5}END{print sum}' command to calculate the actual size of the local directory.

The message such as unknown command "java" or unknown command "nohup" is returned for a command run in Linux.

  • Cause: You have not installed the package that is required to use the command.

  • Solution: Run the yum, apt-get, or zypper command to install the corresponding package.

Can I use srcPrefix in the configuration file to specify files?

No, you cannot use srcPrefix to specify file formats. You can only use srcPrefix to specify directories or prefixes.

Can I configure a proxy for ossimport?

No, you cannot configure a proxy for ossimport.

Why am I charged for migrating data within OSS?

If you migrate data by using an internal endpoint, you are charged based on the number of requests. You are not charged traffic fees.

Are the files that are deleted from the local directory deleted from OSS if the incremental data migration mode is enabled?

No, local delete operations are not synchronized to OSS.

Why are new local files not synchronized to OSS when the incremental data migration mode is enabled?

In incremental data migration mode, the last modification time of a file is used to determine whether the file is new. The Linux mv command and Windows commands such as cp, mv, and rsync with the -t or -a parameter do not modify the last modification time of files. Files modified by running these commands are not scanned and synchronized to OSS during migration.

Does ossimport synchronize the original file permissions to OSS?

No, ossimport does not synchronize the original file permissions to OSS. After the migration is complete, you can run the set-meta command of ossutil to modify the permissions. For more information, see set-meta (manage object metadata).