edit-icon download-icon

Use nasimport in Windows

Last Updated: Dec 28, 2017

You can download and unzip the Windows version of the NAS Data Migration Tool (Nasimport). The tool synchronizes objects on an object storage server (such as OSS) or an ephemeral disk to the NAS file system.

Background

Nasimport supports the following features:

  • Data sources such as ephemeral disks, OSS, Amazon S3, Baidu Object Storage, Tencent Cloud COS, Jinshan Object Storage, UPYUN, Qiniu, and HTTP links.
  • Synchronization of existing data (allowing synchronization of the objects after a specified time point).
  • Automatic synchronization of incremental data.
  • Resumable data transfer.
  • Parallel data uploading and downloading.

Prerequisites

Note the following prerequisites before proceeding.

  • Nasimport must run on an ECS instance that can mount the target NAS file system. See Mount a file system to check whether the ECS can mount a NAS file system and how to mount a NAS file system.

  • Nasimport supports the following operating systems:

    • Windows Server 2008 standard edition SP2 32-bit
    • Windows Server 2008 R2 data center edition 64-bit
    • Windows Server 2012 R2 data center edition 64-bit
    • Windows Server 2016 data center edition 64-bit

Deploy nasimport

  1. Create a synchronization working directory (for example, C:\NasImport) on the local server, and then download the nasimport toolkit to this directory.

  2. Create a new working directory, and unzip the package to the directory.

  3. Edit the configuration file (config/sys.properties) under the working directory.

    Note: We recommend that you use the default configuration. However, you can edit the configuration field values as needed.

Field Description
workingDir The current working directory. The directory to which the toolkit is extracted.
slaveTaskThreadNum The number of simultaneous work threads that perform synchronization.
slaveMaxThroughput(KB/s) Maximum migration throughput.
slaveAbortWhenUncatchedException Whether to skip or stop an unknown error. Unknown errors are not stopped by default.
dispatcherThreadNum Number of parallel threads for dispatching a job. We recommend retaining the default value for most scenarios.

Use nasimport

Nasimport command

Nasimport supports the following commands:

  • Submit a job:

    1. nasimport -c config/sys.properties submit <your-job-configuration>
  • Cancel a job:

    1. nasimport -c config/sys.properties clean <job-name>
  • View a job:

    1. nasimport -c config/sys.properties stat detail
  • Retry a job:

    1. nasimport -c config/sys.properties retry <job-name>
  • Start the service:

    1. nasimport -c config/sys.properties start

Start the service

  1. Enter the working directory, and open the CLI in the current directory.

  2. Run the following command in the CLI: nasimport –c config/sys.properties start

    nasimport

Note:

  • You must keep nasimport running, or set it as a Windows background service.
  • When starting nasimport, you can redirect the log to a file to allow future viewing. Use this command:

    1. nasimport -c config\sys.properties start > nasimport.log 2>&1

Define a job

The file config\local_job.cfg is a template of a job definition file. You can define your own jobs following this template.

FieldDescription
jobNameThe custom job name which uniquely identifies a job. You can submit multiple jobs with different names.
jobTypeYou can set this field to import (performing a data synchronization action) or audit (only verifying the global consistency of the synchronized source data and destination data).
isIncremental=falseThis field specifies whether to enable the automatic incremental mode. If it is set to true, incremental data is rescanned at the interval specified by incrementalModeInterval (unit: seconds) and synchronized to the NAS.
incrementalModeInterval=86400This field specifies a synchronization interval in incremental mode.
importSinceThis field specifies a time value. Data generated later than this value are synchronized. This time value is specified as a Unix timestamp (number of seconds) and the default value is 0.
srcTypeThis field specifies the synchronization source type. Currently, OSS, Qiniu, Baidu, KS3, Youpai, and other China-based sources are supported.
srcAccessKeyIf srcType is set to OSS, Qiniu, Baidu, KS3, or Youpai, this field must be the AccessKey of the data source.
srcSecretKeyIf srcType is set to OSS, Qiniu, Baidu, KS3, or Youpai, this field must be the Secret Key of the data source.
srcDomainThe source endpoint.
srcBucketThe name of the source bucket.
srcPrefixThe source prefix. This field is empty by default.
  • If srcType is set to local, enter the local directory to be synchronized as the prefix. Note that the directory must end with a ‘/‘.
  • If srcType is set to OSS, Qiniu, Baidu, KS3, or Youpai, enter the prefix of the objects to be synchronized.
  • To synchronize all objects, leave the prefix empty.
destTypeThe synchronized object type (NAS by default).
destMountDirThe local mount directory of NAS.
destMountTargetThe NAS mount point domain name.
destNeedMount=trueDetermines whether the tool performs automatic mounting. The default value is true. You can also set it to false, and manually mount the NAS to the destMountDir directory.
destPrefixThe prefix of the synchronized destination objects. The default value is empty.
taskObjectCountLimitThe maximum number of objects for each task. This affects the parallel execution of tasks and is generally set to the total number of objects/the number of download threads you have configured. If the total number of objects is unknown, retain the default value.
taskObjectSizeLimitThe size limit (in bytes) of data downloaded in each task.
scanThreadCountThe number of threads that scan objects in parallel. This field affects object scanning efficiency.
maxMultiThreadScanDepthThe maximum directory depth for parallel scans. We recommend retaining the default value for most scenarios.

Note:

  • If you have configured automatic incremental mode, the job automatcially runs periodically to scan the latest data. The job does not end automatically.

  • If srcType is Youpai, the list object operation cannot implement checkpoints due to API limitations of UPYUN itself. Terminating the process before all list operations are completed causes re-listing of all objects.

Submit a job

In this example, the local C:\\Program Files\\Internet Explorer is copied to NAS.

  1. Edit a job: Copy config\local_job.cfg to the working directory, and edit the following fields.

    edit a job

    Note: For destMountDir, you must select a drive letter that does not currently exist. Otherwise, it may conflict with the existing drive. The destMountTarget is the NAS mount point.

  2. Submit a job: Re-start the CLI in the working directory, and run the following command:

    1. nasimport -c config\sys.properties submit local_job.cfg

    submit a job

    Note:

    • If a job with the same name is in progress, the job cannot be submitted and fails.
    • To pause a synchronization job, you can stop the nasimport process. Restart the nasimort process to resume synchronization from the point at which it was paused.

View the job status

Run the following command in the CLI:

  1. nasimport -c config\sys.properties stat detail

view job status

The overall progress of the current job, and progress of the current task, are displayed. For example, in the preceding figure,

  • 4158464/30492741 indicates the total amount of data to be uploaded (30,492,741 bytes) and the amount of data already uploaded (4,158,464 bytes).
  • 1/55 indicates the total number of objects to be uploaded (55) and the number of objects already uploaded (1).

The migration tool splits a job into multiple tasks for parallel execution. Once all tasks are completed, the job is completed. After the job is completed, the JobState becomes Successful or Failed, indicating the result of the job.

If the job fails, you can view the cause of failure for each task in the following file: master/jobs/$jobName/failed_tasks/*/audit.log

If failure is due to temporary unavailability of the source or destination data, use the following command to retry the failed task:

  1. nasimport -c config/sys.properties retry <job-name>

Then run the stat detail command again after a short wait.

stat detail

Here, SucceededTasks is 1 indicating that the task has been completed successfully. Open your file explorer to verify that the H drive already contains the file.

Reasons for failure

Jobs may fail if any of the following scenarios occur:

  • Job configuration errors, such as AccessKey/ID errors and insufficient permissions occur. In this case, all tasks fail.

    To identify the cause, check the nasimport.log file under the working directory (you can redirect the log to this file when starting nasimport, or directly view the log on the CLI by running the nasimport start command).

  • The encoding of the source object name is inconsistent with the system’s default object name encoding. For example,

    • In Windows, the default object name encoding is GBK.
    • In Linux, the default object name encoding is UTF-8.

      This issue is more likely to occur when the data source is NFS.

  • A change is made to the object in the source directory during the upload process. This is indicated by a SIZE_NOT_MATCH error in audit.log. In this case, the old object is uploaded successfully, but the change is not synchronized to the NAS.

  • A source object is deleted during the upload process. In this case, downloading the object fails.

  • An error occurs in the data source, causing the download of the source data to fail.

  • Performing clean without terminating the process first may cause exceptions in program execution.

  • The program unexpectedly exits, and the job status is Abort.

Recommendations

When configuring the migration service, if the source is OSS, set srcDomain to an intranet domain name such as internal so that you do not incur the cost of downstream traffic from the OSS source and retain faster migration speeds. Fees are charged only by the number of accesses to OSS. You can retrieve the OSS intranet domain name in the OSS console.

If your NAS is in a VPC and the data source is OSS, set srcDomain to the VPC environment domain name provided by OSS. See Regions and endpoints for more information about the corresponding VPC environment domain names of each region.

Thank you! We've received your feedback.