Deploy and run ossimport in standalone mode - Data Online Migration

ossimport is a data migration tool that migrates data to Object Storage Service (OSS). In standalone mode, the Master, Worker, TaskTracker, and Console modules run within a single JAR package on a single machine. This mode supports both Linux and Windows.

Prerequisites

Before you begin, make sure that you have:

Java 1.7 or Java 1.8

Download and configure ossimport

Download and decompress ossimport-2.3.7.zip. After decompression, the directory structure is as follows:

   ossimport
   ├── bin
   │   └── ossimport2.jar  # JAR package containing the Master, Worker, TaskTracker, and Console modules
   ├── conf
   │   ├── local_job.cfg   # Migration job configuration file
   │   └── sys.properties  # System parameters configuration file
   ├── console.bat         # Windows command-line tool for running tasks
   ├── console.sh          # Linux command-line tool for running tasks
   ├── import.bat          # Windows one-click import script (start, migration, verification, and retry)
   ├── import.sh           # Linux one-click import script (start, migration, verification, and retry)
   ├── logs                # Log directory
   └── README.md           # ossimport description file

Edit conf/sys.properties and conf/local_job.cfg based on your migration requirements. Do not modify the following items: For configuration examples, see Configuration file examples. > Important: Check all parameters in sys.properties and local_job.cfg before submitting a migration job. Parameters cannot be modified after a job is submitted.
- The workingDir, workerUserName, workerPassword, and privateKeyFile parameters in conf/sys.properties
- The name and path of the conf/local_job.cfg file
- The jobName parameter in conf/local_job.cfg

Run a migration job

ossimport supports two methods to run migration jobs:

One-click import -- Runs all migration steps through a single script. Best for first-time users.
Step-by-step import -- Runs each migration step individually through Console commands.

One-click import

Start the import script. > Note: If the last migration job is incomplete, ossimport prompts whether to continue from the recorded checkpoint or restart data synchronization. Restart data synchronization if you want to start a different migration job or if you modified the migration source or destination.
- Windows: Double-click import.bat.
- Linux: Run bash import.sh.
Monitor the job progress.
- Windows: A new Command Prompt window opens to run the job and display logs. The ossimport Command Prompt window displays job status every 10 seconds. Do not close either Command Prompt window while the job is running.
- Linux: The job runs in the background.
Handle failed tasks. After the job runs, if a task fails, ossimport prompts whether to retry. Enter y to retry or n to skip the task and exit.
Investigate failures. If the job fails, check the master/jobs/local_test/failed_tasks/<taskid>/audit.log file to identify the cause.

Step-by-step import

The following table lists the Console commands for step-by-step import:

Step	Command	Windows	Linux
1. Clear existing jobs	`clean`	`console.bat clean`	`bash console.sh clean`
2. Submit a migration job	`submit`	`console.bat submit`	`bash console.sh submit`
3. Start the service	`start`	`console.bat start`	`bash console.sh start`
4. View job status	`stat`	`console.bat stat`	`bash console.sh stat`
5. Retry failed tasks	`retry`	`console.bat retry`	`bash console.sh retry`
6. Stop the service	`stop`	Close `%JAVA_HOME%/bin/java.exe` window	`bash console.sh stop`

Step 1: Clear existing jobs

Run the clean command only when running a migration job with the same name as an existing job. Do not run clean for new migration jobs or when retrying failed tasks.

Step 2: Submit a migration job

The configuration file for the job is conf/local_job.cfg. The default job name is local_test. Make sure the migration job name is unique before submitting.

Step 3: Start the service

Start the ossimport service to begin processing submitted jobs.

Step 4: View job status

Check the current status of the migration job.

Step 5: Retry failed tasks

Tasks may fail due to reasons such as network issues. The retry command only retries failed tasks.

Step 6: Stop the service

Windows: Close the %JAVA_HOME%/bin/java.exe window.
Linux: Run bash console.sh stop.

View job status and logs

After a migration job is submitted, the Master splits the job into tasks, the Worker runs the tasks, and the TaskTracker collects the task status. The following directory structure shows the key files generated after a job completes:

ossimport
├── bin
│   └── ossimport2.jar          # JAR package for standalone mode
├── conf
│   ├── local_job.cfg           # Migration job configuration file
│   └── sys.properties          # System parameters configuration file
├── console.sh                  # Command-line tool
├── import.sh                   # One-click import script
├── logs
│   ├── import.log              # Migration logs
│   ├── job_stat.log            # Job status logs
│   ├── ossimport2.log          # ossimport run logs for standalone mode
│   └── submit.log              # Job submission logs
├── master
│   ├── jobqueue                # Jobs not yet split into tasks
│   └── jobs
│       └── local_test          # Job name
│           ├── checkpoints     # Checkpoints from Master splitting the job into tasks
│           │   └── 0
│           │       └── 034DC9DD2860B0CFE884242BC6FF92E7.cpt
│           ├── dispatched      # Tasks dispatched to the Worker but not yet complete
│           │   └── localhost
│           ├── failed_tasks    # Failed tasks
│           ├── pending_tasks   # Tasks not yet dispatched
│           └── succeed_tasks   # Successfully completed tasks
│               └── A41506C07BF1DF2A3EDB4CE31756B93F_1499744514501@localhost
│                   ├── audit.log   # Task run logs (use for error diagnosis)
│                   ├── DONE        # Mark file indicating task completion
│                   ├── error.list  # Task errors
│                   ├── STATUS      # Task status: Failed or Completed
│                   └── TASK        # Task description
└── worker                      # Tasks currently running (managed by Master after completion)
    └── jobs
        └── local_test
            └── tasks

Important

To view job information, check logs/ossimport2.log or logs/import.log.
To troubleshoot a failed task, check master/jobs/${JobName}/failed_tasks/${TaskName}/audit.log.
To view errors for a failed task, check master/jobs/${JobName}/failed_tasks/${TaskName}/error.list.
These logs are for reference only. Do not deploy your service and application based on these logs.

Verify migration results

ossimport does not verify data after migration and does not ensure data consistency or integrity. After a migration job completes, verify data consistency between the migration source and destination.

Warning

If you delete source data without verifying data consistency between the migration source and destination, you are responsible for any losses and consequences that arise.

Troubleshooting

The following table lists common causes of migration failures:

Failure scenario	Cause	Details
`SIZE_NOT_MATCH` error in `log/audit.log`	Source data modified during migration	The original data has been uploaded, but modifications are not uploaded to OSS.
Migration job fails	Source data deleted during migration	The migration job cannot proceed without the source data.
Migration to OSS fails	Source file names violate OSS naming rules	Object names in OSS cannot start with a forward slash (`/`) and cannot be empty.
Source data fails to download	Network issues or insufficient permissions	Check `logs/ossimport2.log` or `logs/import.log` for details.
Job state shows `Abort`	ossimport exits unexpectedly during migration	Contact technical support.