Copy data: Copy Slice - Elasticsearch - Alibaba Cloud Documentation Center

Copy Slice duplicates the eligible documents of a source Slice into a new or empty destination Slice in the same Collection. The source Slice and its data are retained. The destination Slice is reindexed based on the shard configuration of its own Backing, so the same data can use a different number of primary shards or routing partitions to adapt to changes in data volume or query load. Applications still access the data through the Collection and the Slice, so the shard adjustment is transparent to callers. Copy prepares a complete copy of the data and is not a point-in-time snapshot, a backup, or an atomic traffic switching tool.

Copy rewrites documents one by one, so both the duration and the read and write overhead increase with the data volume.

Copy data semantics

Copy produces an independent copy of the data and leaves the source Slice in its original location. The following list describes how the destination Slice is allocated and how documents are processed during Copy.

The source Slice and the destination Slice must belong to the same Collection and must not have the same name.
When Copy starts, the destination Slice must not contain searchable documents. If it does, Copy returns target_not_empty. This check does not lock the destination Slice, so concurrent writes may still cause conflicts between documents that share an ID.
If the destination Slice does not exist, Copy registers it automatically based on the current allocation rule of the Collection for regular Slices, even when auto_create_slice=false. If the destination Slice already exists, Copy uses its Backing and shard location. Admission failures that occur later do not automatically delete the newly registered destination Slice.
Copy can copy data across Backings and shards. The source Slice and the destination Slice can use different shard counts and routing partition configurations, and documents are reindexed based on the destination configuration.
The Copy API does not provide a parameter for specifying the destination Backing or shard, and it does not guarantee that a new destination Slice and the source Slice are located on different shards. If you adjust the shard count or the routing partition configuration for subsequent Backings, only Backings created afterwards use the new configuration. Copy does not change the shard layout of an existing destination Slice. For more information, see Settings that take effect on subsequent Backings.
Copy uses create-only writes and does not overwrite an existing document with the same _id in the destination Slice. Conflicts are counted in version_conflicts, and the remaining documents continue to be copied. After Copy completes, check this field and handle documents that share an ID based on your business requirements.
Copy preserves document content, including the field referenced by routing_field. For example, when routing_field=cluster_id and cluster-a is copied to cluster-b, the cluster_id of the destination document remains cluster-a. To write documents back to the destination Slice later with Index or Bulk, explicitly specify _slice=cluster-b and change cluster_id in the document to cluster-b. If you write the document back unchanged and omit _slice or routing, the field value routes the document to the source Slice. If you specify the destination Slice explicitly but retain the source field value, the default validation returns HTTP 400. Do not disable validation to bypass this behavior. Adjust the write logic before the application switches traffic.
In vector cluster mode, Copy copies only documents, not the center vector of the Slice. An automatically registered destination Slice has no center vector and therefore does not participate in automatic Slice selection. To let the destination Slice participate, register a center vector separately, or register an empty destination Slice with a center vector before you run Copy.
Both the source Slice and the destination Slice remain readable and writable during Copy. The result is therefore not an atomic point-in-time snapshot of the source Slice.
The source Slice must retain the complete _source required to reconstruct documents. Copy is rejected if _source is disabled or if synthetic _source is used. Copy is also rejected when a non-empty _source.includes is configured, or when _source.excludes excludes content that is still used for indexing, sorting, copy_to, or multi-fields. Excluding only fields that can be safely recovered from the index may work. The server-side validation result prevails.
Copy is currently not supported when the Mapping of the source Slice contains inference fields such as semantic_text.
The default or final ingest pipeline is not executed again during Copy.
Documents are reapplied against the current Mapping of the destination Slice. For example, a newly added multi-field is generated in the destination Slice. Avoid Mapping changes that may affect how the results are interpreted while Copy is running.

Adjust the destination shard configuration (optional)

If you do not need to adjust the shard configuration, start Copy directly. The Copy API cannot specify a destination Backing, so to make the destination Slice use a different number of primary shards or routing partitions, complete the following steps before you start Copy:

Set the parameters for subsequent Backings by using Update a Collection, and then pre-create a Backing. After you pre-create multiple Backings, the system switches the allocation rule for new Slices to random.
If the destination must use the Backing that you just created, change the allocation rule back to last.
Register the destination Slice.
Confirm that the backing_index of the destination Slice uses the new Backing, and then start Copy.

Copy: start the operation

Before you start Copy, confirm that the source data is searchable and that the destination is a new Slice or a Slice without searchable documents. The source documents must meet the _source and Mapping requirements described in Copy data semantics. Copy reads the source data and writes it to the destination Slice, but ongoing writes are not synchronized automatically. Handle incremental data separately before you switch traffic.

Send the following request. Replace {collection}, {source}, and {target} with the Collection name, the source Slice name, and the destination Slice name:

POST /_slice_collection/{collection}/slices/{source}/_copy/{target}

Query parameters:

Parameter	Default value	Description
`wait_for_completion`	`true`	Specifies whether the request waits for Copy to enter a terminal state. If this parameter is set to `false`, the request returns asynchronously.
`timeout`	`30s`	The maximum time that the current HTTP request waits for Copy to enter a terminal state. A timeout does not stop Copy in the background.

All request body parameters are optional:

Parameter	Default value	Value range or description
`workers`	Half the number of processors on the executing node, rounded up and limited to the range `1` to `64`	The number of worker tasks used for parallel replication. This value cannot be adjusted while Copy is running.
`batch_size`	`5000`	The maximum number of documents copied in each batch. Valid values: 1 to 10,000. When memory pressure triggers the circuit breaker, the server halves the batch size automatically and retries, down to a minimum of `1`. The actual batch size can therefore be smaller than the configured value. If Copy still fails after the batch size is reduced to `1`, Copy enters the `FAILED` state.
`requests_per_second`	`-1`	The target document processing rate for the entire Copy operation, in documents per second, shared across all workers. Throttling is applied per batch and does not represent the number of HTTP requests per second. `-1` indicates no rate limit.

The following example starts with one worker, 500 documents per batch, and a total rate of 500 documents per second. These are example values that help you control the initial load, not default values or throughput guarantees. Evaluate the document size, the source query overhead, and the destination write load before you adjust them.

POST /_slice_collection/knowledge-chunks/slices/tenant-a/_copy/tenant-a-copy?wait_for_completion=false
{
  "workers": 1,
  "batch_size": 500,
  "requests_per_second": 500
}

An asynchronous request returns HTTP 202 Accepted. Save the copy_id from the response, and use the status_url and cancel_url that the server returns. The Location response header also provides the status query URL. Sample response:

{
  "copy_id": "opaque-copy-id",
  "completed": false,
  "timed_out": false,
  "state": "RUNNING",
  "source": "tenant-a",
  "target": "tenant-a-copy",
  "progress": { "total": 0, "created": 0, "version_conflicts": 0 },
  "status_url": "/_slice_collection/knowledge-chunks/_copy/opaque-copy-id",
  "cancel_url": "/_slice_collection/knowledge-chunks/_copy/opaque-copy-id/_cancel"
}

The response includes workers only after the execution concurrency is determined, and the value is always greater than 0. Do not assume that the response returned immediately after submission contains this field. When Copy fails, the response also includes error, which describes the cause of the failure.

The copy_id is an opaque identifier that may contain URL reserved characters such as colons. URL-encode the identifier if you construct the status query URL yourself. Use the status_url and cancel_url returned in the response.

A response for a Copy operation in progress contains the progress object. Only state=SUCCEEDED uses result. The FAILED and CANCELLED states also use progress to report the work completed before Copy ended. All three terminal states contain completed=true and took, the elapsed time in milliseconds. version_conflicts appears in the corresponding progress or result object.

With the default wait_for_completion=true, the API returns HTTP 200 if Copy enters a terminal state within the timeout period. If the wait times out, the API returns HTTP 202 with timed_out=true, and Copy continues to run in the background. Disconnecting the client does not cancel Copy automatically.

Important

HTTP 200 only indicates that Copy has entered a terminal state, which can also be FAILED or CANCELLED.

Copy: query the status

Send the following request. Replace <copy_id> with the copy_id that the start request returned:

GET /_slice_collection/knowledge-chunks/_copy/<copy_id>

State	Meaning
`RUNNING`	Copy is in progress or is waiting for automatic recovery.
`CANCELLING`	The cancellation request is accepted and Copy is stopping.
`SUCCEEDED`	Copy completed successfully.
`FAILED`	Copy cannot continue. The response contains `error`.
`CANCELLED`	Copy was cancelled. The destination Slice and the copied data are retained.

A successful status query always returns HTTP 200, including when state=FAILED. Always check state and error instead of relying only on the HTTP status code.

Query the Copy status at intervals and stop polling after Copy enters a terminal state. SUCCEEDED means that Copy is complete, but not that all destination data is searchable. Before you verify the results or switch traffic, wait for the background refresh and then check the query results of the destination Slice. If cold query latency matters to your workload, warm up the destination Slice in advance. If Copy stays in the RUNNING state for a long time and progresses slowly, the server may have reduced the batch size automatically because of memory pressure. This is self-protection behavior, so continue to query at intervals instead of cancelling and retrying Copy immediately.

Adjust the status retention period (optional)

Completed states are retained for at least 1d by default. After that period, queries may return HTTP 404. Do not use the Copy status API as long-term audit storage. The retention period is controlled by the dynamic cluster setting apack.slice_collection.copy.reservation_retention, and the minimum value is 1h. Run the following request to change it:

PUT /_cluster/settings
{
  "persistent": {
    "apack.slice_collection.copy.reservation_retention": "1d"
  }
}

Copy: cancel the operation

Send the following request. Replace <copy_id> with the copy_id that the start request returned:

POST /_slice_collection/knowledge-chunks/_copy/<copy_id>/_cancel

Cancelling Copy does not delete the destination Slice or the documents that have already been copied. When an asynchronous cancellation is accepted, the API returns HTTP 202 with state=CANCELLING. Continue to poll the status until completed=true, which typically returns state=CANCELLED. If Copy succeeded or failed before the cancellation took effect, the corresponding terminal state is returned. After Copy ends, the response no longer includes cancel_url. Cancelling a Copy operation that has already ended returns the current state and HTTP 200.

Error codes

The following table lists the errors that the Copy API returns at startup:

HTTP	`error.type`	Suggested action
`400`	`source_disabled`	Use a `_source` configuration that retains the complete content required to reconstruct the documents.
`400`	`source_representation_unsupported`	Check for synthetic `_source`, `_source` filters, and inference fields.
`409`	`target_not_empty`	The destination Slice already contains searchable data. Confirm the destination Slice name. In most cases, use a new destination Slice.
`409`	`target_lifecycle_conflict`	The destination Slice is being used by another Copy or delete operation. If the response contains `status_url`, check the existing Copy first.

Next steps

After Copy is complete, check the Copy status and result at the application layer, and then switch service traffic to the destination Slice through your business configuration.

For information about choosing between Copy and Branch, see Writable data branch: Branch Slice (Beta).