Image migration and synchronization between image repositories are the basis for migrating data of a Kubernetes cluster. image-syncer can migrate or synchronize multiple container images at the same time. It can be used to smoothly migrate existing container images to Alibaba Cloud Container Registry. This topic describes how to use image-syncer to migrate container images.

Background information

Compared with the Kubernetes clusters built and maintained by other cloud vendors, Alibaba Cloud Container Service for Kubernetes is superior in terms of service cost, maintenance cost, ease-of-use, and long-term stability. Many cloud vendors want to migrate their Kubernetes loads to Container Service for Kubernetes. If the number of images is small, they can use the docker pull and docker push commands to migrate images. If hundreds or even thousands of images need to be migrated or the size of the image repository is as large as several TB, it takes a long period of time to migrate images with the preceding commands and data may be lost during this process. In this case, the image synchronization capability is required for migrating images between image repositories. The open-source tool image-syncer developed by Alibaba Cloud provides this capability and has helped many cloud vendors migrate their images, among which the maximum image repository capacity is over 3 TB. The server used for running synchronization tasks can utilize its full bandwidth, and no requirement exists for the disk capacity on the server.

image-syncer overview

Image migration and synchronization between image repositories are the basis for migrating data of a Kubernetes cluster. The traditional image synchronization method by using a script containing the docker pull and docker push commands has the following limits:
  • Depends on the disk storage. You must store images to the local disk during the synchronization. Therefore, you must clear local images in a timely manner to prevent the disk space from being used up. In addition, extra time is required for writing images to the local disk. Due to these limits, the traditional method cannot be used to migrate a large number of images in production scenarios.
  • Depends on Docker daemon. Docker daemon strictly limits the number of images that can be pulled or pushed concurrently. As a result, the traditional method cannot synchronize images at a high concurrency.
  • Depends on the HTTP API to implement some features. You cannot synchronize images by using only Docker CLI. As a result, you must write a complex synchronization script.
image-syncer is a simple and easy-to-use tool for migrating or synchronizing multiple images at the same time. With image-syncer, you can synchronize Docker images from or to almost all mainstream image repository services based on Docker Registry V2, such as Container Registry, Docker, Hub, Quay.io, and Harbor. It has been verified for migrating TB-level images in the production environment. For more information, see image-syncer.

Features

image-syncer has the following features:
  • Synchronizes images from multiple source image repositories to multiple destination image repositories.
  • Supports Docker image repository services based on Docker Registry V2.

    For example, image-syncer supports Docker Hub, Quay.io, Container Registry, and Harbor.

  • Synchronizes images only through the memory and network, without storing images to the local disk. This increases the synchronization speed.
  • Supports incremental synchronization.

    image-syncer uses a file to record the blob information about synchronized images. In this way, images are not synchronized repeatedly.

  • Supports concurrent synchronization.

    You can adjust the number of images that can be pulled or pushed concurrently in the configuration file.

  • Automatically retries failed synchronization tasks to resolve most image synchronization issues caused by network jitters.
  • Does not depend on programs such as Docker daemon.

To use image-syncer to migrate, copy, or incrementally synchronize images between image repositories, you only need to make sure that image-syncer can communicate with the source and destination registries over networks. image-syncer almost has no requirements on hardware resources. image-syncer strictly limits the number of network connections to the number of images that are concurrently synchronized. The memory occupied by image-syncer is less than or equal to the number of images that are concurrently synchronized multiplied by the size of the maximum image layer. Therefore, the memory on the server for running synchronization tasks may be used up only when the size of a single image layer and the number of images that are concurrently synchronized are too large. In addition, image-syncer provides the retransmission feature to resolve occasional failures during synchronization. image-syncer counts the number of images that fail to be synchronized when synchronization ends and provides detailed logs to help you locate issues.

Preparations

To use image-syncer, you only need to prepare a configuration file. Example:
{
    "auth": {                   // The authentication information field. Each object contains the username and password for accessing a registry.
                                // Typically, image-syncer must have permissions to pull images from and access tags in the source registry.
                                // image-syncer must have permissions to push images to and create repositories in the destination registry. If no authentication information is provided for a registry, image-syncer accesses the registry in anonymous mode by default.

        "quay.io": {            // The URL of the registry, which must be the same as that of the registry in image URLs.
            "username": "xxx",               // Optional. The username.
            "password": "xxxxxxxxx",         // Optional. The password.
            "insecure": true                 // Optional. Specifies whether the registry is accessed through HTTP. If so, set this parameter to true. Default value: false. Only image-syncer of V1.0.1 or a later version supports this parameter.
        },
        "registry.cn-beijing.aliyuncs.com": {
            "username": "xxx",
            "password": "xxxxxxxxx"
        },
        "registry.hub.docker.com": {
            "username": "xxx",
            "password": "xxxxxxxxxx"
        }
    },
    "images": {
        // The field that describes image synchronization rules. Each rule is a key-value pair, in which the key specifies the URL of source repository and the value specifies the URL of the destination repository.

        // You cannot synchronize a whole namespace or a registry with one rule. You can synchronize a repository at most with one rule.

        // The URLs of the source and destination repositories are in the format of registry/namespace/repository:tag, which is similar to the image URL format used in the docker pull or docker push command.
        // The URL of the source repository must contain registry/namespace/repository at least. If the URL of the destination repository is not an empty string, it must also contain registry/namespace/repository at least.
        // The URL of the source repository cannot be an empty string. To synchronize images from a source repository to multiple destination repositories, you must configure multiple rules.
        // The repository name and tags of the destination can be different from those of the source. In this case, the image synchronization rule works like the combination of the docker pull, docker tag, and docker push commands in sequence.
        "quay.io/coreos/kube-rbac-proxy": "quay.io/ruohe/kube-rbac-proxy",
        "xxxx":"xxxxx",
        "xxx/xxx/xx:tag1,tag2,tag3":"xxx/xxx/xx"
        // If the URL of the source repository does not contain tags, all images in the source repository are synchronized to the destination repository with the original tags. In this case, the URL of the destination repository cannot contain tags either.
        // If the URL of source repository contains only one tag, only images with this tag in the source repository are synchronized to the destination repository. If the URL of the destination repository does not contain a tag, synchronized images keep the original tag by default.
        // If the URL of the source repository contains multiple tags separated with commas (,), such as "a/b/c:1,2,3", the URL of the destination repository cannot contain tags. Synchronized images keep the original tags by default.

        // If the URL of the destination repository is an empty string, images are synchronized to a repository with the same name and tags in the default namespace of the default registry. The default registry and namespace can be set through command parameters or environment variables.
    }     
}

Examples