Image Editing - Vector Retrieval Service for Milvus - Alibaba Cloud Documentation Center

The AI_IMAGE_EDIT function generates edited image URLs from one or more input images based on text prompts. Use this function for white-background image creation, background replacement, style transfer, and reference image fusion.

Syntax

You can call AI_IMAGE_EDIT directly through the REST API or integrate it as a Collection Function in a Milvus collection schema.

REST API

{
  "model_name": "wan2.7-image-pro",
  "texts": ["<image_url>"],
  "params": {"prompt": "<edit_instruction>", "n": 1}
}

For multi-image input, use image_inputs, where each inner array represents one editing task:

{
  "model_name": "wan2.7-image-pro",
  "image_inputs": [["<base_image_url>", "<reference_image_url>"]],
  "params": {"prompt": "<edit_instruction>", "n": 2}
}

Python

schema = MilvusClient.create_schema(auto_id=True, enable_dynamic_field=False)
schema.add_field("id", DataType.INT64, is_primary=True)
schema.add_field("image_url", DataType.VARCHAR, max_length=4096)
schema.add_field("instruction", DataType.VARCHAR, max_length=512)
schema.add_field("edited_image", DataType.VARCHAR, max_length=8192)
schema.add_field("dummy_vector", DataType.FLOAT_VECTOR, dim=2)
schema.add_function(
    Function(
        name="edit_image",
        function_type=texttransform_function_type(),
        input_field_names=["image_url", "instruction"],
        output_field_names=["edited_image"],
        params={
            "provider": "aliyun_milvus",
            "model_name": "wan2.7-image-pro",
            "task": "ai_image_edit",
            "image_fields": "image_url",
            "prompt": "${instruction}",
            "n": "1",
            "size": "1024*1024",
            "watermark": "false",
            "timeout_sec": "180",
        },
    )
)

Parameters

Parameter	Description
`model_name`	Required. Model that returns image URLs synchronously. Use `wan2.7-image-pro` (recommended), `wan2.7-image`, or another compatible model.
`prompt`	Required. Describes the desired edit. In a Collection Function schema, use `${field_name}` to reference other input fields.
`texts`	Array of image URLs for single-image input. Each element represents one editing task. Cannot be used together with `image_inputs`.
`image_inputs`	2D array for multi-image input. Each inner array contains the images for one task. The number of images per task must be consistent within the same request.
`image_fields`	Collection Function only. Optional. Specifies the image fields, defaulting to the first input field. Accepts a comma-separated string such as `base_image,reference_image` or a JSON string array.
`n`	Optional. Default: `1`. Valid values: 1–6. When `n=1`, returns a single image URL. When `n > 1`, returns a list of image URLs, and the Collection Function output field must use `JSON` type instead of `VARCHAR`.
`size` / `negative_prompt` / `seed` / `bbox_list` / `watermark`	Optional. Passed through to the model based on model capabilities.
`timeout_sec` / `max_concurrency`	Optional. Controls the per-request timeout and batch concurrency respectively.
`provider` / `task`	Required for Collection Function only. Set `provider` to `aliyun_milvus` and `task` to `ai_image_edit`.

Return values

When n=1, each element of data.output.outputs is an image URL. When n > 1, each element is a JSON string containing an images array. data.usage may include image_tokens and total_tokens.

Example 1: Single image background replacement (texts, n=1)

Replace the background of a person-and-pet photo with a clean white background while preserving the main subjects. The default asset is a photo of a person and a dog. The curl example requires jq.

REST API

#!/usr/bin/env bash
set -euo pipefail

MILVUS_REST_BASE_URL="http://c-xxxx.milvus.aliyuncs.com:19530"
MILVUS_AUTH_TOKEN="<yourUsername>:<yourPassword>"

post_json() {
  local path="$1"
  local body="$2"
  curl -X POST \
    "$MILVUS_REST_BASE_URL$path" \
    -H "Authorization: Bearer $MILVUS_AUTH_TOKEN" \
    -H "Content-Type: application/json" \
    -d "$body"
}

BODY=$(cat <<JSON
{
  "model_name": "wan2.7-image-pro",
  "texts": ["https://dashscope.oss-cn-beijing.aliyuncs.com/images/dog_and_girl.jpeg"],
  "params": {
    "prompt": "Keep the main subject and replace the background with a clean white background.",
    "n": 1,
    "size": "1024*1024",
    "watermark": false,
    "timeout_sec": 180
  }
}
JSON
)
RESPONSE_BODY="$(post_json "/v2/vectordb/ai/image_edit" "$BODY")"
if command -v jq >/dev/null 2>&1; then
  echo "$RESPONSE_BODY" | jq .
  [ "$(echo "$RESPONSE_BODY" | jq -r '.code // -1')" = "0" ] || exit 1
else
  echo "$RESPONSE_BODY"
fi

Python

from __future__ import annotations

from typing import Any

from pymilvus import DataType, Function, FunctionType, MilvusClient

MILVUS_URI = "http://c-xxxx.milvus.aliyuncs.com:19530"
MILVUS_TOKEN = "<yourUsername>:<yourPassword>"

DUMMY_VECTOR_DIM = 2
TEXTTRANSFORM_FUNCTION_TYPE = 9

def texttransform_function_type() -> Any:
    for type_name in ("TEXTTRANSFORM", "TEXT_TRANSFORM", "TextTransform"):
        function_type = getattr(FunctionType, type_name, None)
        if function_type is not None:
            return function_type
    # Alibaba Cloud Milvus provides TEXTTRANSFORM as a hosted extension (function type value 9);
    # some pymilvus versions do not have this enum member built in, while Function(...) validates via FunctionType(...).
    existing = getattr(FunctionType, "_value2member_map_", {}).get(TEXTTRANSFORM_FUNCTION_TYPE)
    if existing is not None:
        return existing
    extension = int.__new__(FunctionType, TEXTTRANSFORM_FUNCTION_TYPE)
    extension._name_ = "TEXTTRANSFORM"
    extension._value_ = TEXTTRANSFORM_FUNCTION_TYPE
    FunctionType._value2member_map_[TEXTTRANSFORM_FUNCTION_TYPE] = extension
    FunctionType._member_map_["TEXTTRANSFORM"] = extension
    return extension

def add_id(schema: Any) -> None:
    schema.add_field("id", DataType.INT64, is_primary=True)

def add_dummy_vector(schema: Any) -> None:
    schema.add_field("dummy_vector", DataType.FLOAT_VECTOR, dim=DUMMY_VECTOR_DIM)

def run_texttransform_example(*, client, collection_name, input_fields, output_field, function_name, function_params, rows) -> None:
    if client.has_collection(collection_name):
        client.drop_collection(collection_name)
    schema = MilvusClient.create_schema(auto_id=True, enable_dynamic_field=False)
    add_id(schema)
    for name, data_type, max_length in input_fields:
        field_params = {"max_length": max_length} if max_length is not None else {}
        schema.add_field(name, data_type, **field_params)
    output_name, output_data_type, output_max_length = output_field
    output_params = {"max_length": output_max_length} if output_max_length is not None else {}
    schema.add_field(output_name, output_data_type, **output_params)
    add_dummy_vector(schema)
    schema.add_function(
        Function(
            name=function_name,
            function_type=texttransform_function_type(),
            input_field_names=[name for name, _, _ in input_fields],
            output_field_names=[output_name],
            params=function_params,
        )
    )
    index_params = client.prepare_index_params()
    index_params.add_index(field_name="dummy_vector", index_type="AUTOINDEX", metric_type="COSINE")
    client.create_collection(collection_name=collection_name, schema=schema, index_params=index_params)
    client.insert(collection_name, rows)
    client.flush(collection_name)
    fields = [name for name, _, _ in input_fields] + [output_name]
    for row in client.query(collection_name, filter="", output_fields=fields, limit=len(rows)):
        print(row)

MODEL_NAME = "wan2.7-image-pro"
client = MilvusClient(uri=MILVUS_URI, token=MILVUS_TOKEN)

run_texttransform_example(
    client=client,
    collection_name="simple_ai_image_edit_schema",
    input_fields=[("image_url", DataType.VARCHAR, 4096), ("instruction", DataType.VARCHAR, 512)],
    output_field=("edited_image", DataType.VARCHAR, 8192),
    function_name="edit_image",
    function_params={"provider": "aliyun_milvus", "model_name": MODEL_NAME, "task": "ai_image_edit", "image_fields": "image_url", "prompt": "${instruction}", "n": "1", "size": "1024*1024", "watermark": "false", "timeout_sec": "180"},
    rows=[{"image_url": "https://dashscope.oss-cn-beijing.aliyuncs.com/images/dog_and_girl.jpeg", "instruction": "Keep the main subject and replace the background with a clean white background.", "dummy_vector": [0.1, 0.2]}],
)

Expected result:data.output.outputs[0] is a non-empty edited image URL that is stored in the edited_image field (VARCHAR). The generated image is non-deterministic and should not be asserted against a fixed visual output.

Example 2: Generate brand concept image from person-pet photo and clothing reference

Use a person-and-pet photo as the subject reference and a clothing material as the color and texture reference to generate two social media concept images in one request for manual review.

REST API

#!/usr/bin/env bash
set -euo pipefail

MILVUS_REST_BASE_URL="http://c-xxxx.milvus.aliyuncs.com:19530"
MILVUS_AUTH_TOKEN="<yourUsername>:<yourPassword>"

post_json() {
  local path="$1"
  local body="$2"
  curl -X POST \
    "$MILVUS_REST_BASE_URL$path" \
    -H "Authorization: Bearer $MILVUS_AUTH_TOKEN" \
    -H "Content-Type: application/json" \
    -d "$body"
}

BODY=$(cat <<JSON
{
  "model_name": "wan2.7-image-pro",
  "image_inputs": [["https://dashscope.oss-cn-beijing.aliyuncs.com/images/dog_and_girl.jpeg", "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20260415/hynnff/wan-video-edit-clothes.webp"]],
  "params": {
    "prompt": "Keep the person, dog, and seaside composition from the first image. Use the second image only as clothing color and texture reference. Generate brand-safe social media concept images.",
    "n": 2,
    "size": "1024*1024",
    "watermark": false,
    "timeout_sec": 180
  }
}
JSON
)
RESPONSE_BODY="$(post_json "/v2/vectordb/ai/image_edit" "$BODY")"
if command -v jq >/dev/null 2>&1; then
  echo "$RESPONSE_BODY" | jq .
  [ "$(echo "$RESPONSE_BODY" | jq -r '.code // -1')" = "0" ] || exit 1
else
  echo "$RESPONSE_BODY"
fi

Python

from __future__ import annotations

from typing import Any

from pymilvus import DataType, Function, FunctionType, MilvusClient

MILVUS_URI = "http://c-xxxx.milvus.aliyuncs.com:19530"
MILVUS_TOKEN = "<yourUsername>:<yourPassword>"

DUMMY_VECTOR_DIM = 2
TEXTTRANSFORM_FUNCTION_TYPE = 9

def texttransform_function_type() -> Any:
    for type_name in ("TEXTTRANSFORM", "TEXT_TRANSFORM", "TextTransform"):
        function_type = getattr(FunctionType, type_name, None)
        if function_type is not None:
            return function_type
    # Alibaba Cloud Milvus provides TEXTTRANSFORM as a hosted extension (function type value 9);
    # some pymilvus versions do not have this enum member built in, while Function(...) validates via FunctionType(...).
    existing = getattr(FunctionType, "_value2member_map_", {}).get(TEXTTRANSFORM_FUNCTION_TYPE)
    if existing is not None:
        return existing
    extension = int.__new__(FunctionType, TEXTTRANSFORM_FUNCTION_TYPE)
    extension._name_ = "TEXTTRANSFORM"
    extension._value_ = TEXTTRANSFORM_FUNCTION_TYPE
    FunctionType._value2member_map_[TEXTTRANSFORM_FUNCTION_TYPE] = extension
    FunctionType._member_map_["TEXTTRANSFORM"] = extension
    return extension

def add_id(schema: Any) -> None:
    schema.add_field("id", DataType.INT64, is_primary=True)

def add_dummy_vector(schema: Any) -> None:
    schema.add_field("dummy_vector", DataType.FLOAT_VECTOR, dim=DUMMY_VECTOR_DIM)

def run_texttransform_example(*, client, collection_name, input_fields, output_field, function_name, function_params, rows) -> None:
    if client.has_collection(collection_name):
        client.drop_collection(collection_name)
    schema = MilvusClient.create_schema(auto_id=True, enable_dynamic_field=False)
    add_id(schema)
    for name, data_type, max_length in input_fields:
        field_params = {"max_length": max_length} if max_length is not None else {}
        schema.add_field(name, data_type, **field_params)
    output_name, output_data_type, output_max_length = output_field
    output_params = {"max_length": output_max_length} if output_max_length is not None else {}
    schema.add_field(output_name, output_data_type, **output_params)
    add_dummy_vector(schema)
    schema.add_function(
        Function(
            name=function_name,
            function_type=texttransform_function_type(),
            input_field_names=[name for name, _, _ in input_fields],
            output_field_names=[output_name],
            params=function_params,
        )
    )
    index_params = client.prepare_index_params()
    index_params.add_index(field_name="dummy_vector", index_type="AUTOINDEX", metric_type="COSINE")
    client.create_collection(collection_name=collection_name, schema=schema, index_params=index_params)
    client.insert(collection_name, rows)
    client.flush(collection_name)
    fields = [name for name, _, _ in input_fields] + [output_name]
    for row in client.query(collection_name, filter="", output_fields=fields, limit=len(rows)):
        print(row)

MODEL_NAME = "wan2.7-image-pro"
client = MilvusClient(uri=MILVUS_URI, token=MILVUS_TOKEN)

# Each item in image_inputs contains a base image and a reference image; n=2 returns two candidate results.
# Note: for ai_image_edit, when n > 1, the output field must be DataType.JSON (VARCHAR is not supported).
run_texttransform_example(
    client=client,
    collection_name="simple_ai_image_edit_multi_schema",
    input_fields=[
        ("base_image_url", DataType.VARCHAR, 4096),
        ("reference_image_url", DataType.VARCHAR, 4096),
        ("instruction", DataType.VARCHAR, 1024),
    ],
    output_field=("edited_images", DataType.JSON, None),
    function_name="edit_images_with_reference",
    function_params={
        "provider": "aliyun_milvus",
        "model_name": MODEL_NAME,
        "task": "ai_image_edit",
        "image_fields": '["base_image_url","reference_image_url"]',
        "prompt": "${instruction}",
        "n": "2",
        "size": "1024*1024",
        "watermark": "false",
        "timeout_sec": "180",
    },
    rows=[
        {
            "base_image_url": "https://dashscope.oss-cn-beijing.aliyuncs.com/images/dog_and_girl.jpeg",
            "reference_image_url": "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20260415/hynnff/wan-video-edit-clothes.webp",
            "instruction": "Keep the person, dog, and seaside composition from the base image. Use the reference image only as clothing color and texture guidance. Generate brand-safe social media concept images.",
            "dummy_vector": [0.1, 0.2],
        }
    ],
)

Expected result: Because n=2, data.output.outputs[0] is a JSON string containing an images array with two non-empty image URLs.

Important

When n > 1 for the ai_image_edit task, the Collection Function output field must use JSON type instead of VARCHAR. Otherwise, collection creation fails with the error: output field must be a JSON field for task [ai_image_edit] when n > 1.