Object Storage Service:Gunakan AI VibeFlow untuk menghasilkan data vektor

Konsol

1. Masuk ke Konsol OSS.

2. Di panel navigasi sebelah kiri, klik AI VibeFlow.

3. Di halaman AI VibeFlow, klik Create Data Pipeline.

4. Di panel Create Data Pipeline, konfigurasikan item berikut sesuai petunjuk, lalu klik OK.

   • Pengaturan dasar

     • Rule Name: maksimal 64 karakter. Huruf besar dan kecil, angka, tanda hubung (-), dan garis bawah (_) didukung. Nama harus unik dalam akun dan wilayah yang sama.

     • Rule Description: opsional, maksimal 200 karakter.

   • Pengaturan data sumber

     • Source Bucket: pilih wilayah terlebih dahulu, lalu pilih bucket target.

     • Source File Scope: All Files memproses semua objek dalam bucket; Specify File Name Prefix hanya memproses objek yang sesuai dengan awalan tersebut.

     • Source File Type: Anda dapat memilih beberapa tipe dari Text, Image, dan Video. Pilihan ini harus sesuai dengan kemampuan model vektor yang akan digunakan nanti.

     • Processing Mode: Process Existing Data Only (file yang sudah ada saat pipeline dibuat); Process Incremental Data Only (file baru yang diunggah akan memicu pemrosesan secara otomatis); Process Existing and Incremental Data.

   • Pengaturan vektorisasi

     • Vector Model Source: saat ini hanya mendukung Alibaba Cloud Model Studio.

     • API Key: masukkan kunci API Model Studio, yang diawali dengan sk- dan memiliki panjang 35 hingga 128 karakter.

     • Vector Model: model teks seperti text-embedding-v4 (default) dan text-embedding-v3; model multimodal seperti tongyi-embedding-vision-plus dan tongyi-embedding-vision-flash. Pastikan dimensi output model sama dengan dimensi indeks vektor.

     • Video Snapshot Frequency: hanya dapat dikonfigurasi untuk model multimodal. Rentang nilainya (0, 1], dengan nilai default 1,0. Nilai yang lebih kecil mengambil lebih sedikit frame.

     • Cascade Delete Settings: Retain vector data when the source object is deleted (default); Delete corresponding vector data when the source object is deleted.

   • Pengaturan data tujuan

     • Vector Bucket: harus berada di wilayah yang sama dengan bucket sumber.

     • Vector Index: pilih indeks target dari daftar indeks bucket vektor yang dipilih. Dimensi indeks harus sama dengan dimensi output model vektor.

     • Vector Key Naming Method: opsional. Jika Anda memasukkan awalan, formatnya adalah {prefix}/{ObjectKey}. Jika dibiarkan kosong, kunci vektor akan sama dengan ObjectKey.

     • Use ObjectTag as Metadata: opsional, maksimal 10 kunci, digunakan untuk penyaringan skalar selanjutnya saat pengambilan.

     • Use UserMeta as Metadata: opsional, maksimal 10 kunci. Sistem secara otomatis menambahkan awalan X-OSS-META-.

   • Penanganan pengecualian dan otorisasi

     • Exception Handling: Continue tasks despite errors; Skip exceptions and continue with subsequent tasks while recording exception details (Anda harus mengonfigurasi Exception Information Bucket dan Exception Log Prefix untuk menerima pesan kesalahan; bucket penerima harus berada di wilayah yang sama dengan bucket sumber dan tidak boleh merupakan bucket sumber itu sendiri); Stop subsequent tasks on failure.

     • Authorization Role: pilih New RAM Role untuk secara otomatis membuat dan mengaitkan oss-embedding-{uuid}.

   • Konfirmasi pengaturan: periksa semua pengaturan dan klik OK setelah memastikan semuanya benar.

ossutil

<?xml version="1.0" encoding="UTF-8" ?>
<DataPipelineConfiguration>
  <Sources>
      <InputBucket>my-bucket</InputBucket>
      <InputDataScope>All</InputDataScope>
  </Sources>
  <DataPipelineEmbeddingConfiguration>
      <EmbeddingProvider>bailian</EmbeddingProvider>
      <ApiKey>sk-xxxx</ApiKey>
      <Model>qwen2.5-vl-embedding</Model>
  </DataPipelineEmbeddingConfiguration>
  <Destination>
      <VectorBucketName>my-vector-bucket</VectorBucketName>
      <VectorIndexNames>myIndex</VectorIndexNames>
  </Destination>
  <DataPipelineError>
      <ErrorMode>ignore</ErrorMode>
  </DataPipelineError>
</DataPipelineConfiguration>

ossutil api do-data-pipe-line-action --action putDataPipelineConfiguration \
  --parameters dataPipelineName=my-data-pipeline \
  --parameters role=acs:ram::<AccountId>:role/my-data-pipeline-role \
  --body file://config.xml \
  --region cn-hangzhou

SDK

import com.aliyun.sdk.service.oss2.credentials.CredentialsProvider;
import com.aliyun.sdk.service.oss2.credentials.EnvironmentVariableCredentialsProvider;
import com.aliyun.sdk.service.oss2.dataprocess.OSSDataProcessClient;
import com.aliyun.sdk.service.oss2.dataprocess.models.*;
import java.util.Arrays;
import java.util.Collections;

public class PutDataPipelineConfigurationSample {
    public static void main(String[] args) throws Exception {
        String role = "acs:ram::<AccountId>:role/my-data-pipeline-role";
        CredentialsProvider provider = new EnvironmentVariableCredentialsProvider();
        try (OSSDataProcessClient client = OSSDataProcessClient.newBuilder()
                .region("cn-hangzhou")
                .credentialsProvider(provider)
                .build()) {

            DataPipelineSourceFilterConfiguration filterConfig =
                    DataPipelineSourceFilterConfiguration.newBuilder()
                            .prefixSet(Arrays.asList("data/"))
                            .objectMediaTypes(Arrays.asList("image", "video"))
                            .build();

            DataPipelineSource source = DataPipelineSource.newBuilder()
                    .inputBucket("my-bucket")
                    .inputDataScope("All")
                    .ignoreDelete(true)
                    .filterConfiguration(filterConfig)
                    .build();

            DataPipelineEmbeddingConfiguration embeddingConfig =
                    DataPipelineEmbeddingConfiguration.newBuilder()
                            .embeddingProvider("bailian")
                            .apiKey("sk-xxxx")
                            .model("qwen2.5-vl-embedding")
                            .fps(1.0f)
                            .build();

            DataPipelineDestination destination = DataPipelineDestination.newBuilder()
                    .vectorBucketName("my-vector-bucket")
                    .vectorIndexNames(Collections.singletonList("my-index"))
                    .build();

            PutDataPipelineConfigurationConfiguration config =
                    PutDataPipelineConfigurationConfiguration.newBuilder()
                            .dataPipelineDescription("Data pipeline sample")
                            .sources(Collections.singletonList(source))
                            .dataPipelineEmbeddingConfiguration(embeddingConfig)
                            .destination(destination)
                            .build();

            PutDataPipelineConfigurationResult result = client.putDataPipelineConfiguration(
                    PutDataPipelineConfigurationRequest.newBuilder()
                            .dataPipelineName("my-data-pipeline")
                            .role(role)
                            .putDataPipelineConfigurationConfiguration(config)
                            .build());

            System.out.printf("Status code:%d, request id:%s%n",
                    result.statusCode(), result.requestId());

        } catch (Exception e) {
            System.out.printf("error:%n%s", e);
        }
    }
}

package main

import (
	"context"
	"log"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/dataprocess"
)

func main() {
	region := "cn-hangzhou"
	role := "acs:ram::<AccountId>:role/my-data-pipeline-role"
	apiKey := "sk-xxxx"
	dataPipelineName := "my-data-pipeline"

	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion(region)

	client := dataprocess.NewClient(cfg)

	result, err := client.PutDataPipelineConfiguration(context.TODO(), &dataprocess.PutDataPipelineConfigurationRequest{
		DataPipelineName: oss.Ptr(dataPipelineName),
		Role:             oss.Ptr(role),
		DataPipelineConfiguration: &dataprocess.DataPipelineConfiguration{
			DataPipelineDescription: oss.Ptr("Vectorize business data using the BERT multimodal model"),
			Sources: []dataprocess.DataPipelineSource{
				{
					InputBucket:    oss.Ptr("bucket"),
					InputDataScope: oss.Ptr("All"),
					FilterConfiguration: &dataprocess.DataPipelineSourceFilterConfiguration{
						PrefixSet:        []string{"prefix1"},
						ObjectMediaTypes: []string{"text"},
					},
				},
			},
			DataPipelineEmbeddingConfiguration: &dataprocess.DataPipelineEmbeddingConfiguration{
				ApiKey:            oss.Ptr(apiKey),
				EmbeddingProvider: oss.Ptr("bailian"),
				FPS:               oss.Ptr(float64(1)),
				Model:             oss.Ptr("qwen2.5-vl-embedding"),
			},
			Destination: &dataprocess.DataPipelineDestination{
				VectorBucketName:    oss.Ptr("my-vector-bucket"),
				VectorIndexNames:    []string{"index"},
				VectorKeyPrefix:     oss.Ptr("prefix"),
				ObjectTagToMetadata: []string{"key1"},
				UsermetaToMetadata:  []string{"x-oss-meta-key1"},
			},
			DataPipelineError: &dataprocess.DataPipelineError{
				ErrorMode:   oss.Ptr("ignoreAndRecord"),
				ErrorBucket: oss.Ptr("my-error-bucket"),
				ErrorPrefix: oss.Ptr("error-output/"),
			},
		},
	})

	if err != nil {
		log.Fatalf("failed to put pipeline configuration %v", err)
	}
	log.Printf("put pipeline configuration result:%#v\n", result)
}

API

Panggil operasi PutDataPipelineConfiguration untuk membuat DataPipeline.

Konsol

1. Di halaman AI VibeFlow, klik Rule Name dari DataPipeline target atau klik View Details di kolom Actions.

2. Di halaman detail, lihat informasi lengkap, termasuk pengaturan dasar, pengaturan data sumber, pengaturan vektorisasi, pengaturan tujuan, pengaturan penanganan pengecualian, dan pengaturan otorisasi.

ossutil

ossutil api do-data-pipe-line-action --action getDataPipelineConfiguration \
  --parameters dataPipelineName=my-data-pipeline \
  --region cn-hangzhou

SDK

import com.aliyun.sdk.service.oss2.credentials.CredentialsProvider;
import com.aliyun.sdk.service.oss2.credentials.EnvironmentVariableCredentialsProvider;
import com.aliyun.sdk.service.oss2.dataprocess.OSSDataProcessClient;
import com.aliyun.sdk.service.oss2.dataprocess.models.*;

public class GetDataPipelineConfigurationSample {
    public static void main(String[] args) throws Exception {
        CredentialsProvider provider = new EnvironmentVariableCredentialsProvider();
        try (OSSDataProcessClient client = OSSDataProcessClient.newBuilder()
                .region("cn-hangzhou")
                .credentialsProvider(provider)
                .build()) {

            GetDataPipelineConfigurationResult result = client.getDataPipelineConfiguration(
                    GetDataPipelineConfigurationRequest.newBuilder()
                            .dataPipelineName("my-data-pipeline")
                            .build());

            System.out.printf("Status code:%d, request id:%s%n",
                    result.statusCode(), result.requestId());

            DataPipelineConfiguration cfg = result.dataPipelineConfiguration();
            if (cfg != null) {
                System.out.printf("Pipeline name:%s, status:%s, role:%s, createTime:%s%n",
                        cfg.dataPipelineName(), cfg.status(),
                        cfg.dataPipelineRole(), cfg.createTime());
            }

        } catch (Exception e) {
            System.out.printf("error:%n%s", e);
        }
    }
}

package main

import (
	"context"
	"log"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/dataprocess"
)

func main() {
	region := "cn-hangzhou"
	dataPipelineName := "my-data-pipeline"

	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion(region)

	client := dataprocess.NewClient(cfg)

	result, err := client.GetDataPipelineConfiguration(context.TODO(), &dataprocess.GetDataPipelineConfigurationRequest{
		DataPipelineName: oss.Ptr(dataPipelineName),
	})

	if err != nil {
		log.Fatalf("failed to get pipeline configuration %v", err)
	}
	log.Printf("get pipeline configuration result:%#v\n", result)
}

API

Panggil operasi GetDataPipelineConfiguration untuk mendapatkan konfigurasi DataPipeline.

Konsol

Di halaman AI VibeFlow, Anda dapat langsung melihat daftar semua DataPipeline akun saat ini di wilayah yang dipilih, termasuk informasi seperti nama aturan, deskripsi, status, dan waktu pembuatan.

ossutil

ossutil api do-data-pipe-line-action --action listDataPipelineConfigurations \
  --parameters maxResults=100 \
  --region cn-hangzhou

SDK

import com.aliyun.sdk.service.oss2.credentials.CredentialsProvider;
import com.aliyun.sdk.service.oss2.credentials.EnvironmentVariableCredentialsProvider;
import com.aliyun.sdk.service.oss2.dataprocess.OSSDataProcessClient;
import com.aliyun.sdk.service.oss2.dataprocess.models.*;

public class ListDataPipelineConfigurationsSample {
    public static void main(String[] args) throws Exception {
        CredentialsProvider provider = new EnvironmentVariableCredentialsProvider();
        try (OSSDataProcessClient client = OSSDataProcessClient.newBuilder()
                .region("cn-hangzhou")
                .credentialsProvider(provider)
                .build()) {

            String nextToken = null;
            do {
                ListDataPipelineConfigurationsRequest.Builder reqBuilder =
                        ListDataPipelineConfigurationsRequest.newBuilder().maxResults(100);
                if (nextToken != null) {
                    reqBuilder.nextToken(nextToken);
                }

                ListDataPipelineConfigurationsResult result =
                        client.listDataPipelineConfigurations(reqBuilder.build());

                System.out.printf("Status code:%d, request id:%s%n",
                        result.statusCode(), result.requestId());

                if (result.dataPipelineConfigurations() != null) {
                    for (DataPipelineConfiguration cfg : result.dataPipelineConfigurations()) {
                        System.out.printf("Pipeline: name=%s, status=%s, createTime=%s%n",
                                cfg.dataPipelineName(), cfg.status(), cfg.createTime());
                    }
                }

                nextToken = result.nextToken();
            } while (nextToken != null && !nextToken.isEmpty());

        } catch (Exception e) {
            System.out.printf("error:%n%s", e);
        }
    }
}

package main

import (
	"context"
	"log"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/dataprocess"
)

func main() {
	region := "cn-hangzhou"

	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion(region)

	client := dataprocess.NewClient(cfg)

	p := client.NewListDataPipelineConfigurationsPaginator(&dataprocess.ListDataPipelineConfigurationsRequest{})

	var i int
	log.Println("Data Pipeline Configurations:")
	for p.HasNext() {
		i++

		page, err := p.NextPage(context.TODO())
		if err != nil {
			log.Fatalf("failed to get page %v, %v", i, err)
		}

		for _, data := range page.DataPipelineConfigurations {
			log.Printf("Data Pipeline Name:%v, Data Pipeline Description:%v, Data Pipeline Role:%v, Status:%v, Phase:%v\n", oss.ToString(data.DataPipelineName), oss.ToString(data.DataPipelineDescription), oss.ToString(data.DataPipelineRole), oss.ToString(data.Status), oss.ToString(data.Phase))
		}
	}
}

API

Panggil operasi ListDataPipelineConfigurations untuk menampilkan daftar DataPipeline.

Konsol

Di halaman AI VibeFlow, temukan DataPipeline target dan klik Suspend di kolom Actions.

ossutil

ossutil api do-data-pipe-line-action --action pauseDataPipeline \
  --parameters dataPipelineName=my-data-pipeline \
  --region cn-hangzhou

SDK

package com.example.oss.dataprocess;

import com.aliyun.sdk.service.oss2.credentials.CredentialsProvider;
import com.aliyun.sdk.service.oss2.credentials.EnvironmentVariableCredentialsProvider;
import com.aliyun.sdk.service.oss2.dataprocess.OSSDataProcessClient;
import com.aliyun.sdk.service.oss2.dataprocess.models.*;

public class PauseDataPipelineSample {
    public static void main(String[] args) throws Exception {
        CredentialsProvider provider = new EnvironmentVariableCredentialsProvider();
        try (OSSDataProcessClient client = OSSDataProcessClient.newBuilder()
                .region("cn-hangzhou")
                .credentialsProvider(provider)
                .build()) {

            PauseDataPipelineResult result = client.pauseDataPipeline(
                    PauseDataPipelineRequest.newBuilder()
                            .dataPipelineName("my-data-pipeline")
                            .build());

            System.out.printf("Status code:%d, request id:%s%n",
                    result.statusCode(), result.requestId());

        } catch (Exception e) {
            System.out.printf("error:%n%s", e);
        }
    }
}

package main

import (
	"context"
	"log"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/dataprocess"
)

func main() {
	region := "cn-hangzhou"
	bucketName := "exampleBucket"
	dataPipelineName := "my-data-pipeline"

	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion(region)

	client := dataprocess.NewClient(cfg)

	result, err := client.PauseDataPipeline(context.TODO(), &dataprocess.PauseDataPipelineRequest{
		Bucket:           oss.Ptr(bucketName),
		DataPipelineName: oss.Ptr(dataPipelineName),
	})

	if err != nil {
		log.Fatalf("failed to pause pipeline %v", err)
	}
	log.Printf("pause pipeline result:%#v\n", result)
}

API

Panggil operasi PauseDataPipeline untuk menjeda DataPipeline.

Konsol

Di halaman AI VibeFlow, temukan DataPipeline target dan klik Restart di kolom Actions.

ossutil

ossutil api do-data-pipe-line-action --action restartDataPipeline \
  --parameters dataPipelineName=my-data-pipeline \
  --region cn-hangzhou

SDK

import com.aliyun.sdk.service.oss2.credentials.CredentialsProvider;
import com.aliyun.sdk.service.oss2.credentials.EnvironmentVariableCredentialsProvider;
import com.aliyun.sdk.service.oss2.dataprocess.OSSDataProcessClient;
import com.aliyun.sdk.service.oss2.dataprocess.models.*;

public class RestartDataPipelineSample {
    public static void main(String[] args) throws Exception {
        CredentialsProvider provider = new EnvironmentVariableCredentialsProvider();
        try (OSSDataProcessClient client = OSSDataProcessClient.newBuilder()
                .region("cn-hangzhou")
                .credentialsProvider(provider)
                .build()) {

            RestartDataPipelineResult result = client.restartDataPipeline(
                    RestartDataPipelineRequest.newBuilder()
                            .dataPipelineName("my-data-pipeline")
                            .build());

            System.out.printf("Status code:%d, request id:%s%n",
                    result.statusCode(), result.requestId());

        } catch (Exception e) {
            System.out.printf("error:%n%s", e);
        }
    }
}

package main

import (
	"context"
	"log"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/dataprocess"
)

func main() {
	region := "cn-hangzhou"
	dataPipelineName := "my-data-pipeline"

	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion(region)

	client := dataprocess.NewClient(cfg)

	result, err := client.RestartDataPipeline(context.TODO(), &dataprocess.RestartDataPipelineRequest{
		DataPipelineName: oss.Ptr(dataPipelineName),
	})

	if err != nil {
		log.Fatalf("failed to restart pipeline %v", err)
	}
	log.Printf("restart pipeline result:%#v\n", result)
}

API

Panggil operasi RestartDataPipeline untuk memulai ulang DataPipeline.

Konsol

Di halaman AI VibeFlow, temukan DataPipeline target, klik Delete di kolom Actions, lalu konfirmasi untuk menyelesaikan penghapusan.

ossutil

ossutil api do-data-pipe-line-action --action deleteDataPipelineConfiguration \
  --parameters dataPipelineName=my-data-pipeline \
  --region cn-hangzhou

SDK

import com.aliyun.sdk.service.oss2.credentials.CredentialsProvider;
import com.aliyun.sdk.service.oss2.credentials.EnvironmentVariableCredentialsProvider;
import com.aliyun.sdk.service.oss2.dataprocess.OSSDataProcessClient;
import com.aliyun.sdk.service.oss2.dataprocess.models.*;

public class DeleteDataPipelineConfigurationSample {
    public static void main(String[] args) throws Exception {
        CredentialsProvider provider = new EnvironmentVariableCredentialsProvider();
        try (OSSDataProcessClient client = OSSDataProcessClient.newBuilder()
                .region("cn-hangzhou")
                .credentialsProvider(provider)
                .build()) {

            DeleteDataPipelineConfigurationResult result = client.deleteDataPipelineConfiguration(
                    DeleteDataPipelineConfigurationRequest.newBuilder()
                            .dataPipelineName("my-data-pipeline")
                            .build());

            System.out.printf("Status code:%d, request id:%s%n",
                    result.statusCode(), result.requestId());

        } catch (Exception e) {
            System.out.printf("error:%n%s", e);
        }
    }
}

package main

import (
	"context"
	"log"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/dataprocess"
)

func main() {
	region := "cn-hangzhou"
	dataPipelineName := "my-data-pipeline"

	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion(region)

	client := dataprocess.NewClient(cfg)

	result, err := client.DeleteDataPipelineConfiguration(context.TODO(), &dataprocess.DeleteDataPipelineConfigurationRequest{
		DataPipelineName: oss.Ptr(dataPipelineName),
	})

	if err != nil {
		log.Fatalf("failed to delete pipeline configuration %v", err)
	}
	log.Printf("delete pipeline configuration result:%#v\n", result)
}

API

Panggil operasi DeleteDataPipelineConfiguration untuk menghapus DataPipeline.