Alibaba Cloud Model Studio:Fine-tune a video generation model

curl --location 'https://dashscope-intl.aliyuncs.com/api/v1/fine-tunes/<your_fine-tuning_job_id>' \
--header "Authorization: Bearer $DASHSCOPE_API_KEY" \
--header 'Content-Type: application/json'

curl --location 'https://dashscope-intl.aliyuncs.com/api/v1/deployments' \
--header "Authorization: Bearer $DASHSCOPE_API_KEY" \
--header 'Content-Type: application/json' \
--data '{
    "model_name": "<your_model_name>",
    "aigc_config": {
        "prompt": "Provide a video description based on the image content. The description must include \"Then, the s86b5p money rain effect begins. Countless large USD bills (beige with dark green patterns) pour down like a storm, densely hitting and surrounding him/her. The bills continue to fall as the camera slowly zooms in. He/She stretches his/her arms upward, neck slightly tilted back, with a surprised expression, fully immersed in this wild money rain\".\nOutput template:\nThe video begins by showing [entity description] [background description]. Then, the s86b5p money rain effect begins. Countless large USD bills (beige with dark green patterns) pour down like a storm, densely hitting and surrounding him/her. The bills continue to fall as the camera slowly zooms in. He/She stretches his/her arms upward, neck slightly tilted back, with a surprised expression, fully immersed in this wild money rain.\nExample:\nThe video begins by showing a young woman on a beach. Her hair is wet, dark brown, curly, and slightly messy. She has a bright smile on her face. The background shows crashing waves and distant mountains. Then, the s86b5p money rain effect begins. Countless large USD bills (beige with dark green patterns) pour down like a storm, densely hitting and surrounding her. The bills continue to fall as the camera slowly zooms in. She stretches her arms upward, neck slightly tilted back, with a surprised expression, fully immersed in this wild money rain.",
        "lora_prompt_default": "Then, the s86b5p money rain effect begins. Countless large USD bills (beige with dark green patterns) pour down like a storm, densely hitting and surrounding the main character. The bills continue to fall as the camera slowly zooms in. The main character stretches their arms upward, neck slightly tilted back, with a surprised expression, fully immersed in this wild money rain."
    },
    "capacity": 1,
    "plan": "lora"
}'

curl --location 'https://dashscope-intl.aliyuncs.com/api/v1/deployments/<your_deployed_model_id>' \
--header "Authorization: Bearer $DASHSCOPE_API_KEY" \
--header 'Content-Type: application/json' 

Follow the consistency principle for effect descriptions

For all samples that contain the effect, the motion description part of the effect must be as consistent as possible. Both the training and validation sets must follow this rule.

• Purpose: When the model finds that s86b5p is always followed by a fixed description and the frame always shows a money rain, it learns to associate s86b5p with the money rain visual effect.

• Example: Whether it is a "young woman" or a "man in a suit," as long as it is the money rain effect, the second half of the prompt is uniformly written as: "...then the s86b5p money rain effect begins, countless US dollar bills pour down like a torrential rain..."

  Sample type	Prompt content (Note the consistency of the underlined description)
  Training set sample 1	The video begins showing a young woman standing in front of a brick wall... (environment description omitted)...then the s86b5p money rain effect begins, countless huge-sized US dollar bills (beige background/dark green pattern) pour down like a torrential rain, densely hitting and surrounding her. The bills continue to fall, she stretches her arms upward, expression surprised, completely immersed in this wild money rain.
  Training set sample 2	The video begins showing a man in a suit in a high-end restaurant... (environment description omitted)....then the s86b5p money rain effect begins, countless huge-sized US dollar bills (beige background/dark green pattern) pour down like a torrential rain, densely hitting and surrounding him. The bills continue to fall, he stretches his arms upward, expression surprised, completely immersed in this wild money rain.
  Validation set sample 1	The video begins showing a young child standing in front of a cityscape... (environment description omitted)...then the s86b5p money rain effect begins, countless huge-sized US dollar bills (beige background/dark green pattern) pour down like a torrential rain, densely hitting and surrounding him. The bills continue to fall while the camera slowly zooms in, he stretches his arms upward, neck slightly tilted back, expression surprised, completely immersed in this wild money rain.

1. Use AI to help generate initial descriptions

   1. Brainstorm (find inspiration): If you do not know how to describe the effect, you can use AI to brainstorm ideas.

      • Send the prompt "Describe the video content in detail" and observe what the model outputs.

      • Focus on the vocabulary the model uses to describe the motion trajectory of the effect (such as "pour down like a torrential rain," "camera slowly zooms in"). These words can be used as material for later optimization.

   2. Fixed sentence pattern (standardize output): Once you have a general idea, you can design a fixed sentence pattern based on the annotation formula to guide the AI in generating prompts that conform to the format.

      Sample code

      For more information about code invocation, see Visual understanding.

      import os
      from openai import OpenAI
      
      client = OpenAI(
          # The API keys for the Singapore and Beijing regions are different. To get an API key: https://www.alibabacloud.com/help/en/model-studio/get-api-key
          # If the environment variable is not configured, replace the following line with your Model Studio API key: api_key="sk-xxx",
          api_key=os.getenv("DASHSCOPE_API_KEY"),
          # The following is the base_url for the Singapore region. If you use a model in the Beijing region, replace the base_url with: https://dashscope.aliyuncs.com/compatible-mode/v1
          base_url="https://dashscope-intl.aliyuncs.com/compatible-mode/v1",
      )
      completion = client.chat.completions.create(
          model="qwen3-vl-plus",
          messages=[
              {"role": "user","content": [{
                  # When you pass a video file directly, set the value of type to video_url
                  # When you use the OpenAI SDK, the video file is sampled every 0.5 seconds by default and cannot be changed. To customize the sampling frequency, use the DashScope SDK.
                  "type": "video_url",            
                  "video_url": {"url": "https://cloud.video.taobao.com/vod/Tm1s_RpnvdXfarR12RekQtR66lbYXj1uziPzMmJoPmI.mp4"}},
                  {"type": "text", "text": "Carefully analyze the video and generate a detailed video description according to the following fixed sentence pattern."
                                          "Sentence template: The video begins showing [entity description]. The background is [background description]. Then the s86b5p melting effect begins, [detailed motion description]."
                                          "Requirements:"
                                          "1.[Entity description]: Describe in detail the person or object originally present in the frame, including details such as appearance, clothing, and expression."
                                          "2.[Background description]: Describe in detail the environment where the entity is located, including details such as environment, lighting, and weather."
                                          "3.[Motion description]: Describe in detail the dynamic change process when the effect occurs (such as how objects move, how lighting changes, and how the camera changes)."
                                          "4.All content must be naturally integrated into the sentence pattern. Do not keep the '[ ]' symbols or add any text unrelated to the description."}]
               }]
      )
      print(completion.choices[0].message.content)

2. Refine the effect template

   1. Repeat the process for multiple samples with the same effect to find common, accurate phrases used to describe the effect. From these, you can extract a universal "effect description."

   2. Copy and paste this standardized effect description into all datasets for that effect.

   3. Keep the unique "entity" and "background" descriptions for each sample, and only replace the "effect description" part with the unified template.

3. Manual check

   AI models may hallucinate or make recognition errors. Therefore, you must perform a final manual check to confirm that the descriptions of the entity and background match the actual scene.