Platform For AI:LLM-based intent recognition solution

Intent recognition

Intent recognition allows an AI agent to understand user requests in natural language and then perform corresponding actions or provide relevant information. It is a key component of intelligent interactive systems. Currently, intent recognition technology based on large language models (LLMs) is widely used in the industry.

Use cases

• With smart voice assistants, users interact with the assistant through simple voice commands. For example, if a user says, "I want to listen to music," the system must accurately identify the intent to play music and then execute the corresponding action.

• In intelligent customer service scenarios, the challenge is to process diverse customer requests and accurately route them to the correct workflows, such as returns, exchanges, or complaints. For example, on an e-commerce platform, a user might say, "The product I received is defective. I want to return it." In this case, the LLM-based intent recognition system must quickly capture the user's intent as "return," automatically trigger the return process, and guide the user through the subsequent steps.

Workflow

1. Prepare training data

   You can prepare a training dataset for your specific business scenario that follows the specified data format and preparation strategy. Alternatively, you can prepare your business data and use iTAG to label the raw data. Export the labeled results and convert them to a format supported by PAI-QuickStart for model training.

2. Train the model and perform offline evaluation

   In QuickStart, train the model based on Qwen1.5-1.8B-Chat. After training the model, perform an offline evaluation.

3. Deploy and call the model service

   If the evaluation results are satisfactory, use QuickStart to deploy the trained model as an EAS online service.

Data preparation strategies

To improve model training effectiveness and stability, follow these strategies when preparing your data:

• For single-intent recognition scenarios, ensure each intent category has at least 50 to 100 labeled examples. If model fine-tuning results are not satisfactory, consider increasing the amount of labeled data. Also, ensure the data is balanced across intent categories. Avoid having one category with significantly more data than others.

• For multi-intent or multi-turn dialogue scenarios, use at least 20% more labeled data than you would for single-intent scenarios. Additionally, any intents used in multi-intent or multi-turn scenarios must also be defined in your single-intent data.

• Intent descriptions should cover a wide variety of phrasings and scenarios.

Data format

The training data must be a JSON file that contains instruction and output fields. The instruction field contains the input instruction, and the output field contains the model's predicted intent with its key parameters. The following examples show the training data for different intent recognition scenarios.

• For single-intent recognition scenarios, you need to prepare business-specific data to fine-tune a large language model (LLM). For example, the following is sample training data for a single-turn dialogue in a smart home scenario:

  [
      {
          "instruction": "I want to listen to music",
          "output": "play_music()"
      },
      {
          "instruction": "It's too loud, turn the volume down",
          "output": "volume_down()"
      },
      {
          "instruction": "I don't want to listen anymore, turn it off",
          "output": "music_exit()"
      },
      {
          "instruction": "I want to travel to Hangzhou, help me check the weather forecast",
          "output": "weather_search(China (Hangzhou))"
      },
  ]

• For multi-intent or multi-turn dialogue scenarios, a user's intent might be expressed across multiple turns of conversation. In such cases, you can prepare multi-turn dialogue data and label the user inputs from all relevant turns. For example, consider this multi-turn dialogue with a voice assistant:

  User: I want to listen to music.
  Assistant: What type of music?
  User: Play some *** music for me.
  Assistant: play_music(***)

  The corresponding training data for this multi-turn dialogue is formatted as follows:

  [
      {
          "instruction": "I want to listen to music. Play some *** music for me.",
          "output": "play_music(***)"
      }
  ]

Training a model on multi-turn dialogues significantly increases the input length, and such scenarios are less common in practice. Consider multi-turn dialogue training only if single-turn intent recognition does not meet your business needs. This solution uses a single-turn dialogue example to demonstrate the process.

Use PAI-iTAG to label data

Alternatively, use the PAI-iTAG platform to label data and generate a training dataset in the required format.

1. Register the data to be labeled in iTAG as a PAI dataset.

   1. Prepare a data file in the manifest format according to the data preparation strategies. The following is an example.

      {"data":{"instruction": "I want to listen to music"}}
      {"data":{"instruction": "It's too loud, turn the volume down"}}
      {"data":{"instruction": "I don't want to listen anymore, turn it off"}}
      {"data":{"instruction": "I want to travel to Hangzhou, help me check the weather forecast"}}

   2. Go to the AI Asset Management > Datasets page, select the target workspace, and click Go to Dataset.

   3. Click Create Dataset and configure the following key parameters. For details about other parameters, see Create and manage datasets.

      Parameter	Description
      Storage Type	Select Alibaba Cloud Object Storage Service (OSS).
      Import Format	Select File.
      OSS Path	Select an existing OSS directory and upload the prepared manifest file by following these steps: Click the icon. In the Select OSS file dialog box, click Upload File. Click Browse Local Files or Drag and Drop File to Upload, and then upload the manifest file as prompted.

2. Go to the Data Preparation > iTAG page, click Go to Management Page, and then switch to the Template Management tab.

3. Click Create Template, select Custom Template > Basic Templates, and click Edit. After you configure the parameters, click Save Template Name. The following table describes the key configurations. For more information about other configurations, see template management.

   Configuration	Description
   Basic template canvas area	Select Text and click Generate Content Card. Click the text area. In the Import Dataset dialog box, select an existing dataset. Then, in the Basic Template Configuration area on the right, select Dataset Field Name > instruction.
   Basic template answering area	Select Input Field and click Generate Title Card. Then, change the Title to output.

4. In the left-side navigation pane, choose Management Center > Task Management. On the Task Management tab, click Create Task. On the Create Annotation Task page, configure the parameters and click Create. The following table describes the key parameters. For details about other parameters, see Create a labeling task.

   Parameter	Description
   Input data set	Select the dataset you created in the previous step. Note Make sure that the input data corresponds to the template that you are using.
   Template Type	Select Custom Template and select the template you created from the drop-down list.

5. After the task is created, begin labeling the data. For more information, see Process labeling tasks.

6. After you finish labeling the data, export the results to an OSS directory. For more information, see Export labeled data.

   The output manifest file will be similar to the following example. For more information about the data format, see Overview of labeled data formats.

   {"data":{"instruction":"I want to listen to music","_itag_index":""},"label-1947839552568066048-system":{"fixedFlag":0,"results":[{"MarkResultId":"1947839554911772672","MarkTitle":"Basic Template","MarkResult":"{\"tabId\":\"CommonExtensions\",\"annotations\":[{\"id\":null,\"labels\":{\"output\":\"play_music()\"},\"exif\":null}],\"type\":\"CommonExtensions\",\"version\":\"v2\"}","QuestionId":"CommonExtensions","ResultType":"OPEN_GROUP","Progress":null,"Version":"1753236185165","MarkTime":"Wed Jul 23 10:03:05 CST 2025","UserMarkResultId":null,"IsNeedVoteJudge":false}],"abandonFlag":0},"label-1947839552568066048":{"results":[{"MarkResultId":"1947839554911772672","MarkTitle":"Basic Template","MarkResult":"{\"tabId\":\"CommonExtensions\",\"annotations\":[{\"id\":null,\"labels\":{\"output\":\"play_music()\"},\"exif\":null}],\"type\":\"CommonExtensions\",\"version\":\"v2\"}","QuestionId":"CommonExtensions","ResultType":"OPEN_GROUP","Progress":null,"Version":"1753236185165","MarkTime":"Wed Jul 23 10:03:05 CST 2025","UserMarkResultId":"1947839763671740416","IsNeedVoteJudge":false}]},"abandonFlag":0,"abandonRemark":null}
   {"data":{"instruction":"It's too loud, turn the volume down","_itag_index":""},"label-1947839552568066048-system":{"fixedFlag":0,"results":[{"MarkResultId":"1947839554891464704","MarkTitle":"Basic Template","MarkResult":"{\"tabId\":\"CommonExtensions\",\"annotations\":[{\"id\":null,\"labels\":{\"output\":\"volume_down()\"},\"exif\":null}],\"type\":\"CommonExtensions\",\"version\":\"v2\"}","QuestionId":"CommonExtensions","ResultType":"OPEN_GROUP","Progress":null,"Version":"1753236198979","MarkTime":"Wed Jul 23 10:03:19 CST 2025","UserMarkResultId":null,"IsNeedVoteJudge":false}],"abandonFlag":0},"label-1947839552568066048":{"results":[{"MarkResultId":"1947839554891464704","MarkTitle":"Basic Template","MarkResult":"{\"tabId\":\"CommonExtensions\",\"annotations\":[{\"id\":null,\"labels\":{\"output\":\"volume_down()\"},\"exif\":null}],\"type\":\"CommonExtensions\",\"version\":\"v2\"}","QuestionId":"CommonExtensions","ResultType":"OPEN_GROUP","Progress":null,"Version":"1753236198979","MarkTime":"Wed Jul 23 10:03:19 CST 2025","UserMarkResultId":"1947839868520656896","IsNeedVoteJudge":false}]},"abandonFlag":0,"abandonRemark":null}
   {"data":{"instruction":"I don't want to listen anymore, turn it off","_itag_index":""},"label-1947839552568066048-system":{"fixedFlag":0,"results":[{"MarkResultId":"1947839554992373760","MarkTitle":"Basic Template","MarkResult":"{\"tabId\":\"CommonExtensions\",\"annotations\":[{\"id\":null,\"labels\":{\"output\":\"music_exit()\"},\"exif\":null}],\"type\":\"CommonExtensions\",\"version\":\"v2\"}","QuestionId":"CommonExtensions","ResultType":"OPEN_GROUP","Progress":null,"Version":"1753236212152","MarkTime":"Wed Jul 23 10:03:32 CST 2025","UserMarkResultId":null,"IsNeedVoteJudge":false}],"abandonFlag":0},"label-1947839552568066048":{"results":[{"MarkResultId":"1947839554992373760","MarkTitle":"Basic Template","MarkResult":"{\"tabId\":\"CommonExtensions\",\"annotations\":[{\"id\":null,\"labels\":{\"output\":\"music_exit()\"},\"exif\":null}],\"type\":\"CommonExtensions\",\"version\":\"v2\"}","QuestionId":"CommonExtensions","ResultType":"OPEN_GROUP","Progress":null,"Version":"1753236212152","MarkTime":"Wed Jul 23 10:03:32 CST 2025","UserMarkResultId":"1947839936657285120","IsNeedVoteJudge":false}]},"abandonFlag":0,"abandonRemark":null}
   {"data":{"instruction":"I want to travel to Hangzhou, help me check the weather forecast","_itag_index":""},"label-1947839552568066048-system":{"fixedFlag":0,"results":[{"MarkResultId":"1947839554971426816","MarkTitle":"Basic Template","MarkResult":"{\"tabId\":\"CommonExtensions\",\"annotations\":[{\"id\":null,\"labels\":{\"output\":\"weather_search(China (Hangzhou))\"},\"exif\":null}],\"type\":\"CommonExtensions\",\"version\":\"v2\"}","QuestionId":"CommonExtensions","ResultType":"OPEN_GROUP","Progress":null,"Version":"1753236218730","MarkTime":"Wed Jul 23 10:03:39 CST 2025","UserMarkResultId":null,"IsNeedVoteJudge":false}],"abandonFlag":0},"label-1947839552568066048":{"results":[{"MarkResultId":"1947839554971426816","MarkTitle":"Basic Template","MarkResult":"{\"tabId\":\"CommonExtensions\",\"annotations\":[{\"id\":null,\"labels\":{\"output\":\"weather_search(China (Hangzhou))\"},\"exif\":null}],\"type\":\"CommonExtensions\",\"version\":\"v2\"}","QuestionId":"CommonExtensions","ResultType":"OPEN_GROUP","Progress":null,"Version":"1753236218730","MarkTime":"Wed Jul 23 10:03:39 CST 2025","UserMarkResultId":"1947839975890939904","IsNeedVoteJudge":false}]},"abandonFlag":0,"abandonRemark":null}

7. In a terminal, run the following Python script to convert the manifest file of the labeled data into the training data format required for the QuickStart.

   import json
   # Input and output file paths
   input_file_path = 'test_json.manifest'
   output_file_path = 'train.json'
   converted_data = []
   with open(input_file_path, 'r', encoding='utf-8') as file:
       for line in file:
           try:
               # Parse the JSON data from each line
               data = json.loads(line)
               # Extract the instruction
               instruction = data['data']['instruction']
               # Iterate through all keys that start with "label-"
               for key in data.keys():
                   if key.startswith('label-'):
                       # Extract MarkResult and parse its content
                       mark_result_str = data[key]['results'][0]['MarkResult']
                       mark_result = json.loads(mark_result_str)  # Parse the MarkResult string as JSON
                       # Extract labels["output"] from annotations
                       output = mark_result['annotations'][0]['labels']['output']
                       # Construct the new data structure
                       converted_data.append({
                           'instruction': instruction,
                           'output': output
                       })
                       break
           except Exception as e:
               print(f"Error processing line: {line.strip()}. Error: {e}")
   # Write the converted data to the output file
   with open(output_file_path, 'w', encoding='utf-8') as outfile:
       json.dump(converted_data, outfile, ensure_ascii=False, indent=4)
   print(f"Conversion completed. Output saved to {output_file_path}")
   

Train the model

QuickStart offers excellent pre-trained models from the open-source AI community. It lets you manage the entire model development workflow—from training and deployment to inference—without writing any code.

This guide uses the Qwen1.5-1.8B-Chat model as an example to show how to train a model in QuickStart with your training data. The steps are as follows:

1. Go to the Model Gallery page.

   1. Log on to the PAI console.

   2. In the upper-left corner, select a region.

   3. In the left-side navigation pane, choose Workspaces. Click the target workspace.

   4. In the left-side navigation pane, choose QuickStart > Model Gallery.

2. On the QuickStart page, find and click the Qwen1.5-1.8B-Chat model card in the model list on the right.

3. On the model details page, click Train in the upper-right corner. In the Train configuration panel, configure the following key parameters and leave the rest at their default values.

   Parameter		Description
   Training Mode		full-parameter fine-tuning: This method is resource-intensive and time-consuming but generally yields better results. Note Models with fewer parameters support full-parameter fine-tuning. Choose this method based on your specific requirements. QLoRA: A lightweight fine-tuning method that requires fewer resources and less training time than full-parameter fine-tuning, though the results may be less optimal. LoRA: Similar to QLoRA, this is a lightweight fine-tuning method.
   Dataset configuration	Training dataset	Follow these steps to select your training dataset. From the drop-down list, select OSS file or directory. Click the Browse icon to select your OSS directory. In the Select OSS File dialog box, click Upload File, drag and drop your training dataset file, and then click OK.
   Output Configuration	Model output path	Select an OSS directory to store the configuration and model files generated by the training job.
   	Tensorboard Output Path
   Hyperparameter Configuration		For a detailed description of all hyperparameters, see Table 1. Hyperparameter reference. Use the following strategies to configure hyperparameters. For recommended key hyperparameter settings for different training methods, see Table 2. Recommended hyperparameter configurations. Ensure you configure the hyperparameters based on the selected training method. global batch size = Number of GPUs * per_device_train_batch_size * gradient_accumulation_steps To maximize training performance, prioritize increasing the number of GPUs and per_device_train_batch_size. Typically, set the global batch size to a value between 64 and 256. If your training dataset is very small, you can use a smaller value. The sequence length (seq_length) can be adjusted based on your specific scenario. For example, if the maximum sequence length in your dataset is 50, you can set the sequence length to 64, which is typically a power of 2. If the training loss decreases too slowly or does not converge, consider increasing the learning rate (learning_rate). Also, ensure the quality of your training data.

   Table 1. Hyperparameters

   Parameter	Type	Description	Default
   learning_rate	FLOAT	The learning rate for model training.	5e-5
   num_train_epochs	INT	The number of training epochs.	1
   per_device_train_batch_size	INT	The training batch size for each GPU.	1
   seq_length	INT	The maximum sequence length for model inputs.	128
   lora_dim	INT	The LoRA dimension. When lora_dim > 0, lightweight fine-tuning with LoRA or QLoRA is used.	32
   lora_alpha	INT	The LoRA scaling factor. This parameter takes effect for LoRA or QLoRA lightweight fine-tuning when lora_dim > 0.	32
   load_in_4bit	BOOL	If true, loads the base model in 4-bit precision. This enables 4-bit QLoRA when lora_dim > 0 and load_in_8bit is false.	false
   load_in_8bit	BOOL	If true, loads the base model in 8-bit precision. This enables 8-bit QLoRA when lora_dim > 0 and load_in_4bit is false.	false
   gradient_accumulation_steps	INT	The number of gradient accumulation steps.	8
   apply_chat_template	BOOL	Specifies whether the algorithm adds the model's default Chat Template to the training data. For Qwen1.5 series models, the format is as follows: Question: <|im_start|>user\n + instruction + <|im_end|>\n Answer: <|im_start|>assistant\n + output + <|im_end|>\n	true
   system_prompt	STRING	The default system prompt for model training. This parameter takes effect when apply_chat_template is set to true. When training Qwen1.5 series models, you can configure a custom system_prompt to assign a specific role to the large language model. The algorithm automatically expands the training data, so you do not need to manage the details. For example, if you set system_prompt to "You are an expert in intent recognition. You can identify the user's intent from their question and return the corresponding function calling and parameters." For the following training sample: [ { "instruction": "I want to listen to music", "output": "play_music()" } ] The actual data used for training is formatted as follows: <|im_start|>system\nYou are an expert in intent recognition. You can identify the user's intent from their question and return the corresponding function calling and parameters<|im_end|>\n<|im_start|>user\nI want to listen to music<|im_end|>\n<|im_start|>assistant\nplay_music()<|im_end|>\n	You are a helpful assistant

   Table 2. Recommended hyperparameters

   Parameter	Full-parameter fine-tuning	LoRA/QLoRA
   learning_rate	5e-6, 5e-5	3e-4
   global batch size	256	256
   seq_length	256	256
   num_train_epochs	3	5
   lora_dim	0	64
   lora_alpha	0	16
   load_in_4bit	false	true/false
   load_in_8bit	false	true/false

4. Click Fine-tune, and then click OK in the Billing Notification dialog box.

   You are automatically redirected to the training job details page. After the job is created, its status changes to Running. The page displays the job's progress through several stages: Creating, Preparing Environment, and Running. Basic information is displayed, including the pre-trained model qwen1.5-1.8b-chat, the training method SFT supervised fine-tuning: LoRA, and the job category Generative AI / large language model. You can view details such as input paths, output paths, and hyperparameter configurations on the Task Details, Task Log, and Task Monitoring tabs.

Evaluate the model

Once training is complete, you can evaluate the model's performance by running a Python script in a terminal.

1. Prepare an evaluation data file named testdata.json. The following is an example:

   [
       {
           "instruction": "Who sings the song 'Ten Years'?",
           "output": "music_query_player(Ten Years)"
       },
       {
           "instruction": "What's the weather like in Hangzhou today?",
           "output": "weather_search(Hangzhou)"
       }
   ]

2. Run the following Python script in a terminal to evaluate the model offline.

   #encoding=utf-8
   from transformers import AutoModelForCausalLM, AutoTokenizer
   import json
   from tqdm import tqdm
   device = "cuda" # the device to load the model onto
   # Modify the model path
   model_name = '/mnt/workspace/model/qwen14b-lora-3e4-256-train/'
   print(model_name)
   model = AutoModelForCausalLM.from_pretrained(
       model_name,
       torch_dtype="auto",
       device_map="auto"
   )
   tokenizer = AutoTokenizer.from_pretrained(model_name)
   count = 0
   ecount = 0
   # Modify the training data path
   test_data = json.load(open('/mnt/workspace/data/testdata.json'))
   system_prompt = 'You are an expert in intent recognition. You can identify the user\'s intent from their question and return the corresponding function calling and parameters.'
   for i in tqdm(test_data[:]):
       prompt = '<|im_start|>system\n' + system_prompt + '<|im_end|>\n<|im_start|>user\n' + i['instruction'] + '<|im_end|>\n<|im_start|>assistant\n'
       gold = i['output']
       gold = gold.split(';')[0] if ';' in gold else gold
       model_inputs = tokenizer([prompt], return_tensors="pt").to(device)
       generated_ids = model.generate(
           model_inputs.input_ids,
           max_new_tokens=64,
           pad_token_id=tokenizer.eos_token_id,
           eos_token_id=tokenizer.eos_token_id,
           do_sample=False
       )
       generated_ids = [
           output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
       ]
       pred = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
       if gold.split('(')[0] == pred.split('(')[0]:
           count += 1
           gold_list = set(gold.strip()[:-1].split('(')[1].split(','))
           pred_list = set(pred.strip()[:-1].split('(')[1].split(','))
           if gold_list == pred_list:
               ecount += 1
       else:
           pass
   print("Intent recognition accuracy:", count/len(test_data))
   print("Parameter recognition accuracy:", ecount/len(test_data))

   Note

   If the code execution returns the message Using low_cpu_mem_usage=True or a device_map requires Accelerate: pip install accelerate, run pip install accelerate to install the required dependency.

Deploy the model service

If the model's evaluation results meet your expectations, you can deploy the model as an EAS online service.

1. On the Task details page, click Deploy in the upper-right corner. The system pre-fills the basic and resource information. For Deployment Method, select vLLM accelerated deployment. You can modify these settings as needed. After configuring the parameters, click Deploy.

2. In the Billing Notification dialog box, click OK.

   The system automatically redirects you to the deployment task page. When the Status changes to Running, the deployment is successful.

Invoke the model service

The following example shows how to call the API from a client:

1. Get the endpoint and token.

   1. On the Service details page, in the Basic Information section, click View Endpoint Information.

   2. In the Invocation Information dialog box, find the endpoint and token, and save them locally.

2. The following is a code example for a vLLM accelerated deployment. Run this code in a terminal to invoke the service.

   from openai import OpenAI
   ##### API Configuration #####
   openai_api_key = "<YOUR_EAS_SERVICE_TOKEN>"
   openai_api_base = "<YOUR_EAS_SERVICE_ENDPOINT>/v1/"
   client = OpenAI(
       api_key=openai_api_key,
       base_url=openai_api_base,
   )
   models = client.models.list()
   model = models.data[0].id
   print(model)
   def main():
       stream = True
       chat_completion = client.chat.completions.create(
           messages=[
                {
                   "role": "system",
                   "content": [
                       {
                           "type": "text",
                           "text": "You are an expert in intent recognition. You can identify the user's intent based on their question and return the corresponding intent and parameters.",
                       }
                   ],
               },
               {
                   "role": "user",
                   "content": [
                       {
                           "type": "text",
                           "text": "I want to listen to music",
                       }
                   ],
               }
           ],
           model=model,
           max_completion_tokens=2048,
           stream=stream,
       )
       if stream:
           for chunk in chat_completion:
               print(chunk.choices[0].delta.content, end="")
       else:
           result = chat_completion.choices[0].message.content
           print(result)
   if __name__ == "__main__":
       main()
   

   Where:

   • <YOUR_EAS_SERVICE_ENDPOINT>: Replace with your service endpoint.

   • <YOUR_EAS_SERVICE_TOKEN>: Replace with your service token.