Mobile Use Android - AgentBay

開放能力概覽

工具	名稱	描述	支援說明
System	`create_sandbox`	建立一個新的AgentBay沙箱並返回其ID。	支援
	`get_sandbox_url`	當使用者需要訪問無影MCP運行時，用於擷取其URL的命令。每次擷取的URL僅單次有效，使用後即失效。	支援
	`system_screenshot`	捕獲當前顯示螢幕的全屏截圖，並返回一個可共用的URL。截圖會自動經過處理並安全儲存。為保障安全，產生的URL將在64分鐘後到期。	支援
	`kill_sandbox`	任務完成後釋放資源。	支援
	`shell`	在Android平台上執行shell命令，支援設定逾時，執行完成後返回命令輸出或錯誤資訊。	支援
UI	`click`	在螢幕的特定座標處點擊。	支援
	`input_text`	輸入文本。	支援
	`send_key`	發送按鍵。支援 Android 平台按鍵： 3:HOME 4:BACK 24:VOLUME_UP 25:VOLUME_DOWN 26:POWER 82:MENU	支援
	`swipe`	在螢幕上執行滑動手勢。	支援
	`get_all_ui_elements`	在逾時時間內擷取裝置上的所有 UI 元素（包括非互動元素）。	支援
	`get_clickable_ui_elements`	在逾時時間內擷取所有可點擊的 UI 元素。	支援
App	`get_installed_apps`	檢索系統中安裝的應用程式列表。支援通過開始菜單條目和案頭捷徑篩選，並可選擇排除系統應用。返回應用程式詳細資料，包括名稱、啟動命令、可選的停止命令和工作目錄。	支援
	`start_app`	使用提供的命令和可選工作目錄啟動指定應用程式。返回與啟動應用程式關聯的進程列表，包括進程名稱、PID 和啟動命令。	支援
	`stop_app_by_cmd`	使用提供的停止命令終止應用程式。請謹慎使用，因為這將強制終止指定進程。	支援
Shell	`shell`	在 Android 平台上執行 shell 命令並返回輸出或錯誤。	支援

System

名稱	描述	參數
`create_sandbox`	建立一個新的AgentBay沙箱並返回其ID。	`{ "type": "object", "properties": {}, "required": [] }`
`get_sandbox_url`	當使用者需要訪問無影MCP運行時，用於擷取其URL的命令。每次擷取的URL僅單次有效，使用後即失效。	`{ "type": "object", "properties": { "sandbox_id": { "type": "string", "title": "Sandbox Id", "description": "The sandbox ID is the identifier for the tool execution environment. This sandbox_id comes from the create_sandbox tool." } }, "required": [ "sandbox_id" ] }`
`system_screenshot`	捕獲當前顯示螢幕的全屏截圖，並返回一個可共用的URL。截圖會自動經過處理並安全儲存。為保障安全，產生的URL將在64分鐘後到期。	`{ "type": "object", "properties": { "sandbox_id": { "type": "string", "title": "Sandbox Id", "description": "The sandbox ID is the identifier for the tool execution environment. This sandbox_id comes from the create_sandbox tool." } }, "required": [ "sandbox_id" ] }`
`kill_sandbox`	任務完成後釋放資源。	`{ "type": "object", "properties": { "sandbox_id": { "type": "string", "title": "Sandbox Id", "description": "The sandbox ID is the identifier for the tool execution environment. This sandbox_id comes from the create_sandbox tool." } }, "required": [ "sandbox_id" ] }`
`shell`	在Android平台上執行shell命令，支援設定逾時，執行完成後返回命令輸出或錯誤資訊。	{ "type": "object", "required": [ "sandbox_id", "command", "timeout_ms" ], "properties": { "sandbox_id": { "type": "string", "title": "Sandbox Id", "description": "The sandbox ID is the identifier for the tool execution environment. This sandbox_id comes from the create_sandbox tool." }, "timeout_ms": { "default": 1000, "description": "Command execution timeout (unit: milliseconds). If not specified, the default value (such as 1000 milliseconds) is used", "type": "integer" }, "command": { "description": "client input command", "type": "string" } } }

UI

名稱	描述	參數
`click`	在螢幕的特定座標處點擊。	`{ "inputSchema": { "properties": { "button": { "description": "按鈕類型。可選值：left, middle, right，預設：left", "type": "string" }, "x": { "description": "X 座標", "type": "integer" }, "y": { "description": "Y 座標", "type": "integer" } }, "required": ["x", "y", "button"], "type": "object" } }`
`input_text`	輸入文本。	`{ "inputSchema": { "properties": { "text": { "description": "用戶端輸入的文本", "type": "string" } }, "required": ["text"], "type": "object" } }`
`send_key`	發送按鍵。支援 Android 平台按鍵： 3:HOME 4:BACK 24:VOLUME_UP 25:VOLUME_DOWN 26:POWER 82:MENU	`{ "inputSchema": { "properties": { "key": { "description": "用戶端發送的按鍵", "type": "integer" } }, "required": ["key"], "type": "object" } }`
`swipe`	在螢幕上執行滑動手勢。	{ "inputSchema": { "properties": { "duration_ms": { "default": 300, "description": "滑動期間（毫秒），預設：300", "type": "integer" }, "end_x": { "description": "結束 X 座標", "type": "integer" }, "end_y": { "description": "結束 Y 座標", "type": "integer" }, "start_x": { "description": "起始 X 座標", "type": "integer" }, "start_y": { "description": "起始 Y 座標", "type": "integer" } }, "required": ["start_x", "start_y", "end_x", "end_y", "duration_ms"], "type": "object" } }
`get_all_ui_elements`	在逾時時間內擷取裝置上的所有 UI 元素（包括非互動元素）。	`{ "inputSchema": { "properties": { "timeout_ms": { "default": 1000, "description": "命令執行逾時時間（單位：毫秒）。若未指定，預設值（例如 1000 毫秒）", "type": "integer" } }, "required": ["timeout_ms"], "type": "object" } }`
`get_clickable_ui_elements`	在逾時時間內擷取所有可點擊的 UI 元素。	`{ "inputSchema": { "properties": { "timeout_ms": { "default": 1000, "description": "命令執行逾時時間（單位：毫秒）。若未指定，預設值（例如 1000 毫秒）", "type": "integer" } }, "required": ["timeout_ms"], "type": "object" } }`

App

名稱	描述	參數
`get_installed_apps`	檢索系統中安裝的應用程式列表。支援通過開始菜單條目和案頭捷徑篩選，並可選擇排除系統應用。返回應用程式詳細資料，包括名稱、啟動命令、可選的停止命令和工作目錄。	`{ "inputSchema": { "properties": { "desktop": { "default": false, "description": "包含案頭捷徑（預設：false）", "type": "boolean" }, "ignore_system_app": { "default": true, "description": "排除系統應用（預設：true）", "type": "boolean" }, "start_menu": { "default": true, "description": "包含開始菜單應用（預設：true）", "type": "boolean" } }, "required": [], "type": "object" } }`
`start_app`	使用提供的命令和可選工作目錄啟動指定應用程式。返回與啟動應用程式關聯的進程列表，包括進程名稱、PID 和啟動命令。	{ "inputSchema": { "properties": { "activity": { "description": "Android 上的活動名稱，例如 .xxActivity 或 com.xx/.xxActivity。", "type": "string" }, "start_cmd": { "description": "啟動應用程式的命令。在 Android 上，使用 'monkey -p ' 文法（例如：'monkey -p <package_name> -c android.intent.category.LAUNCHER 1'）。", "type": "string" }, "work_directory": { "default": "", "description": "啟動應用程式的工作目錄。若省略，將使用預設目錄。", "type": "string" } }, "required": ["start_cmd"], "type": "object" } }
`stop_app_by_cmd`	使用提供的停止命令終止應用程式。請謹慎使用，因為這將強制終止指定進程。	`{ "inputSchema": { "properties": { "stop_cmd": { "description": "用於終止應用程式的命令。", "type": "string" } }, "required": ["stop_cmd"], "type": "object" } }`

Shell

名稱

描述

參數

`shell`

在 Android 平台上執行 shell 命令並返回輸出或錯誤。

{
  "inputSchema": {
    "properties": {
      "command": {
        "description": "用戶端輸入的命令",
        "type": "string"
      },
      "timeout_ms": {
        "default": 1000,
        "description": "命令執行逾時時間（單位：毫秒）。若未指定，預設值（例如 1000 毫秒）",
        "type": "integer"
      }
    },
    "required": ["command", "timeout_ms"],
    "type": "object"
  }
}

MCP Tool List

{
  "wuying_oss": {
    "tools": [
      {
        "name": "oss_env_init",
        "description": "Create and initialize OSS environment variables with the specified endpoint, access key ID, access key secret, security token, and region. The temporary security credentials obtained from the STS (Security Token Service). For more information, see: https://www.alibabacloud.com/help/zh/oss/developer-reference/use-temporary-access-credentials-provided-by-sts-to-access-oss?spm=a2c4g.11186623.help-menu-search-31815.d_1#9ab17afd7cs4t .",
        "inputSchema": {
          "properties": {
            "access_key_id": {
              "description": "The Access Key ID for OSS authentication",
              "type": "string"
            },
            "access_key_secret": {
              "description": "The Access Key Secret for OSS authentication",
              "type": "string"
            },
            "endpoint": {
              "description": "The OSS service endpoint, e.g.,  If not specified, the default is https://oss-cn-hangzhou.aliyuncs.com",
              "type": "string"
            },
            "region": {
              "description": "The OSS region, e.g., cn-hangzhou. If not specified, the default is cn-hangzhou",
              "type": "string"
            },
            "security_token": {
              "description": "The Security Token for OSS authentication",
              "type": "string"
            }
          },
          "required": [
            "access_key_id",
            "access_key_secret",
            "security_token"
          ],
          "type": "object"
        }
      },
      {
        "name": "oss_upload",
        "description": "Upload a local file or directory to the specified OSS bucket. If a directory is specified, it will be compressed into a ZIP file before uploading. The object name in OSS can be specified; if not, the file or ZIP name will be used by default. Note: You must call the oss_env_init tool to initialize OSS environment variables before using this tool.",
        "inputSchema": {
          "properties": {
            "bucket": {
              "description": "OSS bucket name",
              "type": "string"
            },
            "object": {
              "description": "Object path in OSS bucket, e.g., test/test.txt",
              "type": "string"
            },
            "path": {
              "description": "Local file or not empty directory full path to upload, e.g., /tmp/test.txt /tmp on Linux or C:/tmp/test.txt C:/tmp on Windows",
                            "type": "string"
                        }
                    },
                    "required": [
                        "bucket",
                        "object",
                        "path"
                    ],
                    "type": "object"
                }
            },
            {
                "name": "oss_download",
                "description": "Download an object from the specified OSS bucket to the given local path. If the parent directory does not exist, it will be created automatically. If the target file already exists, it will be overwritten. Note: You must call the oss_env_init tool to initialize OSS environment variables before using this tool.",
                "inputSchema": {
                    "properties": {
                        "bucket": {
                            "description": "OSS bucket name",
                            "type": "string"
                        },
                        "object": {
                            "description": "Object path in OSS bucket, e.g., test/test.txt",
                            "type": "string"
                        },
                        "path": {
                            "description": "Local full path to save the downloaded file, e.g., /tmp/test.txt on Linux or C:/tmp/test.txt on Windows",
                            "type": "string"
                        }
                    },
                    "required": [
                        "bucket",
                        "object",
                        "path"
                    ],
                    "type": "object"
                }
            },
            {
                "name": "oss_upload_annon",
                "description": "Upload a local file or directory to the specified URL using HTTP PUT. If a directory is specified, it will be compressed into a ZIP file before uploading. If the upload target already exists, it will be overwritten.",
                "inputSchema": {
                    "properties": {
                        "path": {
                            "description": "Local file or not empty directory full path to upload, e.g., /tmp/test.txt /tmp on Linux or C:/tmp/test.txt  C:/tmp on Windows",
                            "type": "string"
                        },
                        "url": {
                            "description": "The HTTP/HTTPS URL to upload the file to",
                            "type": "string"
                        }
                    },
                    "required": [
                        "url",
                        "path"
                    ],
                    "type": "object"
                }
            },
            {
                "name": "oss_download_annon",
                "description": "Download a file from the specified URL to the given local path. If the parent directory does not exist, it will be created automatically. If the target file already exists, it will be overwritten.",
                "inputSchema": {
                    "properties": {
                        "path": {
                            "description": "The full local file path to save the downloaded file, e.g., /tmp/test.txt on Linux or C:/tmp/test.txt on Windows",
                            "type": "string"
                        },
                        "url": {
                            "description": "The HTTP/HTTPS URL to download the file from",
                            "type": "string"
                        }
                    },
                    "required": [
                        "url",
                        "path"
                    ],
                    "type": "object"
                }
            }
        ]
    },
    "wuying_ui": {
        "tools": [
            {
                "name": "click",
                "description": "Click on the screen at specific coordinates.",
                "inputSchema": {
                    "properties": {
                        "button": {
                            "description": "button type,Available values:left,middle,right,default:left",
                            "type": "string"
                        },
                        "x": {
                            "description": "X coordinate",
                            "type": "integer"
                        },
                        "y": {
                            "description": "Y coordinate",
                            "type": "integer"
                        }
                    },
                    "required": [
                        "x",
                        "y",
                        "button"
                    ],
                    "type": "object"
                }
            },
            {
                "name": "input_text",
                "description": "Input text",
                "inputSchema": {
                    "properties": {
                        "text": {
                            "description": "client input text",
                            "type": "string"
                        }
                    },
                    "required": [
                        "text"
                    ],
                    "type": "object"
                }
            },
            {
                "name": "send_key",
                "description": "Send a key.Supported Keys on Android Platform: 3:HOME,4:BACK,24:VOLUME_UP,25:VOLUME_DOWN,26:POWER,82:MENU",
                "inputSchema": {
                    "properties": {
                        "key": {
                            "description": "client send key",
                            "type": "integer"
                        }
                    },
                    "required": [
                        "key"
                    ],
                    "type": "object"
                }
            },
            {
                "name": "swipe",
                "description": "Perform a swipe gesture on the screen.",
                "inputSchema": {
                    "properties": {
                        "duration_ms": {
                            "default": 300,
                            "description": "Duration of swipe in milliseconds,default:300",
                            "type": "integer"
                        },
                        "end_x": {
                            "description": "Ending X coordinate",
                            "type": "integer"
                        },
                        "end_y": {
                            "description": "Ending Y coordinate",
                            "type": "integer"
                        },
                        "start_x": {
                            "description": "Starting X coordinate",
                            "type": "integer"
                        },
                        "start_y": {
                            "description": "Starting Y coordinate",
                            "type": "integer"
                        }
                    },
                    "required": [
                        "start_x",
                        "start_y",
                        "end_x",
                        "end_y",
                        "duration_ms"
                    ],
                    "type": "object"
                }
            },
            {
                "name": "get_all_ui_elements",
                "description": "Get all UI elements from the device with timeout, including non-interactive elements.",
                "inputSchema": {
                    "properties": {
                        "timeout_ms": {
                            "default": 1000,
                            "description": "Command execution timeout (unit: milliseconds). If not specified, the default value (such as 1000 milliseconds) is used",
                            "type": "integer"
                        }
                    },
                    "required": [
                        "timeout_ms"
                    ],
                    "type": "object"
                }
            },
            {
                "name": "get_clickable_ui_elements",
                "description": "Get all clickable UI elements within timeout.",
                "inputSchema": {
                    "properties": {
                        "timeout_ms": {
                            "default": 1000,
                            "description": "Command execution timeout (unit: milliseconds). If not specified, the default value (such as 1000 milliseconds) is used",
                            "type": "integer"
                        }
                    },
                    "required": [
                        "timeout_ms"
                    ],
                    "type": "object"
                }
            }
        ]
    },
    "wuying_app": {
        "tools": [
            {
                "name": "get_installed_apps",
                "description": "Retrieve a list of installed applications on the system. Supports filtering by Start Menu entries and Desktop shortcuts, with an option to exclude system applications. Returns application details including name, start command, optional stop command, and working directory.",
                "inputSchema": {
                    "properties": {
                        "desktop": {
                            "default": false,
                            "description": "Include Desktop shortcuts (default: false)",
                            "type": "boolean"
                        },
                        "ignore_system_app": {
                            "default": true,
                            "description": "Exclude system applications (default: true)",
                            "type": "boolean"
                        },
                        "start_menu": {
                            "default": true,
                            "description": "Include Start Menu applications (default: true)",
                            "type": "boolean"
                        }
                    },
                    "required": [],
                    "type": "object"
                }
            },
            {
                "name": "start_app",
                "description": "Start a specified application using the provided command and optional working directory. Returns a list of processes associated with the launched application, including their process names, PIDs, and startup commands.\n\n",
                "inputSchema": {
                    "properties": {
                        "activity": {
                            "description": "The activity name on android,like .xxActivity or com.xx/.xxActivity.",
                            "type": "string"
                        },
                        "start_cmd": {
                            "description": "The command to start the application. On Android, use the 'monkey -p ' syntax (e.g., 'monkey -p <package_name> -c android.intent.category.LAUNCHER 1').",
                            "type": "string"
                        },
                        "work_directory": {
                            "default": "",
                            "description": "The directory from which the application should be launched. If omitted, the default directory is used.",
                            "type": "string"
                        }
                    },
                    "required": [
                        "start_cmd"
                    ],
                    "type": "object"
                }
            },
            {
                "name": "stop_app_by_cmd",
                "description": "Terminate an application using the provided stop command. Use with caution as this will forcefully terminate the specified process.",
                "inputSchema": {
                    "properties": {
                        "stop_cmd": {
                            "description": "The command used to terminate the application.",
                            "type": "string"
                        }
                    },
                    "required": [
                        "stop_cmd"
                    ],
                    "type": "object"
                }
            }
        ]
    },
    "wuying_shell": {
        "tools": [
            {
                "name": "shell",
                "description": "Executes an shell command with timeout and returns the output or an error on android platform.",
                "inputSchema": {
                    "properties": {
                        "command": {
                            "description": "client input command",
                            "type": "string"
                        },
                        "timeout_ms": {
                            "default": 1000,
                            "description": "Command execution timeout (unit: milliseconds). If not specified, the default value (such as 1000 milliseconds) is used",
                            "type": "integer"
                        }
                    },
                    "required": [
                        "command",
                        "timeout_ms"
                    ],
                    "type": "object"
                }
            }
        ]
    }
}

AgentBay：Mobile Use

開放能力概覽

System

`create_sandbox`

`get_sandbox_url`

`system_screenshot`

`kill_sandbox`

`shell`

UI

`click`

`input_text`

`send_key`

`swipe`

`get_all_ui_elements`

`get_clickable_ui_elements`

App

`get_installed_apps`

`start_app`

`stop_app_by_cmd`

Shell

`shell`

MCP Tool List