UI Automation - AgentBay - Alibaba Cloud Documentation Center

Overview

The Computer Use module provides UI automation for cloud computers across three categories:

Mouse operations - Click, move, drag, and scroll with pixel-level precision.
Keyboard operations - Type text and send key combinations.
Screen operations - Capture screenshots and retrieve screen dimensions.

Method summary

Category	Method	Description
Mouse	`click_mouse`	Click at specified coordinates.
Mouse	`move_mouse`	Move cursor without clicking.
Mouse	`drag_mouse`	Drag from one point to another.
Mouse	`scroll`	Scroll the mouse wheel at a location.
Mouse	`get_cursor_position`	Get current cursor coordinates.
Keyboard	`input_text`	Type a string at the cursor position.
Keyboard	`press_keys`	Send key combinations or shortcuts.
Keyboard	`release_keys`	Release previously held keys.
Screen	`screenshot`	Capture the screen and get a download URL.
Screen	`get_screen_size`	Get screen dimensions and DPI scaling.

Coordinate system

All operations use pixel coordinates with origin (0, 0) at the top-left corner. X increases rightward, Y increases downward. Use get_screen_size() to get screen dimensions and get_cursor_position() for current cursor location.

Result objects

Computer Use methods return one of two result types:

BoolResult (returned by mouse clicks, movement, scrolling, text input, and key operations):

Property	Type	Description
`success`	`bool`	Whether the operation completed successfully.
`data`	`bool` or `None`	The boolean result of the operation.
`error_message`	`str`	Error details if the operation failed. Empty string on success.

OperationResult (returned by screenshot, get_cursor_position, and get_screen_size):

Property	Type	Description
`success`	`bool`	Whether the operation completed successfully.
`data`	varies	Operation-specific data. For `get_cursor_position` and `get_screen_size`, this is a JSON string that you must parse with `json.loads()`. For `screenshot`, this is a download URL string.
`error_message`	`str`	Error details if the operation failed. Empty string on success.

Prerequisites

Ensure the following:

AgentBay SDK installed (pip install wuying-agentbay-sdk) and configured with valid credentials.
A supported cloud computer image is available (windows_latest or linux_latest).

Create a session

Create a session connected to a cloud computer before performing UI automation.

from agentbay import AgentBay, CreateSessionParams

agent_bay = AgentBay()
# Use windows_latest or linux_latest
session_params = CreateSessionParams(image_id="windows_latest")
session = agent_bay.create(session_params).session

Delete the session when finished to release resources:

agent_bay.delete(session)

All examples below assume an existing agent_bay instance and active session as shown above. Session creation and deletion code is omitted for brevity.

Mouse operations

click_mouse

Clicks at specified screen coordinates.

Signature

session.computer.click_mouse(x, y, button=MouseButton.LEFT)

Parameters

Parameter	Type	Required	Default	Description
`x`	`int`	Yes	-	Horizontal position in pixels.
`y`	`int`	Yes	-	Vertical position in pixels.
`button`	`MouseButton`	No	`MouseButton.LEFT`	Button to use.

MouseButton enum values

Import: from agentbay import MouseButton

Value	Description
`MouseButton.LEFT`	Standard left-click.
`MouseButton.RIGHT`	Right-click (context menu).
`MouseButton.MIDDLE`	Middle-click (scroll wheel button).
`MouseButton.DOUBLE_LEFT`	Double left-click.

Returns: A BoolResult object. Check result.success to confirm the click was registered.

Example

from agentbay import MouseButton

# Left-click (default button)
result = session.computer.click_mouse(x=500, y=300)
if result.success:
    print("Left-click successful")
# Output: Left-click successful

# Right-click
result = session.computer.click_mouse(x=500, y=300, button=MouseButton.RIGHT)
if result.success:
    print("Right-click successful")
# Output: Right-click successful

# Middle-click
result = session.computer.click_mouse(x=500, y=300, button=MouseButton.MIDDLE)
if result.success:
    print("Middle-click successful")
# Output: Middle-click successful

# Double left-click
result = session.computer.click_mouse(x=500, y=300, button=MouseButton.DOUBLE_LEFT)
if result.success:
    print("Double left-click successful")
# Output: Double left-click successful

move_mouse

Moves the cursor to specified coordinates without clicking.

Signature

session.computer.move_mouse(x, y)

Parameters

Parameter	Type	Required	Description
`x`	`int`	Yes	Horizontal position in pixels.
`y`	`int`	Yes	Vertical position in pixels.

Returns: A BoolResult object.

Example

result = session.computer.move_mouse(x=600, y=400)
if result.success:
    print("Mouse move successful")
# Output: Mouse move successful

drag_mouse

Drags from one point to another while holding the specified button.

Signature

session.computer.drag_mouse(from_x, from_y, to_x, to_y, button=MouseButton.LEFT)

Parameters

Parameter	Type	Required	Default	Description
`from_x`	`int`	Yes	-	Starting horizontal position in pixels.
`from_y`	`int`	Yes	-	Starting vertical position in pixels.
`to_x`	`int`	Yes	-	Ending horizontal position in pixels.
`to_y`	`int`	Yes	-	Ending vertical position in pixels.
`button`	`MouseButton`	No	`MouseButton.LEFT`	Button to hold during the drag.

Supported button values for drag: MouseButton.LEFT, MouseButton.RIGHT, MouseButton.MIDDLE.

MouseButton.DOUBLE_LEFT is not supported for drag operations.

Returns: A BoolResult object.

Example

from agentbay import MouseButton

result = session.computer.drag_mouse(
    from_x=100,
    from_y=100,
    to_x=200,
    to_y=200,
    button=MouseButton.LEFT
)
if result.success:
    print("Drag operation successful")
# Output: Drag operation successful

scroll

Scrolls the mouse wheel at a specified screen location.

Signature

session.computer.scroll(x, y, direction=ScrollDirection.UP, amount=1)

Parameters

Parameter	Type	Required	Default	Description
`x`	`int`	Yes	-	X position for the scroll, in pixels.
`y`	`int`	Yes	-	Y position for the scroll, in pixels.
`direction`	`ScrollDirection`	No	`ScrollDirection.UP`	The direction to scroll.
`amount`	`int`	No	`1`	Number of scroll increments.

ScrollDirection enum values

Import: from agentbay import ScrollDirection

Value	Description
`ScrollDirection.UP`	Scroll upward.
`ScrollDirection.DOWN`	Scroll downward.
`ScrollDirection.LEFT`	Scroll left.
`ScrollDirection.RIGHT`	Scroll right.

Returns: A BoolResult object.

Example

from agentbay import ScrollDirection

# Scroll up
result = session.computer.scroll(x=500, y=500, direction=ScrollDirection.UP, amount=3)
if result.success:
    print("Scroll up successful")
# Output: Scroll up successful

# Scroll down
result = session.computer.scroll(x=500, y=500, direction=ScrollDirection.DOWN, amount=5)
if result.success:
    print("Scroll down successful")
# Output: Scroll down successful

get_cursor_position

Returns the current cursor position.

Signature

session.computer.get_cursor_position()

Parameters: None.

Returns: An OperationResult object. When result.success is True, result.data contains a JSON string with x and y fields.

Example

import json

result = session.computer.get_cursor_position()
if result.success:
    cursor_data = json.loads(result.data)
    print(f"Cursor position: x={cursor_data['x']}, y={cursor_data['y']}")
# Output: Cursor position: x=512, y=384

Keyboard operations

input_text

Types text at the current cursor position.

Signature

session.computer.input_text(text)

Parameters

Parameter	Type	Required	Description
`text`	`str`	Yes	The text to type.

Returns: A BoolResult object.

Example

result = session.computer.input_text("Hello AgentBay!")
if result.success:
    print("Text input successful")
# Output: Text input successful

press_keys

Sends one or more keys simultaneously, including modifier keys. Use for shortcuts like Ctrl+C or Alt+Tab.

Signature

session.computer.press_keys(keys, hold=False)

Parameters

Parameter	Type	Required	Default	Description
`keys`	`list[str]`	Yes	-	A list of key names to press simultaneously.
`hold`	`bool`	No	`False`	When set to `True`, the keys are held down instead of being pressed and released. You must call `release_keys()` afterward.

Returns: A BoolResult object.

Example

# Press Ctrl+A to select all
result = session.computer.press_keys(keys=["Ctrl", "a"])
if result.success:
    print("Key press successful")
# Output: Key press successful

# Press Ctrl+C to copy
result = session.computer.press_keys(keys=["Ctrl", "c"])
if result.success:
    print("Copy command sent")
# Output: Copy command sent

release_keys

Releases keys previously held with press_keys(hold=True). Always release held keys to prevent interference with subsequent operations.

Signature

session.computer.release_keys(keys)

Parameters

Parameter	Type	Required	Description
`keys`	`list[str]`	Yes	A list of key names to release.

Returns: A BoolResult object.

Example

# Hold down the Ctrl key
session.computer.press_keys(keys=["Ctrl"], hold=True)

# ... Perform other operations ...

# Release the Ctrl key
result = session.computer.release_keys(keys=["Ctrl"])
if result.success:
    print("Key release successful")
# Output: Key release successful

Screen operations

screenshot

Captures the screen and returns a download URL. The image is saved to Object Storage Service (OSS).

Signature

session.computer.screenshot()

Parameters: None.

Returns: An OperationResult object. When result.success is True, result.data contains the download URL for the screenshot image (not the raw image data).

Example

result = session.computer.screenshot()
if result.success:
    screenshot_url = result.data
    print(f"Screenshot URL: {screenshot_url}")
# Output: Screenshot URL: https://***.***.aliyuncs.com/***/screenshot_1234567890.png?***

get_screen_size

Returns screen dimensions and the display scale factor of the cloud computer.

Signature

session.computer.get_screen_size()

Parameters: None.

Returns: An OperationResult object. When result.success is True, result.data contains a JSON string with the following fields:

Field	Type	Description
`width`	`int`	Screen width in pixels.
`height`	`int`	Screen height in pixels.
`dpiScalingFactor`	`float`	The display scale factor (DPI scaling). A value of `1.0` means 100% scaling (96 DPI).

Example

import json

result = session.computer.get_screen_size()
if result.success:
    screen_data = json.loads(result.data)
    print(f"Screen width: {screen_data['width']}")
    print(f"Screen height: {screen_data['height']}")
    print(f"DPI scaling factor: {screen_data['dpiScalingFactor']}")
# Output: Screen width: 1024
# Output: Screen height: 768
# Output: DPI scaling factor: 1.0

Troubleshooting

"Tool not found" error

Symptom: A "Tool not found" error occurs when calling a Computer Use method.

Cause: The session is not running on a supported cloud computer image.

Solution: Set image_id to a supported value (windows_latest or linux_latest) when creating a session.

# Correct - use a supported image ID
session_params = CreateSessionParams(image_id="windows_latest")

Screenshot returns a URL, not image data

Symptom: result.data from screenshot() contains a URL string instead of raw image bytes.

Cause: Expected behavior. The screenshot() method stores images in OSS and returns a download URL.

Solution: Download the image from the returned URL using an HTTP client or browser.

Held keys not released

Symptom: Subsequent keyboard operations behave unexpectedly after using press_keys(hold=True).

Cause: Keys held with press_keys(hold=True) remain active until explicitly released.

Solution: Pair every press_keys(hold=True) call with a corresponding release_keys() call.

# Hold a key
session.computer.press_keys(keys=["Shift"], hold=True)

# Perform operations that need Shift held...

# Always release afterward
session.computer.release_keys(keys=["Shift"])

Coordinates out of bounds

Symptom: A mouse operation produces unexpected results or no visible effect.

Cause: Target coordinates may exceed screen bounds. The SDK does not validate coordinates client-side — they are sent directly to the cloud computer.

Solution: Call get_screen_size() to determine valid bounds, then ensure x and y stay within (0, 0) to (width, height).