Desktop Control

• Automating repetitive GUI workflows — filling out forms, navigating menus, or batch-processing files across applications that lack an API.
• Building desktop test scripts — simulating user input and capturing screenshots to verify application state.
• Orchestrating multi-app pipelines — switching between windows, copying data from one app, and pasting it into another.
• Screen monitoring and capture — taking timed or region-specific screenshots, detecting on-screen elements via image recognition, or reading pixel colors for state detection.

• Headless or server environments — this skill requires a visible display; it will not work on machines without a desktop session.
• Applications that block simulated input — certain games, DRM-protected apps, and secure login screens actively reject synthetic mouse/keyboard events.
• High-frequency, low-latency automation — if you need sub-millisecond precision or are automating thousands of actions per second, a native OS API or dedicated RPA platform will be more reliable.

Move the cursor to absolute screen coordinates or relative offsets, with optional smooth bezier-curve paths that mimic natural human movement. Supports left, right, and middle clicks, double/triple clicks, drag-and-drop, and both vertical and horizontal scrolling.

Type text at any speed from instant to human-like WPM, execute multi-key hotkeys (Ctrl+C, Win+R, etc.), press special and function keys, and manually hold or release modifier keys for complex interactions like multi-file selection.

Capture the full screen or any rectangular region and save the result as a PNG. Optionally use OpenCV to locate a template image anywhere on screen with a configurable confidence threshold, enabling element detection without hardcoded coordinates.

List all open windows, activate any window by partial title match, retrieve the currently focused window, and read window position, size, and title — making it straightforward to orchestrate multi-application workflows.

A failsafe mode aborts automation when the mouse reaches any screen corner. An approval mode prompts for user confirmation before each action. Bounds checking prevents out-of-screen operations, and all actions are logged for auditing.

Programmatically write text to the clipboard or read its current contents, enabling seamless data transfer between applications without simulating keyboard shortcuts.

An agent activates a target application window, clicks into each form field in sequence, types the appropriate values at human-like speed, and submits — replicating a user filling out a complex web or desktop form without any app API access.

After triggering an application action, capture a specific screen region to save as a timestamped PNG, then use image recognition to confirm that the expected button or dialog is visible before proceeding.

Hold Ctrl and click multiple files in a file manager to select them, then drag the selection to a destination folder — all scripted as a single reproducible automation sequence.

Activate a source application, select and copy data using keyboard shortcuts, switch to a destination application via window activation, and paste — automating a workflow that would otherwise require manual copy-paste across programs.