# Desktop Automation

> Drive the mouse, keyboard and windows and read the screen the way a real user would (pyautogui-based).

| | |
|---|---|
| Version | 1.0.0 |
| Category | System |
| Permissions | process.spawn, shell.exec, filesystem.read, filesystem.write |
| Depends on | ai-chat |

# Desktop Automation

Dieu khien chuot/ban phim/cua so va nhan dien man hinh nhu nguoi dung that (pyautogui) - mo public API + MCP cho plugin/agent khac

**Type:** `backend` — see `.claude/skills/kazuo-create-plugin/references/plugin-examples.md`

## Files

- `manifest.json` — metadata, `defaultSettings`, `defaultWidgetConfig`
- `widget.html` / `js/widget.js` / `css/widget.css` — main UI
  - has its own "⚙" button + settings overlay built into widget.html/widget.js (host no longer provides a per-widget gear icon — see references/plugin-runtime-api.md section 1b)
- `settings.html` / `js/settings.js` / `css/settings.css` — same fields as the widget overlay (intentionally duplicated), used by the Plugin Manager screen and the pop-out window
- `locales/en.json5` / `locales/vi.json5` — UI strings, `widget.*`/`settings.*` namespaces
- `main/index.ts` (+ compiled `main/index.js`) — runs in the Electron main process; manages the python daemon, exposes IPC (private + public) and `getMcpTools` (ai-chat AI agent, Dev Mode only)
- `python/da_agent.py` — long-lived automation daemon (pyautogui/pygetwindow), JSON-line protocol over stdin/stdout
- `package.json` — declares `zod` (only used by the MCP schema; host runs `npm install --omit=dev` on install)

## Default settings

- `pythonPath` — default `""` → uses `python` (Windows) / `python3` on PATH
- `failsafe` — default `true` (slam mouse to a screen corner to abort)
- `actionPause` — default `0` (seconds between pyautogui actions)

## Python dependencies (auto-installed)

The backend checks for `pyautogui pillow opencv-python pygetwindow` **on enable** and, if any are
missing, runs `python -m pip install …` automatically (with the configured interpreter) and restarts
the daemon. The widget shows an "Install deps" bar you can also trigger manually.

Manual equivalent, if you prefer:

```
pip install pyautogui pillow opencv-python pygetwindow
```

`opencv-python` is optional — without it `locate` still works but `confidence` matching is off. If the
python interpreter is a system-wide install, pip may need admin rights (the log will say so).

## Public API (for other plugins / agents)

```js
await window['kazuo-ai'].invoke('plugin:desktop-automation:public:exec', 'caller-id',
  'click', { x: 500, y: 300, button: 'left', clicks: 2 });
await window['kazuo-ai'].invoke('plugin:desktop-automation:public:capabilities'); // list commands
```

Commands: move / move_rel / click / double_click / right_click / mouse_down / mouse_up / drag /
scroll / type / paste / press / hotkey / key_down / key_up / pixel / screenshot / locate / locate_all /
windows / active_window / window_find / activate / minimize / maximize / restore / close_window /
move_window / resize_window / position / size / probe.

## Test

Open Kazuo AI → Plugin Manager — this plugin appears automatically → Install → Enable → click **Probe**
in the widget. If it reports `pyautogui missing`, run the pip command above.

After editing `main/index.ts`, recompile before testing: `npx tsc -p tsconfig.plugins.json` (run from the project root).