I wanted to solve a more practical problem: how can one AI conversation safely operate the Windows computers where the actual work lives?

That became YourHand - an open-source Windows control layer that connects supported AI clients to one or more authorized Windows PCs.

The goal is simple:

One AI chat should be able to work across the computers you already use.

That means opening applications, reading the screen, clicking and typing, managing files, running commands, and coordinating tasks across multiple machines - without forcing everything into a browser-only workflow.

Why I built it

My work is distributed across several Windows machines. Different PCs have different applications, project files, sessions, and workloads.

A normal AI assistant can explain what to do, but I still have to switch machines and execute each step manually.

I wanted the interaction to feel more like this:

Me: Open Revit on PC-075 and check whether the project is already running.
AI: [checks the device, opens the app if needed, reports the result]

Me: Now open Chrome on BeDo-Tower and test the latest build.
AI: [works on the second machine without losing the first context]

The important part is not the mouse click itself. The important part is persistent, account-scoped access to multiple real Windows computers from the same conversation.

The architecture

The system is split into a few layers:

AI client
   
YourHand MCP / plugin
   
Control plane
   
Capability router
   
Windows agent
   
OS / browser / UI / files / commands

The control plane handles account identity, device ownership, sharing, routing, and request coordination.

Each Windows machine runs a lightweight agent. The agent exposes capabilities such as:

• window and application control
• keyboard and mouse input
• screen observation
• file access
• command execution
• browser control
• device health and telemetry

The agent is designed to run quietly in the background. The user should not need to keep a terminal window open just to make the computer controllable.

Why a capability router matters

One of the biggest lessons from building this is that desktop automation should not depend on a single technique.

A click can be executed through several possible paths:

1. direct OS or application APIs
2. browser CDP when the target is a web page
3. semantic Windows UI Automation
4. pixel-based interaction
5. foreground input as a fallback
6. vision-assisted fallback when the UI cannot be addressed semantically

So instead of treating "click this button" as one primitive, YourHand routes the task to the most appropriate capability.

That is more robust than relying entirely on screenshots and blind cursor movement.

Multi-device was a first-class requirement

A lot of automation tools assume one session, one machine, one active task.

That was not enough for what I wanted.

YourHand treats devices as explicit resources tied to the user's account. A conversation can target different computers by name, and the control plane routes each request to the correct agent.

That also makes sharing important.

The design goal is that one Windows agent can remain installed while access is granted to another authorized account. The device does not need a second competing agent just because another team member needs access.

Authentication without handing over passwords

I did not want users copying AI-provider passwords or long-lived manual tokens into the desktop agent.

The current flow uses Google sign-in and OAuth-based authorization for the YourHand account. Devices are then paired to that account and can be explicitly shared.

The separation is important:

• the AI client gets authorization to call YourHand
• YourHand decides which devices that account may use
• the Windows agent only accepts work routed through the control plane

Observability matters more than I expected

Desktop automation fails in messy ways.

A command can time out even though the action happened. A window can disappear between discovery and input. A browser can be alive while the target tab is not. A device can reconnect with a stale socket.

Because of that, I added telemetry around calls: success/failure, execution method, latency, retries, and device health.

This has been one of the most valuable parts of the project. Without observability, every failure looks like "the AI clicked the wrong thing." With telemetry, you can distinguish routing failures, UI failures, transport failures, and application failures.

Open source, but not just a demo

I published a community repository because I want the Windows/AI control layer to be inspectable and extensible.

The project is still in beta, but the target is not a toy demo. The architecture is being built around:

• multiple Windows devices
• multiple authorized users
• account isolation
• browser and native UI control
• background execution
• recovery after restarts
• device telemetry
• controlled sharing

The hard part is not making a mouse move once. The hard part is making the whole path reliable enough that a user can trust the system across real machines.

What I am testing now

The current beta is focused on three things:

1. Reliability - fewer fragile UI assumptions and better recovery.
2. Speed - choosing direct APIs or semantic automation before falling back to screenshots.
3. Multi-device workflows - keeping one conversation useful across several computers.

I am also testing the onboarding flow for people who have never configured an MCP server or a Windows automation agent before.

Try it

YourHand is currently free during beta.

Website: https://yourhand.wolvexai.com/

Open-source community repository: https://github.com/archahmedzaki/YourHand-Community

Technical write-up on DEV: https://dev.to/archahmedzak/i-built-an-open-source-bridge-that-lets-one-ai-chat-work-across-multiple-windows-pcs-1m5l

I would especially like feedback from people who build developer tools, IT automation, MCP integrations, or Windows desktop software.

What would make you trust an AI chat enough to let it work across your real computers?