ScreenControl

🖥️ ScreenControl -- Cross-Platform AI Desktop Automation via MCP

Long-Form Project Summary for KNWS Showcase

ScreenControl is a cross-platform AI desktop automation system that gives large language models direct, tool-based control over any connected computer's screen, keyboard, mouse, filesystem, shell, and browser. Built around the Model Context Protocol (MCP), it provides 90+ tools that LLMs can invoke to interact with desktops as naturally as a human operator would -- clicking buttons, reading screens via OCR, typing text, managing files, running shell commands, and automating browser workflows.

The system spans five major codebases: a native macOS application (Objective-C), a C++ service for Linux and Windows headless operation, a C#/.NET 8 tray application for Windows GUI, a Next.js 16 control server with Stripe billing and agent management, and browser extensions for Firefox, Chrome, and Safari. It supports both local stdio connections (for Claude Code) and remote SSE/WebSocket connections (for cloud-hosted LLMs), making it deployable from a single developer's laptop to a fleet of managed enterprise machines.

Version: 2.0.20 Live at: https://screencontrol.knws.co.uk GitHub: github.com/datagram1/screen_control

---

🚀 Project Goals

1. Enable any MCP-compatible LLM to control a desktop computer through a standardised set of tools.
2. Support all major operating systems -- macOS, Windows, and Linux -- with native implementations on each.
3. Provide 90+ automation tools spanning screen interaction, filesystem operations, shell access, and browser control.
4. Offer both local and remote connection modes -- stdio for local Claude Code usage, SSE/WebSocket for remote cloud connections.
5. Build a commercial-grade control server with user accounts, Stripe billing, agent licensing, and a management dashboard.
6. Include browser extensions for deep web automation beyond what screen-level tools can achieve.
7. Create a Rescue Boot system for recovering machines via USB boot with pre-installed ScreenControl agents.
8. Maintain production quality -- auto-updates, crash reporting, signed binaries, and enterprise-grade security.

---

🧠 Core Features

1. Screenshot Grid with OCR

The primary visual interaction tool:

• Takes a screenshot of any window or the full desktop
• Overlays a labelled grid (A-T columns, 1-15 rows)
• Runs Tesseract OCR to detect all visible text with coordinates
• Returns both the grid image and a structured element list with {text, centerX, centerY, cell}
• LLMs can then click by grid cell reference, element text, or pixel coordinates
• Multi-monitor aware -- handles different DPI scales and display arrangements

This grid-based approach gives LLMs spatial awareness of the screen without requiring computer vision models.

---

2. Mouse and Keyboard Control

Full input simulation:

• Click -- left, right, middle button at absolute, relative, or grid-referenced positions
• Double-click -- for file opening, text selection
• Drag -- from point A to point B for drag-and-drop operations
• Scroll -- directional scrolling with configurable amounts
• Type text -- string input at the current cursor position
• Press keys -- individual key presses including modifiers (Cmd, Ctrl, Shift, Alt)
• Keyboard shortcuts -- compound key combinations (e.g., Cmd+C, Ctrl+Shift+T)
• Hover -- mouse positioning without clicking

All input methods auto-focus the target window before acting, ensuring clicks land on the intended application.

---

3. Application Management

Process and window control:

• List running applications with bundle IDs and window bounds
• Launch applications by name or bundle ID
• Focus applications -- bring to front across monitors and Spaces
• Close applications -- graceful or force quit
• Window listing -- enumerate all open windows with titles and positions

---

4. Filesystem Operations

Complete file management:

• Read/write files -- text and binary, with line range support for large files
• Search files -- glob pattern matching across directory trees
• Grep -- regex content search with match context
• Patch -- structured file modifications
• Move, delete, create directories -- full filesystem manipulation
• Clipboard -- read and write system clipboard contents

---

5. Shell Access

Command execution:

• One-shot execution -- run a command and capture output
• Interactive sessions -- start long-running processes, send input, read output
• Working directory and environment variable control
• Timeout management -- prevent runaway processes
• Session lifecycle -- start, interact, stop with signal control

---

6. Browser Automation

Deep web interaction via browser extensions:

• Navigate to URLs
• Click elements by CSS selector, text content, or accessibility role
• Fill forms -- text inputs, checkboxes, dropdowns, radio buttons
• Read page text -- extract visible content
• Manage tabs -- open, close, switch between tabs
• Execute JavaScript -- run arbitrary scripts in page context
• Wait for elements -- polling for dynamic content

Extensions are available for Firefox, Chrome, and Safari (via Xcode conversion).

---

7. Agent Licensing and Auto-Update

Commercial deployment features:

• Each agent authenticates with the control server using a licence key
• Licence validation on connection, with expiry and tier enforcement
• Auto-update system -- agents check for new versions and update in place
• Version tracking per agent in the control server dashboard
• Signed binaries on macOS (Key Network Services Ltd) and Windows

---

8. Rescue Boot USB System

Emergency recovery capability:

• Bootable USB image with a minimal Linux environment
• Pre-installed ScreenControl agent configured to connect on boot
• Allows remote recovery of machines that cannot boot their primary OS
• Network configuration via DHCP with fallback static IP
• Useful for IT support, remote administration, and disaster recovery

---

🛠 Technical Architecture

macOS Agent (Objective-C)

• Native Cocoa application using AppKit
• Accessibility API integration for UI element inspection
• ScreenCaptureKit / CGWindowListCreateImage for screenshots
• CGEvent for mouse/keyboard simulation
• Runs as a menu bar application with status indicator
• Tesseract OCR framework embedded for text detection
• Supports both MCP stdio mode and WebSocket remote mode

Linux/Windows Service (C++)

• Cross-platform C++ with CMake build system
• Headless operation -- no GUI required
• X11/XCB on Linux for screen capture and input simulation
• Windows API (SendInput, BitBlt) on Windows
• WebSocket client for remote control server connection
• Tesseract OCR linked statically

Windows Tray Application (C#/.NET 8)

• .NET 8 Windows Forms tray application
• Communicates with the C++ service via localhost TCP (port 3460)
• Provides system tray icon, settings UI, and licence management
• Bridges between the GUI layer and the headless service

Control Server (Next.js 16)

• Framework: Next.js 16 with React 19 and TypeScript
• Database: PostgreSQL via Prisma ORM
• Authentication: User accounts with session management
• Billing: Stripe integration for subscriptions and usage metering
• Agent Management: Dashboard showing connected agents, versions, last seen
• MCP Endpoint: SSE streaming endpoint for cloud LLM connections
• WebSocket Hub: Routes tool calls from MCP clients to connected agents
• API: RESTful endpoints for agent registration, licence validation, status

Browser Extensions

• Firefox: WebExtensions API (Manifest V2)
• Chrome: Manifest V3
• Safari: Converted from Firefox extension via Xcode
• Communicates with the local agent via WebSocket (port 3457)
• Injects content scripts for DOM interaction

---

🔧 Connection Modes

Local stdio (Claude Code)

Claude Code --> spawns --> ScreenControl.app --mcp-stdio

• Direct process communication, no network
• Lowest latency, simplest setup
• Configured in .mcp.json

Remote SSE/WebSocket

Cloud LLM --> SSE --> Control Server --> WebSocket --> Agent

• Agent connects to wss://screencontrol.knws.co.uk/ws
• LLM connects to https://screencontrol.knws.co.uk/mcp/<connection-id>
• Supports multiple agents on different machines
• Works across networks and firewalls

---

📈 Outcome & Impact

ScreenControl demonstrates:

• Multi-platform native development -- Objective-C, C++, C#, and Swift across macOS, Linux, and Windows
• Protocol design -- implementation of the Model Context Protocol with 90+ tools
• Full-stack SaaS engineering -- Next.js control server with Stripe billing, Prisma ORM, and real-time WebSocket routing
• Browser extension development -- cross-browser extensions for Firefox, Chrome, and Safari
• Systems programming -- low-level screen capture, input simulation, and OCR integration
• Commercial product thinking -- licensing, auto-updates, rescue boot, and production deployment
• AI tooling -- enabling LLMs to interact with the physical computing environment

The project represents one of the most comprehensive MCP tool implementations available, bridging the gap between AI language models and real-world desktop computing.

---