Windows — Desktop Agent

The Windows agent is a system-tray background service that collects desktop context and performs browser automation tasks. It bridges the gap between Made Open and the web of sites that have no API.

What It Is

A Tauri application — a small Rust binary managing a WebView2 tray UI, with a Node.js Playwright sidecar for browser automation. It syncs sensor data to the hub and executes browser tasks on demand or on schedule.

apps/windows/
├── src-tauri/
│   ├── src/
│   │   ├── main.rs              # Entry point
│   │   ├── lib.rs               # Library root — app setup, Tauri command wiring
│   │   ├── config.rs            # Config loading
│   │   ├── sensor.rs            # Active app, clipboard, WiFi SSID, window titles
│   │   ├── sync.rs              # HTTP client → hub /device-events
│   │   ├── automation.rs        # Playwright sidecar process management
│   │   ├── scheduler.rs         # Task schedule management
│   │   ├── tasks.rs             # Task definitions / execution glue
│   │   ├── commands.rs          # Tauri IPC commands exposed to UI
│   │   ├── permissions.rs       # Permission gating
│   │   ├── tray.rs              # System tray UI
│   │   ├── audio/               # Audio capture + streaming
│   │   ├── capabilities/        # OS-level command handler modules
│   │   ├── setup/               # First-run setup flow
│   │   ├── transport/           # Hub transport layer
│   │   └── wallet/              # Credential wallet integration
│   ├── sidecar/                 # Bundled Playwright sidecar binary resources
│   ├── icons/
│   ├── Cargo.toml
│   └── tauri.conf.json
├── src/                         # TypeScript/React tray UI (WebView2)
│   ├── App.tsx
│   ├── main.tsx
│   ├── components/              # StatusPanel, SensorPanel, TaskForm,
│   │                            # TaskScheduler, CommandsPanel, AudioPanel,
│   │                            # PermissionsPanel, SettingsPanel
│   ├── lib/
│   └── types.ts
└── playwright/                  # Node.js sidecar (separate process)
    └── src/
        ├── runner.ts            # Task runner: receives task definitions, executes
        ├── automation/
        └── tasks/
            └── example.ts       # Example automation task

Tech Stack

As a Data Sensor

The Windows agent collects desktop context and streams it to the hub:

Device Events Posted to Hub

Privacy: Window titles and clipboard content are configurable — users control what gets sent. Clipboard is hashed by default (presence detection only, not content).

Location Inference from WiFi

Without GPS, the Windows agent infers location from WiFi SSID:

• SSID "HomeNetwork" → user is at Home
• SSID "CorporateWifi" → user is at Work
• User defines the SSID → location mapping in Settings

This feeds the Rules Service the same location context as the Android GPS sensor, enabling location-aware rules to work from the desktop.

As a Browser Automation Agent

The browser automation subsystem uses Playwright (running as a Node.js sidecar) to interact with websites that have no API.

Task Definition

Tasks are defined in the Playwright sidecar as TypeScript functions. The Tauri backend triggers them:

// playwright/tasks/bank-statement.ts
export async function downloadBankStatement(config: TaskConfig, browser: Browser) {
  const page = await browser.newPage();

  await page.goto(config.loginUrl);
  await page.fill('#username', config.credentials.username);
  await page.fill('#password', config.credentials.password);
  await page.click('[type="submit"]');

  await page.waitForURL('**/dashboard');
  await page.click('a[href*="statements"]');

  // Download the most recent statement
  const [download] = await Promise.all([
    page.waitForEvent('download'),
    page.click('.download-latest-statement'),
  ]);

  const path = await download.path();
  // POST file to hub storage, which triggers data.EntityCreated
  await hubClient.uploadFile(path, 'application/pdf', 'statements/');
}

Task Triggers

Credential Security

Browser automation credentials (usernames/passwords for sites) are stored in Windows Credential Manager (DPAPI-encrypted), not in plaintext config files. The Playwright sidecar retrieves them from Credential Manager at task execution time.

Tray UI

The system tray UI is minimal:

• Status indicator: connected / disconnected / error
• Last sync timestamp
• Task list with Run/Schedule controls
• "Open settings" → larger window for sensor configuration and task management

First Testable Workflow

1. Build and install the agent (or run in dev mode)

2. System tray icon appears
3. Agent begins collecting:
   - Active application (every 10s)
   - WiFi SSID (every 60s)

4. Open tray UI → verify sensor status shows live data
5. Check hub: GET /api/devices → device appears
6. Check Supabase: location_history or device events table

7. Configure a browser task:
   - Add a task that navigates to example.com and captures a screenshot
   - Click "Run Now"
   - Verify: screenshot file appears in Supabase Storage

8. Configure WiFi → location mapping:
   - Add: SSID "HomeNetwork" = "Home"
   - Verify: hub receives location.Updated with location="Home"
   - Verify: location-based rule evaluates based on Windows agent location

Running in Dev Mode

cd apps/windows
cargo tauri dev

# Playwright sidecar runs as separate process:
cd apps/windows/playwright
pnpm install
pnpm start

Requires: Rust toolchain, Node.js 20+, pnpm, WebView2 runtime (pre-installed on Windows 11).

Capability Handler System

The Windows agent uses a capability handler architecture at src-tauri/src/capabilities/ for OS-level device commands, alongside top-level modules in src-tauri/src/ for sensing, sync, automation, and UI.

Top-level modules in src-tauri/src/:

OS-level capability modules in src-tauri/src/capabilities/ (invoked via the device.CommandRequested flow):

browser.rs, clipboard.rs, display.rs, filesystem.rs, input.rs, network.rs, notifications.rs, power.rs, process.rs, registry.rs, shell.rs.

Capability Notes

Initial scope: Active app tracking, WiFi SSID collection, sync to hub, basic tray UI.

Browser automation: Full browser automation task system, credential management via Windows Credential Manager, task scheduling.

Extended capabilities: Audio capture support, device command handling (filesystem, process, clipboard, display, power, network, registry, notifications, input, shell operations).

The agent collects context from day one, so when browser automation is enabled, it has a rich history of user activity to work with.