Node Capabilities

+15 Mana ✨

Introduction

Once a node is paired with a gateway, agents can invoke its capabilities to interact with the physical world. Each capability category covers a different aspect of device hardware, from screen capture to GPS positioning. This lesson details every capability, its parameters, and which platforms support it.

Key Concepts

  • Canvas: A capability set for taking screenshots, navigating web pages, evaluating JavaScript in a browser context, and performing A2UI (Agent-to-UI) interactions such as clicks and keystrokes.
  • Camera: Captures photos in JPG format from front or back cameras, and records video in MP4 format for up to 60 seconds.
  • Audio/Recording: Records the screen at a configurable FPS, optionally including audio capture.
  • Location: Reads GPS coordinates from the device. Disabled by default for privacy and must be explicitly enabled.
  • SMS: Sends and reads SMS messages. Available only on Android nodes.
  • System Exec: Executes shell commands on the node host machine, enabling remote administration when host=node.

Real World Context

Consider a field-service AI agent that dispatches a technician. The agent uses the Canvas capability to screenshot a dashboard showing equipment status, the Camera capability to have the technician photograph a broken part, the Location capability to confirm they are on-site, and System Exec to restart a local service. Each of these is a single invoke call to a paired node.

Deep Dive

Canvas

The Canvas capability provides four operations: screenshot, web navigation, JavaScript evaluation, and A2UI interaction.

Screenshot captures the current screen state:

json
{
  "capability": "canvas.screenshot",
  "args": { "format": "png" }
}

This invoke call captures the entire visible screen and returns it as a PNG image. The agent can then analyze the image using a vision model.

Web navigation opens a URL in the node's browser:

json
{
  "capability": "canvas.navigate",
  "args": { "url": "https://dashboard.example.com" }
}

This opens the specified URL, allowing subsequent screenshots or JavaScript evaluation against the loaded page.

JavaScript evaluation runs code in the browser context:

json
{
  "capability": "canvas.eval",
  "args": { "script": "document.title" }
}

The script executes in the page context and returns the result. This is useful for extracting data from web applications without parsing screenshots.

A2UI (Agent-to-UI) lets agents interact with GUI elements by simulating clicks, typing, and scrolling, bridging the gap between agent intent and graphical interfaces.

Camera

The Camera capability captures photos and video from physical cameras:

json
{
  "capability": "camera.photo",
  "args": { "format": "jpg", "facing": "back" }
}

The facing parameter accepts front or back. Photos are returned as base64-encoded JPG data. For video:

json
{
  "capability": "camera.video",
  "args": { "format": "mp4", "duration": 30, "facing": "back" }
}

Video recording supports a maximum duration of 60 seconds. The result is an MP4 file streamed back to the gateway.

Audio/Recording

Screen recording captures the display at a configurable frame rate:

json
{
  "capability": "recording.screen",
  "args": { "fps": 15, "audio": true }
}

The fps parameter controls frame rate, and audio toggles whether system audio is included. This is useful for recording demonstrations or capturing UI bugs in action.

Location

GPS location reading is disabled by default for privacy:

json
{
  "capability": "location.gps",
  "args": {}
}

Before this will work, you must explicitly enable location in the node configuration. The result includes latitude, longitude, and accuracy in meters. The default-disabled design ensures location data is never shared accidentally.

SMS (Android Only)

SMS capability is exclusive to Android nodes:

json
{
  "capability": "sms.send",
  "args": { "to": "+1234567890", "body": "Your order has shipped." }
}

This sends a text message from the Android device. It is unavailable on all other platforms.

System Exec

System exec runs shell commands on the node host:

json
{
  "capability": "system.exec",
  "args": { "command": "uptime", "host": "node" }
}

When host is set to node, the gateway forwards the exec request to the node rather than running it locally. This enables remote administration of the machine where the node is running.

Common Pitfalls

  • Assuming Location is enabled by default: GPS is disabled out of the box. Agents will receive an error if they invoke location.gps without enabling it in the node configuration first.
  • Requesting SMS on non-Android nodes: SMS is an Android-only capability. Invoking it on iOS, macOS, or headless nodes will fail with an unsupported capability error.
  • Setting video duration above 60 seconds: Camera video recording has a hard cap of 60 seconds. Requests exceeding this limit are rejected.

Best Practices

  • Check platform capabilities before invoking: Query the node's registered capabilities to avoid runtime errors on unsupported platforms.
  • Use Canvas eval over screenshots when possible: Extracting data via JavaScript is faster and more reliable than parsing screenshots with vision models.
  • Keep screen recording FPS low for bandwidth: High FPS recordings consume significant bandwidth over WebSocket. Use 10-15 FPS unless higher fidelity is required.

Summary

  • Canvas provides screenshot, web navigation, JavaScript evaluation, and A2UI interaction for GUI automation.
  • Camera captures JPG photos (front/back) and MP4 video up to 60 seconds.
  • Audio/Recording captures screen recordings at configurable FPS with optional audio.
  • Location reads GPS coordinates but is disabled by default for privacy.
  • SMS is an Android-only capability for sending text messages.
  • System exec runs shell commands on the node host when host=node, enabling remote administration.
✓ Completed