Vision Sandbox 🔭

Leverage Gemini's native code execution to analyze images with high precision. The model writes and runs Python code in a Google-hosted sandbox to verify visual data, perfect for UI auditing, spatial grounding, and visual reasoning.

Installation

clawhub install vision-sandbox

Usage

uv run vision-sandbox --image "path/to/image.png" --prompt "Identify all buttons and provide [x, y] coordinates."

Pattern Library

📍 Spatial Grounding

Ask the model to find specific items and return coordinates.

• Prompt: "Locate the 'Submit' button in this screenshot. Use code execution to verify its center point and return the [x, y] coordinates in a [0, 1000] scale."

🧮 Visual Math

Ask the model to count or calculate based on the image.

• Prompt: "Count the number of items in the list. Use Python to sum their values if prices are visible."

🖥️ UI Audit

Check layout and readability.

• Prompt: "Check if the header text overlaps with any icons. Use the sandbox to calculate the bounding box intersections."

🖐️ Counting & Logic

Solve visual counting tasks with code verification.

• Prompt: "Count the number of fingers on this hand. Use code execution to identify the bounding box for each finger and return the total count."

Integration with OpenCode

This skill is designed to provide Visual Grounding for automated coding agents like OpenCode.

• Step 1: Use vision-sandbox to extract UI metadata (coordinates, sizes, colors).
• Step 2: Pass the JSON output to OpenCode to generate or fix CSS/HTML.

Configuration

• GEMINI_API_KEY: Required environment variable.
• Model: Defaults to gemini-3-flash-preview.