# OpenEye > Open-source "Ollama for vision": a CLI-first perception engine that turns raw video into structured world models for robots, VLA models, and autonomous agents. Install with `pipx install openeye-sh`. OpenEye runs vision models (YOLO, RF-DETR, Grounding DINO, Depth Anything V2, SAM 2, SmolVLA) locally on CPU, NVIDIA CUDA, or Apple Silicon. `openeye pull ` downloads weights, `openeye run ` prints JSON, and `openeye serve ` exposes REST and WebSocket APIs. Source at https://github.com/aw3-technology/openeye.sh. Full docs as one file: https://openeye.sh/llms-full.txt - Hosted API: `https://openeye.sh/v1` with `Authorization: Bearer oe_...` (keys at https://openeye.sh/dashboard/settings/api-keys). `POST /v1/detect` (YOLOv8, 1 credit), `GET /v1/models`, `GET /v1/usage`; `/v1/describe` returns 503 until enabled; depth and streaming are self-hosted only. CLI: `openeye api detect ` with `OPENEYE_API_KEY`. - MCP server: `openeye-mcp` (install `openeye-sh[mcp]`) exposes local tools (openeye_list, openeye_pull, openeye_run, openeye_serve, openeye_health, openeye_bench, openeye_remove, openeye_nebius_stats) and hosted tools (openeye_api_detect, openeye_api_describe, openeye_api_models, openeye_api_usage; need `OPENEYE_API_KEY`). - Dashboard: https://openeye.sh/dashboard for API keys, credits, inference history, fleet management and observation memory; connect it to a local `openeye serve` at http://localhost:8000 for live inference. ## Getting Started - [Installation](https://openeye.sh/docs/installation.md): There are two ways to use OpenEye: - [Quickstart](https://openeye.sh/docs/quickstart.md): Pick a path. Path A runs models on your own machine with the CLI (free). Path B calls the hosted API at https://openeye.sh/v1 with an API key (paid in… ## CLI Reference - [openeye pull](https://openeye.sh/docs/cli-pull.md): Download model weights from Hugging Face Hub to ~/.openeye/models//. - [openeye run](https://openeye.sh/docs/cli-run.md): Run inference on an image, a directory of images, or stdin, and print JSON results. The model must be pulled first. - [openeye serve](https://openeye.sh/docs/cli-serve.md): Start a FastAPI inference server with REST API, WebSocket streaming, and a built-in browser page. The model must be pulled first. - [openeye watch](https://openeye.sh/docs/cli-watch.md): Live camera feed with real-time detections displayed in the terminal using Rich. - [openeye bench](https://openeye.sh/docs/cli-bench.md): Benchmark a model's inference speed. Reports mean, median, and P95 latency with FPS. - [openeye list](https://openeye.sh/docs/cli-list.md): List every model in the registry and whether it's downloaded. Takes no options. - [Model Management](https://openeye.sh/docs/cli-models.md): Commands for removing weights and extending the model registry. See openeye pull and openeye list for downloading and browsing. - [openeye config](https://openeye.sh/docs/cli-config.md): Store and read key/value settings in ~/.openeye/config.yaml. - [openeye agent](https://openeye.sh/docs/cli-agent.md): Agentic perception loop — continuously perceive, reason, and act. The agent maintains observation memory and can use VLM reasoning to pursue goals. - [openeye api](https://openeye.sh/docs/cli-api.md): Client commands for the hosted inference API at https://openeye.sh/v1. They need an API key from the dashboard (Settings → API Keys) in the… - [openeye fleet](https://openeye.sh/docs/cli-fleet.md): Client for a self-hosted fleet control plane (see Fleet Authentication for setting one up), plus the device agent that runs on each edge device. - [Server Utilities, Demos & Docs](https://openeye.sh/docs/cli-utilities.md): These call a running openeye serve at OPENEYE_SERVER_URL (default http://localhost:8000); each takes --server, -s . ## Architecture - [Overview](https://openeye.sh/docs/architecture-overview.md): OpenEye has four parts. You can use any of them on its own. - [Perception Pipeline](https://openeye.sh/docs/perception-pipeline.md): The perception subsystem provides: - [Plugin System](https://openeye.sh/docs/plugin-system.md): All components are loaded dynamically via plugin loaders: ## VLM via OpenRouter - [VLM via OpenRouter](https://openeye.sh/docs/vlm-openrouter.md): Send images to free Vision Language Models through OpenRouter for scene understanding, hazard detection, and visual reasoning. ## Configuration - [Config Format](https://openeye.sh/docs/config-format.md): The backend uses JSON5 configuration files with schema validation. Configuration files live in backend/config/ and support comments and trailing commas. - [Single-Mode Config](https://openeye.sh/docs/single-mode.md): A single-mode config defines one behavior set for the agent: - [Multi-Mode Config](https://openeye.sh/docs/multi-mode.md): Multi-mode configs define multiple behavioral modes with transition rules: - [Environment Variables](https://openeye.sh/docs/environment-variables.md): Read by the openeye command. URLs have no trailing /v1; the CLI appends paths itself. ## API Reference - [REST Endpoints](https://openeye.sh/docs/rest-endpoints.md): The self-hosted server (openeye serve ) runs on port 8000 by default and exposes the following REST endpoints. The Agent, Governance, and MLOps… - [WebSocket](https://openeye.sh/docs/websocket.md): The self-hosted server exposes four WebSocket endpoints for real-time inference at different levels of the perception stack. - [Schema Reference](https://openeye.sh/docs/schema-reference.md): OpenEye uses Pydantic models for structured output. All models are defined in cli/openeye_ai/schema.py. - [Model Adapters](https://openeye.sh/docs/model-adapters.md): Adapters are the bridge between OpenEye and ML model backends. Each adapter implements loading and inference for a specific model type. ## Agent API - [Agent API](https://openeye.sh/docs/agent-api.md): The agent API provides REST endpoints for controlling the agentic perception loop and querying observation memory. All endpoints are prefixed with /agent. ## Governance API - [Governance API](https://openeye.sh/docs/governance-api.md): The governance API provides endpoints for managing safety policies, presets, and audit logs. All endpoints are prefixed with /governance. The governance… ## MLOps API - [MLOps API](https://openeye.sh/docs/mlops-api.md): The MLOps API provides endpoints for model lifecycle management, A/B testing, retraining pipelines, batch inference, and more. All endpoints are prefixed… ## Hosted API - [Authentication](https://openeye.sh/docs/hosted-authentication.md): The hosted API lives at https://openeye.sh/v1. Every request needs an API key, including GET /v1/models and GET /v1/usage. - [POST /v1/detect](https://openeye.sh/docs/hosted-detect.md): Object detection with YOLOv8 Nano (80 COCO classes), run on CPU with ONNX Runtime. Costs 1 credit per call. - [POST /v1/depth](https://openeye.sh/docs/hosted-depth.md): When depth comes to the hosted API it is priced at 2 credits per call. - [POST /v1/describe](https://openeye.sh/docs/hosted-describe.md): Describe an image, or ask a question about it, with the Qwen2.5-VL vision-language model (qwen/qwen2.5-vl-72b-instruct, via OpenRouter). Costs 3 credits… - [GET /v1/models](https://openeye.sh/docs/hosted-models.md): Lists the hosted models, what each costs, and whether it's available right now. Free; needs an API key. - [GET /v1/usage](https://openeye.sh/docs/hosted-usage.md): Your credit balance and API usage over the last days days (default 30, clamped to 1–365). Free; needs an API key, and isn't rate limited. ## WebSocket Streaming - [Streaming Protocol](https://openeye.sh/docs/ws-protocol.md): Basic real-time inference endpoint. - [WebSocket Examples](https://openeye.sh/docs/ws-examples.md): WebSocket Examples — OpenEye documentation. ## Error Handling - [Error Codes](https://openeye.sh/docs/error-codes.md): The hosted API at https://openeye.sh/v1 returns errors as JSON with a single error message: - [Rate Limits](https://openeye.sh/docs/rate-limits.md): Each API key allows 60 requests per minute by default, counted over a sliding 60-second window. The limit applies to the billed endpoints (POST /v1/detect… ## SDK Examples - [Python](https://openeye.sh/docs/sdk-python.md): Usage and balance: - [JavaScript / TypeScript](https://openeye.sh/docs/sdk-javascript.md): Keep the API key on a server; don't ship it to browsers. - [cURL](https://openeye.sh/docs/sdk-curl.md): Add -i to see the X-RateLimit-Limit and X-RateLimit-Remaining headers. ## Claude Skill - [Try It Out — OpenEye Claude Skill](https://openeye.sh/docs/claude-skill.md): The OpenEye Claude Skill lets you use Claude (via Claude Code or the Claude Desktop app with MCP) as a copilot for your vision AI workflows. Ask Claude to… ## Credits & Billing - [Credits System](https://openeye.sh/docs/credits-system.md): Calls to the hosted API at https://openeye.sh/v1 are paid for with credits, handled by Cred (cred.diy). Buy them in the dashboard under Settings →… - [Usage Tracking](https://openeye.sh/docs/usage-tracking.md): Every successful billed call to https://openeye.sh/v1 is logged with its endpoint, model, credits, and inference time. See it three ways: ## Fleet Management API - [Fleet Authentication](https://openeye.sh/docs/fleet-auth.md): Fleet management works two ways: - [Device Management](https://openeye.sh/docs/fleet-devices.md): These commands and REST routes are for a self-hosted fleet control plane (see Fleet Authentication). On openeye.sh, use the dashboard's Fleet → Devices… - [Deployments](https://openeye.sh/docs/fleet-deployments.md): Deployments push model updates to groups of edge devices using staged rollout strategies. - [Commands & Heartbeats](https://openeye.sh/docs/fleet-commands.md): Edge devices running openeye fleet agent send heartbeats every 15 seconds (--interval) via POST /heartbeats, authenticated with their oek_ device key. The… ## Models - [YOLOv8](https://openeye.sh/models/yolov8): Real-time object detection with 80 COCO classes. `openeye pull yolov8` - [YOLO26](https://openeye.sh/models/yolo26): Latest YOLO generation with 80 COCO classes, optimized for speed. `openeye pull yolo26` - [RF-DETR](https://openeye.sh/models/rfdetr): Real-time detection transformer without NMS post-processing. `openeye pull rfdetr` - [Grounding DINO](https://openeye.sh/models/grounding-dino): Open-vocabulary object detection from text prompts. `openeye pull grounding-dino` - [Depth Anything V2](https://openeye.sh/models/depth-anything): Monocular depth estimation from a single camera. `openeye pull depth-anything` - [SAM 2](https://openeye.sh/models/sam2): Zero-shot segmentation via Segment Anything 2. `openeye pull sam2` - [SmolVLA](https://openeye.sh/models/smolvla): Vision-language-action model for robotic control. `openeye pull smolvla` ## Blog - [Why Robot Perception Needs to Be Open Source](https://openeye.sh/blog/why-robot-perception-needs-to-be-open-source): The robotics industry is building on a foundation of proprietary black boxes. That has to change. Here's why open-source perception isn't just idealism — it's the only viable path to safe, scalable autonomy. - [The Safety Crisis No One in Robotics Is Talking About](https://openeye.sh/blog/the-safety-crisis-no-one-in-robotics-is-talking-about): Robots are shipping faster than safety systems can keep up. Most deployed robots today have no real-time hazard detection. Here's why that's about to become a serious problem — and what we're doing about it. - [From Camera to Action in Six Layers](https://openeye.sh/blog/from-camera-to-action-in-six-layers): Most robot perception pipelines are tangled messes of ad-hoc integrations. OpenEye's six-layer architecture turns raw pixels into executable actions through a clean, modular pipeline. Here's how each layer works. - [Why We Built a CLI Instead of a Dashboard](https://openeye.sh/blog/why-we-built-a-cli-instead-of-a-dashboard): Everyone expects a perception platform to ship with a slick web dashboard. We shipped a terminal command instead. Here's the reasoning behind OpenEye's CLI-first design — and why it's actually better for production robotics. - [The Two-Brain Architecture: YOLO at 30fps Meets VLM Reasoning](https://openeye.sh/blog/two-brain-architecture-yolo-meets-vlm): Fast detection and deep reasoning have always been at odds. OpenEye's two-brain architecture runs both simultaneously — YOLO for reflexive speed, Vision Language Models for contextual intelligence. Here's how they work together. - [Why Robots Forget — and How Persistent Memory Changes Everything](https://openeye.sh/blog/why-robots-forget-and-how-persistent-memory-changes-everything): Most robots perceive the world frame by frame with no memory of what came before. OpenEye's persistent memory system lets robots remember environments, track changes over time, and learn from experience. - [One Pipeline, Every Robot: The Case for Hardware-Agnostic Perception](https://openeye.sh/blog/one-pipeline-every-robot): The robotics industry is fragmented across dozens of hardware platforms, each with its own perception stack. OpenEye's hardware-agnostic architecture means you build your perception once and deploy it everywhere. - [Real-Time Perception Streaming with WebSockets](https://openeye.sh/blog/real-time-perception-streaming-with-websockets): Robot perception data is only valuable if it reaches downstream systems in real-time. Here's how OpenEye uses WebSocket streaming to deliver structured perception data with sub-50ms latency to any system that needs it. - [OpenEye vs. Proprietary Vision Stacks: An Honest Comparison](https://openeye.sh/blog/openeye-vs-proprietary-vision-stacks): How does an open-source perception engine stack up against proprietary offerings? We compare OpenEye against the incumbent solutions honestly — including where we're still catching up. - [Building Safety Systems That Actually Stop the Robot](https://openeye.sh/blog/building-safety-systems-that-actually-stop-the-robot): Most robot safety systems are advisory — they detect hazards and send alerts. OpenEye's safety pipeline is interventionist. It detects hazards and stops the robot. Here's the engineering behind making that reliable. - [Perception as Infrastructure: The Next Layer of the AI Stack](https://openeye.sh/blog/perception-as-infrastructure): Databases, compute, networking — every layer of infrastructure eventually got standardized. Perception is next. Here's why robot perception needs to become a utility, not a product — and what that means for the industry. - [Model-Agnostic Perception: Future-Proofing Your Robot's Vision](https://openeye.sh/blog/model-agnostic-perception-future-proofing-your-robot): AI models improve every few months. Your robot's perception system shouldn't be locked to last year's model. Here's how OpenEye's model-agnostic design lets you upgrade your robot's intelligence without changing your code. - [From Hackathon to Open Source: The OpenEye Origin Story](https://openeye.sh/blog/from-hackathon-to-open-source-the-openeye-origin-story): OpenEye started as a weekend project at a hackathon. It became something bigger when we realized the robotics industry needed open-source perception more than it needed another proprietary platform. Here's how it happened. ## Optional - [Pricing](https://openeye.sh/pricing): Free open-source CLI plus hosted API plans. - [Architecture](https://openeye.sh/architecture): System design, inference pipeline, and fleet management. - [Use Cases](https://openeye.sh/use-cases): Robotics, industrial safety, drone navigation, and home automation. - [Changelog](https://openeye.sh/changelog): Version history and release notes. - [About](https://openeye.sh/about): Mission, team, and project history.