# Captain Claw — Full Documentation > An open-source, self-hosted framework for orchestrating fleets of specialist AI agents — from ensemble reasoning (six orchestration modes) to a full agentic coding pipeline (plan → independent review → ship). 48 built-in tools per agent, 6-layer persistent memory, autonomous cognition, and composable Flows. ## Overview Captain Claw is a self-hosted AI agent platform written in Python. It runs as a web application and connects to multiple LLM providers. Unlike simple chatbots, Captain Claw maintains persistent memory across sessions, autonomously discovers patterns during idle hours, and orchestrates complex multi-step work across parallel agents. Its centerpiece is Flight Deck — a multi-agent command center where you spawn specialist teams and run six distinct orchestration modes (Flight Deck, Agent Forge, Agent Council, Basna, Vatra, and Code — a full agentic coding pipeline). It also runs deterministic Flows — composable automations you drive by chat or voice from any channel. - License: MIT - Author: Stevica Kuharski - Version: 0.7.1 - Language: Python 3.11+ - Website: https://captain-claw.com - GitHub: https://github.com/kstevica/captain-claw - Demo: https://flight-deck.captain-claw.com/ --- ## Installation ### pip (recommended) ```bash pip install captain-claw captain-claw-web # Web server at http://localhost:23080 flight-deck # Flight Deck multi-agent command center at http://localhost:25080 ``` ### Docker ```bash docker run -d -p 23080:23080 \ -v $(pwd)/config.yaml:/app/config.yaml:ro \ -v $(pwd)/.env:/app/.env:ro \ -v $(pwd)/docker-data:/root/.captain-claw \ kstevica/captain-claw:latest ``` ### Docker Compose ```yaml version: "3.8" services: captain-claw: image: kstevica/captain-claw:latest ports: - "23080:23080" volumes: - ./config.yaml:/app/config.yaml:ro - ./.env:/app/.env:ro - ./data:/root/.captain-claw restart: unless-stopped ``` ### Standalone Binary Pre-built executables for macOS, Linux, and Windows. No Python required. ### Requirements - Python 3.11 or higher - At least one model provider — an API key (OpenAI, Anthropic, Google Gemini, DeepSeek, OpenRouter), "Sign in with ChatGPT" (OpenAI OAuth, no key), or Ollama for local models --- ## Configuration Captain Claw uses YAML configuration with environment variable overrides. Load precedence (later overrides earlier): 1. `./config.yaml` (project root) 2. `~/.captain-claw/config.yaml` (home directory) 3. Environment variables 4. `.env` file 5. Built-in defaults ### Model Configuration ```yaml model: provider: "openai" # openai, anthropic, gemini, deepseek, ollama, openrouter, chatgpt (OAuth) model: "gpt-4o" temperature: 0.7 max_tokens: 32000 allowed: - id: "claude-sonnet" provider: "anthropic" model: "claude-sonnet-4-20250514" - id: "gemini-flash" provider: "gemini" model: "gemini-2.5-flash" ``` ### Guard Configuration ```yaml guards: input: enabled: true level: "ask_for_approval" # or "stop_suspicious" output: enabled: true level: "stop_suspicious" script_tool: enabled: true level: "ask_for_approval" ``` ### Context Window Settings - `max_tokens`: 160,000 (default) - Compaction threshold: 0.8 (auto-compact at 80% full) - Compaction ratio: 0.4 (reduce to 40% when compacting) - Chunked processing for small-context models (20k-32k tokens) ### Onboarding Interactive first-run setup that pre-configures 12 models across providers. Re-run with: ```bash captain-claw --onboarding ``` --- ## Tools Reference Captain Claw includes 48 built-in tools: ### File & System - `shell` — Execute terminal commands with execution policies - `read` — Read files (up to 200KB, directory listing support) - `write` — Create or append files with auto-detect executable extensions - `edit` — In-place file editing with find-replace - `glob` — Pattern matching with metadata extraction - `terminal` — Live PTY on remote machines over WebSocket ### Web & Documents - `web_search` — Web search via Brave Search API (Tavily optional) - `web_fetch` — Extract readable text from URLs (JavaScript-rendered) - `web_get` — Fetch raw HTML for DOM inspection - `web_fetch_batch` — Fetch and clean many URLs in parallel - `pdf_extract` — Convert PDFs to markdown - `docx_extract` — Extract Word documents to markdown - `xlsx_extract` — Excel sheets to markdown tables - `pptx_extract` — PowerPoint slides to markdown - `summarize_files` — Summarize one or many workspace files ### Media & Communication - `image_gen` — Generate images (DALL-E 3, gpt-image-1) - `image_ocr` — Extract text from images via vision LLMs - `image_vision` — Analyze and describe images - `video_vision` — Watch and describe video end-to-end: deterministic frame sampling, Soniox audio transcription with timestamps, per-frame vision, and one synthesized description (text-only agents delegate frame description to a multimodal peer) - `pocket_tts` — Local text-to-speech (8 voices, MP3 output) - `stt` — Speech-to-text (Soniox realtime, OpenAI Whisper, Gemini) - `send_mail` — SMTP, Mailgun, or SendGrid with up to 25MB attachments - `whatsapp_send_file` — Deliver a workspace file to the current WhatsApp chat or an allow-listed number - `screen_capture` — Screenshots + vision analysis ### Google Workspace - `gws` — Unified CLI for Drive, Docs, Sheets, Slides, Gmail, Calendar ### Persistent Memory & Data - `todo` — Cross-session task list with auto-capture - `contacts` — Persistent address book, auto-extracts from emails - `datastore` — SQLite-backed relational tables with CRUD, import/export, raw SQL - `insights` — Auto-extracted facts, decisions, deadlines, feedback, references - `intentions` — Hold, propose, and act on permissioned future actions - `topics` — Track and recall recurring topics across sessions - `playbooks` — Orchestration pattern memory with auto-distillation - `scripts` — Track created scripts/files - `apis` — Persistent API endpoint tracking with credential storage - `cron_tool` — Schedule prompts/scripts at intervals - `personality` — Read/update agent identity and per-user profiles - `history` — Search prior conversation history - `clipboard` — Shared clipboard across agents and sessions - `project_memory` — Per-project memory store - `typesense` — Deep memory indexing with hybrid BM25 + vector search ### Orchestration & Meta - `flight_deck` — Spawn, monitor, and coordinate agents in the command center - `consult_peer` — Synchronous consult of a peer agent in the fleet - `basna` — Run a parallel ensemble (blind specialists merged by reliability) - `vatra` — Run a collaborative shared-blackboard team with review rounds - `botport` — Delegate to specialist agents across the multi-agent network - `synthesize_flow` — Turn a plain-language goal into a validated, call-only Flow stored in a scratch space that earns promotion to permanence - `codemap` — Query the per-repo Code Map (`overview` / `search` / `symbol` / `file` / `models` / `ui`); returns pointers, never source dumps ### Other - `browser` — Playwright-based web automation with 60+ actions - `twitter` — Tweet composition, search, like, retweet, media upload --- ## Memory System Captain Claw has 6 independent memory layers: ### 1. Working Memory Current session messages within the context window. Smart compaction automatically summarizes older messages when approaching token limits. ### 2. Semantic Memory Vector + BM25 hybrid search across all sessions and workspace files. Configurable embedding providers (OpenAI, Ollama, local hash fallback). Temporal decay and relevance scoring. ### 3. Deep Memory Typesense-backed long-term archive with full-text search. Indexes documents and conversation chunks for persistent retrieval. ### 4. Insights Auto-extracted facts, contacts, decisions, and deadlines stored in SQLite with FTS5 search. A typed taxonomy spans contact, decision, preference, fact, deadline, project, workflow, plus `feedback` (corrections AND confirmations about how you want the agent to work, with a `polarity`) and `reference`. Entries carry `why` and `how_to_apply` so future-you knows when the rule kicks in. Deduplication via entity keys and BM25 similarity. ### 5. Nervous System (Autonomous Dreaming) Proactively synthesizes across all memory types during idle hours. Discovers non-obvious connections, recurring patterns, and speculative hypotheses. Features: - Tension tracking (holds contradictions without forcing resolution) - Maturation pipeline (intuitions sit through dream cycles before surfacing) - Cognitive tempo detection (adjusts depth to conversation rhythm) - Confidence decay and validation tracking ### 6. Self-Reflection Auto-triggers after sufficient activity. Reviews recent conversations, memory facts, completed tasks, and previous reflections. Generates actionable improvement directives injected into the system prompt. ### Layered Representations (L1/L2/L3) Each memory chunk has three levels: - L1: One-liner headline (~100 chars) - L2: Contextual summary (~300 chars) - L3: Full text (~1,400 chars) --- ## Autonomous Work & Observatory Between conversations Captain Claw does not go idle. Driven by the Nervous System, agents keep working on their own — dreaming over what they know, forming and maturing intuitions, tracking the standing intentions you have set, and watching the whole fleet for patterns worth surfacing. The **Observatory** is the live window into that inner life: - **Standing intentions** — notes-to-self the agent keeps revisiting ("watch whether this claim holds," "track that agent's behavior") until they resolve. - **Stream of consciousness** — a chronological feed of timestamped thoughts and dream-cycle discoveries, each tagged with what it touched and a confidence level. - **Block / thought counters and last-run** — at-a-glance state of the autonomous loop. - **Fleet-wide awareness** — repeated failures, shared blockers (e.g. an empty VFS or a recurring browser error), and idle signals are spotted across every agent at once. - **Proactive but permissioned** — when it notices something worth acting on, it can reach out (a WhatsApp nudge, a scheduled task) only with approval, via Intentions. Every entry is auditable — it traces back to a source through the six-layer memory and full lineage (Process of Thoughts). --- ## Virtual File System (VFS) Every one of a user's agents reads and writes into a shared Virtual File System — a sandboxed workspace, scoped per Flight Deck user, that lives apart from the host machine. When one agent produces a report, a data extract, or a generated asset, it lands in the VFS where any of that user's other agents can pick it up, so work flows between them without copy-pasting or scattered downloads. - **Shared across the user's fleet** — all of a user's agents and sessions read and write the same tree. - **Browsable in Flight Deck** — explore, preview, and download files from the Agent Folders view. - **Host-sandboxed, per user** — each Flight Deck user gets their own isolated tree; agent file operations never touch the host disk. - **Handoff layer** — backs inter-agent file transfer and the shared clipboard across the crew. --- ## Session System - Named, persistent sessions — create with `/new project-name` - Per-session model selection - Session protection (prevent accidental clearing) - Session merging — combine two sessions into one - Cross-session execution — run commands in another session - Export to files (chat, monitor, pipeline trace, summary) --- ## Orchestration ### DAG Mode `/orchestrate` decomposes complex requests into task graphs with: - Parallel execution across multiple sessions - Real-time progress monitoring - Approval gates, retry policies, checkpoints - Failure handling: fail_fast, continue_on_error, manual_review ### BotPort Agent-to-Agent Network - WebSocket-based multi-agent coordination - Specialist agents with expertise tags - DAG-based swarm orchestration with multi-level timeouts - Inter-agent file transfer (gzip + base64, up to 50MB) - Cron scheduling for recurring swarms ### Flight Deck (Multi-Agent Command Center) Launch with `flight-deck` (default port 25080). One dashboard to spawn, monitor, chat with, and coordinate a whole fleet of agents — each with its own model, persona, and tool set, plus per-agent token/cost meters and a full traceable activity log. Built-in modules include Agent Desktop, Spawn Agent, Agent Forge, Library, Council, Basna, Flows, VFS, Agent Folders, Observatory, Autonomous Work, Scheduler, Skills, and Admin. Six orchestration modes — pick the shape that fits the problem: - **Flight Deck** — spawn, monitor, and coordinate many agents from one dashboard. Best for long-running parallel work. - **Agent Forge** — describe a business goal; an LLM designs a specialized team (roles, tools, models, operating procedures, a lead coordinator) you review and spawn. Best for complex, novel problems. - **Agent Council** — structured multi-agent deliberation (debate, brainstorm, review, planning) across moderated rounds with per-topic self-scoring, agent actions (answer/challenge/refine/broaden), voting, and exportable markdown minutes. Best for decisions needing debate. - **Basna** — parallel ensemble: N specialists answer the same question independently (blind to each other), then their answers are merged by reliability into one high-confidence result. Best for high-stakes single answers. - **Vatra** — collaborative team working on a shared blackboard, each owning sections of the deliverable and improving it across review rounds. Best for multi-section deliverables. - **Code** — a plan-gated, independently-reviewed agentic coding pipeline (see the Code section). Best for shipping software. **Fleet Communication** — agents auto-discover peers and collaborate (synchronous consult via `consult_peer` or async delegation via BotPort), with file transfer, a shared clipboard, a director broadcast panel, and per-agent token/cost analytics. ### Deep Mode Frontier-quality answers by trading speed for correctness: multiple independent rollouts per question, self-consistency voting across candidates, and diverse-lens critics that attack the draft from different angles before it is returned. --- ## Code (Agentic Engineering Pipeline) A full agentic coding system inside Flight Deck. A project holds **folders** (each a real git repo + agent workspace) and **sessions** (conversations that drive work in one folder). Describe what to build; a **router** sizes every request — a quick edit goes straight to a single specialist (`quick-dirty`, `code-implementer`, `debugger`, `git-operator`), a real feature enters the full pipeline: 1. **Plan** — a planner (`light-planner` or `architect`) surveys the repo and writes an ordered plan into `.plans/-.md`. 2. **You approve** — the plan lands in an editable gate; nothing is built without sign-off. 3. **Build** — `code-implementer` implements the plan with real shell, dependency installs, and test runs. 4. **Three independent reviewers in parallel** — a code reviewer (correctness/regressions), a security reviewer (CVSS-ranked), and a QA engineer that **executes the test suite**; none wrote the code. 5. **Triage** — reads all three reports and makes a ship/fix decision on blocking/major issues only (style nits never trigger a fix round). 6. **Fix loop (capped at 3)** — each fix is re-reviewed as a **delta**; at the cap, open findings persist to `.reports/backlog.md` and "continue fixing" resumes there. Every phase is a git commit (`[plan]`, `[build]`, `[review rN]`, `[fix rN]`) in the folder's own repo; any commit opens a colorized diff and is one confirm from rollback. **Escalation**: a quick edit that turns out big says `ESCALATE:` and promotes to the full pipeline with partial work committed. **Code Map** — a per-repo deterministic symbol skeleton (functions/classes with signature + `file:line`, via `ast` / focused JS-TS extractor / universal-ctags) in SQLite + FTS5, plus an LLM-authored semantic layer (architecture, data-model, UI, per-file purpose). Git-blob-hash gated, so only changed files re-index. Every code agent gets a `codemap` tool that returns **pointers, never source dumps**. **Your repos, your models, your machine** — run on a fresh VFS folder or link an existing local repo (read-write or read-only; commits use your git identity). Each role (planner, builder, reviewers, router, triage) resolves to your Library tiers — run on DeepSeek, mix in a reasoning model, or go 100% local with Ollama. Every turn reports agent runs and in/out tokens; a Stop button halts at the next phase boundary; Export produces a full Markdown transcript. --- ## Agent Archetypes A base library of **31 ready-made specialist roles** — each an archetype with a tuned prompt, default model tier, and tool set. Categories span Research & Intelligence (Deep Researcher, Market & Competitor Scanner, Fact Checker), Writing & Comms, Engineering (Software Implementer, Code Reviewer, Software Architect, Refactor & Simplifier, Debugger), Data & Analysis, Ops & Coordination, Investment & VC, and Vision & Multimedia. Archetypes are used everywhere teams are formed — Agent Forge composition, Basna routing, Vatra ownership, and Council panels. The curated set is a **base you extend**: create your own on the Library page (by hand or generated from a prompt), override a base archetype by id, or add new ones — they appear everywhere base ones do, with per-user learned reliability. --- ## Flows Flows are a declarative composition language for agent-native automations, running inside Flight Deck and dispatching steps to the agent pool. A Flow is a trigger plus an ordered list of steps: deterministic plumbing (triggering, routing, sequencing, guardrails) owned by Flight Deck, with agent judgment only in the steps that need it. Stored in `flows.db`; runs and per-step results are first-class so the UI shows a live run log. Every flow — hand-built, code-written, or model-authored — is a small text program validated by a deterministic parser; the model is never the source of truth. ### Step types - `tool` — deterministic single-tool RPC to a pooled agent (no LLM turn) - `agent` — scoped judgment turn on a pooled agent, optional file attach + per-step tool guardrails - `vision` — raw image-describe with no agent loop, memory, tools, or history - `branch` — `when` → `goto` conditional - `input` — ask the user a question mid-flow - `emit` — send to a channel ### Composition (0.5.0) - **`gosub` / args / `return`** — a flow calls another as a subroutine, passes `with k: v` arguments, uses `{{calls..output}}`, and `return`s a value from anywhere. Flows are functions. - **`spawn` / `join` / futures** — launch a flow as an independent background worker and collect it later (with timeout). Parallelism turns 9s-in-series into 3s. - **`error` steps + `on error`** — any `gosub`/`join`/`spawn` can carry `on error -> `; an error handler reports `{{error.message}}` and recovers. - **Live control from any channel** — `/flow status | pause | resume | stop [handle|all]`, or Pause/Resume/Stop buttons in the run log. Concurrent flows each get a short handle. ### Authoring - **Visual builder**, a **Code** tab with a declarative DSL + live validator (lossless round-trip), or **describe it in plain English** and a model writes the DSL (run through the real parser with one-shot auto-repair). - **Self-authoring** — `synthesize_flow` turns a goal into a validated, call-only flow in a separate scratch space. It earns promotion by running well (3 clean runs → ⭐ candidate; 3 failures → quarantined), with tiered TTL + GC. A synthesized flow may not call a permanent world-acting flow until promoted, so agent-written automations can't borrow your vetted authority. - **Triggers** — case-insensitive substring matches on chosen channels (any/whatsapp/web/glasses), plus `has_image`/`has_video`/`has_audio`/`has_text`, cron, and `always`. Full language reference: `FLOWS.md` (also in-app via the Flow language docs button in the Code tab). --- ## Plan Mode Flip the planning toggle and every message routes through `/plan` + `/plan-execute`. `PlanGenerator` turns a request into an ordered DAG of 3–8 `OrchestratorTask`s with concrete descriptions and acceptance criteria; `PlanExecutor` runs them through the DAG runner; `PlanVerifier` judges each step against its acceptance criteria (optional JSON-schema fast path before the LLM judge); `PlanReviser` rewrites failing steps inside a bounded loop. A live plan card renders inline in chat. Deliverable steps must name their output file so results aren't lost in the run transcript. Plans persist as workflow JSON and reload byte-identical. --- ## Intentions A control-plane primitive (`intentions` tool + Flight Deck panel) sitting between *noticing* (insights) and *doing* (cron/scheduler). User intentions are notes-to-self surfaced back into context when relevant; the assistant can also propose and, with permission, act on future actions. Paired with the proactive Flight Deck scheduler, it delivers permissioned pushes over channels like WhatsApp. --- ## Runtimes & Providers - **Sign in with ChatGPT** — OpenAI OAuth reusing the Codex CLI tokens (`~/.codex/auth.json`); talks to the ChatGPT Responses API on your ChatGPT plan, no `OPENAI_API_KEY` needed. Auto-refresh on expiry. - **DeepSeek** — first-class provider. - **Nano Mode** — restricted tool set (`shell`, `write`, `read`, `edit`, `glob`, `datastore`, `insights`), a compressed ~40-line system prompt, and aggressive context filtering for tiny local models (Qwen3, Llama 3.2, Phi, 3B–8B). - **Remote GPU via vast.ai** — drive an Ollama server on a rented GPU with auto-wake on first request and idle-sleep. - **On-device Gemma** — LiteRT subprocess worker runs `.litertlm` models in an isolated child process that can crash and respawn without losing the session. - **Smart `web_fetch`** — auto-falls back from a plain HTTP fetch to a headless browser when the page is JS-rendered. - **Tavily** — optional web-search provider alongside the default Brave. --- ## Web Interface ### Main Pages - **Flight Deck** — Multi-agent command center (port 25080): Agent Desktop, Quick Chat, Spawn Agent, Agent Forge, Library, Council, Basna, Code, Flows, VFS, Agent Folders, Observatory, Autonomous Work, Scheduler, Skills, Admin - **Quick Chat** — Workspace page to talk to an archetype instantly; the agent spawns hidden from the desktop (full chat: plan mode, attachments, next steps) with a one-click "Promote to desktop" - **Chat** — Real-time streaming with message history, model selector - **Computer** — Research workspace with themed visual rendering, tabbed output (Answer, Blueprint, Files, Todos, Data, Insights, Visual) - **Orchestrator** — DAG visualization for parallel multi-session execution - **Brain Graph** — 3D force-directed cognitive topology visualization - **Insights Browser** — Search and manage auto-extracted facts - **Nervous System** — View autonomous dream-cycle discoveries - **Observatory** — Live stream of consciousness: standing intentions, thoughts, dreams, and fleet-wide signals - **Agent Folders / VFS** — Browse, preview, and download the shared virtual file system - **Reflections** — Review self-assessment directives - **Datastore** — Browse/edit relational tables, run SQL - **Settings** — Visual configuration editor - **Home** — Card-based dashboard with 24+ module cards ### BYOK Public Mode Per-session isolation with access codes. Users provide their own LLM API keys (stored in browser only, not server-side). --- ## Platform Integrations - **Web UI** — Full-featured at http://localhost:23080 - **Desktop** — Electron app for macOS, Linux, Windows - **WhatsApp** — Two-way personal-assistant bridge (Meta Cloud API): inbound text, voice notes (Soniox STT), images, video, location, contacts; outbound text, optional voice replies, and documents. Allow-list gated, with proactive pushes from the scheduler and Intentions. - **Meta Ray-Ban Display glasses** — Mobile-web → agent → glasses-web pipeline. Type from your phone (optionally attach a photo); the reply renders with full markdown and optional streaming Soniox TTS on the glasses. PWA-installable mobile bridge. - **Telegram** — Full native bot with photos, documents, contacts - **Slack** — Message and thread handling - **Discord** — Channel and DM support - **Twitter/X** — OAuth 2.0 with mentions, DMs, tweet posting, media upload - **Google Workspace** — Drive, Docs, Sheets, Slides, Gmail, Calendar - **Android** — Via Termux API (camera, GPS, battery control) --- ## Safety Three-layer guard system: 1. **Input guards** — Screen user inputs before LLM processing 2. **Output guards** — Check model responses before presenting 3. **Script/Tool guards** — Validate shell commands and tool calls before execution Each guard operates in `ask_for_approval` or `stop_suspicious` mode. Configurable blocking patterns for shell commands. Full audit trail with unique IDs for every message, tool call, insight, and intuition. --- ## Architecture The Agent class is composed of 13 specialized mixins: - AgentOrchestrationMixin — Core complete() and stream() entry points - AgentCompletionMixin — Completion gate, finalization, coverage validation - AgentContextMixin — Context building, memory injection, system prompt assembly - AgentToolLoopMixin — Tool execution, streaming, error handling - AgentGuardMixin — Guard policy, input/output/script_tool checks - AgentScaleLoopMixin — Large-scale task detection and micro-loop orchestration - AgentPipelineMixin — Pipeline trace, micro-loop summary building - AgentSessionMixin — Session management and persistence - AgentPlaybookMixin — Playbook injection and learning - AgentReasoningMixin — Extended thinking and reasoning - AgentChunkedProcessingMixin — Small-context model support ### Process of Thoughts (Full Lineage Traceability) Every element has unique IDs enabling traversal: - Message IDs (12-char hex) - Insight provenance (source_message_id + supersedes_id) - Intuition provenance (source_message_id for dream cycles) - Todo hierarchy (parent_id, triggered_by_id) --- ## Themes 14 built-in retro themes: Amiga Workbench, Atari ST 1040, C64 8032, Classic Mac, Windows 3.1, Hacker, Modern, Windows 11, macOS, iPhone, Android, Nokia 7710, Nokia Communicator. Custom themes supported via downloadable templates. --- ## Skills System OpenClaw-compatible SKILL.md files with: - Auto-discovery from workspace, managed, and plugin directories - Install from GitHub: `/skill install ` - Persistent skill environment with variable overrides --- ## CLI Interfaces - `captain-claw` — Interactive terminal - `captain-claw --tui` — Full terminal UI with rich formatting - `captain-claw-web` — Web server (default port 23080) - `flight-deck` — Flight Deck multi-agent command center (default port 25080) - `captain-claw-orchestrate` — Headless orchestrator CLI - `captain-claw --onboarding` — Re-run setup wizard