About DavidHLPL · RHINE LAB RSS ↗
FILE X-1035 / AI tools

Hindsight Runtime Troubleshooting Matrix and Failure Modes

ENGINEERING NOTES

Published / DavidHLPL

This page summarizes seven historical failures, diagnostic approaches, and response records from Hindsight’s local deployment and Agent integration for troubleshooting similar environments.

Scope and review (2026-09-08): Events come from the Hindsight v0.9.1 raw captured on 2026-08-17; event times were not separately recorded. That revision compared text with official protocol, Docker, and Ollama documentation, without deploying, downloading, uninstalling, or writing memories. Historical outcomes are source reports, not universal current-version rules or production validation.

1. Seven Typical Failure Modes and Troubleshooting Matrix

graph TD
  P1[Pitfall 1: Wrong third-party npm hindsight-mcp] --> S1[Uninstall it and use Hindsight's native FastMCP]
  P2[Pitfall 2: Source reports model not found] --> S2[Check the local alias and model provenance before choosing an import]
  P3[Pitfall 3: Docker UID 1000 permission failure] --> S3[Grant UID 1000 ownership of ~/.hindsight/data]
  P4[Pitfall 4: 6.63GB model sync download timeout] --> S4[Use Python chunked download with Range resume and nohup]
  P5[Pitfall 5: Ollama API missing /v1 returns 404] --> S5[Set BASE_URL explicitly to http://ollama-gpu:11434/v1]
  P6[Pitfall 6: FastMCP session loses its Session ID] --> S6[Capture and forward mcp-session-id in the bridge]
  P7[Pitfall 7: Bank intrusion and subdirectory drift] --> S7[Use OMP per-project plus Git-root-adaptive detection]

1. Installing the wrong package with the same name (hindsight-mcp)

  • Symptom: After npm install -g hindsight-mcp, its README requests https://api.hindsight-ai.com, a PAT Token, and an Agent UUID, and defines tools such as create_memory_block; it cannot connect to the Vectorize open-source endpoint.
  • Root cause: The npm registry contains a legacy third-party package with the same name; it is not an official Vectorize Hindsight artifact. Vectorize Hindsight embeds its own FastMCP service.
  • Solution: Uninstall the npm package and connect directly to the native /mcp/{bank}/ endpoint exposed by Hindsight.
  • Prevention: Before adding ecosystem tools around an open-source project, verify the exported protocol and integration directory (such as hindsight-integrations/) in the official repository.

2. Historical request reports gemma4:12b not found

  • Symptom: Startup LLM connectivity check returns ApiStatusError(ollama/gemma4:12b): HTTP 404: {"message": "model 'gemma4:12b' not found"}.
  • Evidence boundary: Only the requested Ollama instance lacking that model name is established. The raw reported an absent official tag, but supplied no registry snapshot. It cannot establish that only one model size existed or what is available today.
  • Historical handling: A GGUF and Modelfile created local alias gemma4:12b. A customizable alias proves neither architecture, parameter count, nor official identity. Original filenames/import steps remain on the deployment page.
  • Prerequisites: Check the model card, source revision, license, checksum, and chat template before choosing an import method. The revision did not verify the weights’ identity or current download URL, and supplies no new download instructions.

3. Non-root container volume permissions (UID 1000)

  • Symptom: Hindsight exits with [FAIL] The embedded database directory /home/hindsight/.pg0 is not writable by this container (UID 1000).
  • Cause: The image uses non-root user hindsight (UID 1000); a host-created directory may not be writable by its embedded pg0 process.
  • Response: First check mount target, container UID/GID, and host mapping, then change ownership only for dedicated directories. Historical chown 1000:1000 assumes confirmed mapping. UID 1000 does not imply a Rootless Docker daemon; Rootless also uses user namespaces, so host/container UIDs need not coincide. See deployment conditions.
  • Prevention: Explicitly verify UID/permissions for non-root image host mounts; do not blindly apply global 777.

4. Oversized model file (6.63GB) download timeout

  • Symptom: Running the download synchronously through an Agent tool exceeds the 300-second timeout, leaving an orphaned temporary file.
  • Root cause: Model weights are large and cross-border bandwidth can be limited; a synchronous blocking request is easily killed by the process manager or CLI client timeout.
  • Solution: Use a dedicated Python downloader with urllib.request and HTTP Range headers for resumable downloads, and run it as a background process with nohup python3 ... > download.log 2>&1 &.
  • Prevention: Choose transfer methods by size, bandwidth, and tool timeout budget; 1GB is not universal. Historical nohup is not process supervision. Resumption must confirm server Range support and validate the final file, without appending error responses to weights.

5. Ollama API Base URL is missing /v1

  • Symptom: Hindsight cannot connect when configured with HINDSIGHT_API_LLM_BASE_URL=http://ollama-gpu:11434.
  • Root cause: Hindsight’s llm_wrapper.py follows the OpenAI-compatible protocol for the ollama provider and builds requests below /v1 by default (http://localhost:11434/v1). Omitting /v1 with a custom container hostname produces a 404.
  • Solution: Set the complete URL explicitly in Docker Compose: HINDSIGHT_API_LLM_BASE_URL=http://ollama-gpu:11434/v1.
  • Prevention: Check the actual provider/SDK URL-joining rule. /v1 fixed the source environment; do not append it blindly for all clients.

6. Missing FastMCP Session ID causes Bad Request (-32600)

  • Symptom: A simple stdio bridge completes the first initialize request, but later tools/list or tools/call requests return Bad Request: Missing session ID.
  • Root cause: Hindsight uses Streamable FastMCP. After the initial initialize handshake succeeds, the server returns mcp-session-id in the response headers; every later JSON-RPC POST must forward that Session ID in its HTTP headers.
  • Solution: Add lightweight state management to the Node.js bridge: capture mcp-session-id from the first response and send it on subsequent requests.
  • Prevention: Under MCP 2025-06-18, a server may assign a Session ID; forwarding is required once it does. Headers are only part of the protocol. See the integration page for SSE, notification, and recovery gaps.

7. Global hard-coded Bank and subdirectory routing drift

  • Symptom: Hard-coding bank: ulticode contaminates facts when working on another project; deriving the name only from process.cwd() misidentifies a project when the developer is inside a subdirectory, splitting its memory.
  • Root cause: A monorepo or multi-module project often runs with a working directory deep inside a submodule, so the current directory name does not represent the Git repository. OMP’s per-project-tagged mode also has different routing semantics from Codex’s independent Bank.
  • Historical handling: OMP used hindsight.scoping: per-project; the bridge derived the Bank from Git-root basename. This fixes subdirectory drift within one repository, not collisions between identical basenames or normalized names; it is not a unique identifier.
  • Prevention: Check both clients’ actual Bank and explicitly assign distinct IDs if needed. See integration for non-Git fallback and logical isolation.

2. Quick Health Check SOP

Run these commands in order to inspect the infrastructure and Agent state:

# 1. Check the GPU inference container
docker logs hindsight-ollama-gpu | tail -n 20

# 2. List running models and loading details (load the model first)
docker exec hindsight-ollama-gpu ollama ps

# 3. Check Hindsight core logs and health
docker logs hindsight-server | tail -n 20
curl -I http://127.0.0.1:8888/health

# 4. Verify ownership of the persistent data directory
ls -la ~/.hindsight/data/

# 5. Syntax only; does not start the bridge or prove a handshake
node --check ~/.hindsight/hindsight-bridge.mjs

ollama list lists local models; ollama ps lists running models (CLI). Starting a stdio script and waiting for input is not validation: check initialize, notifications, tools/list, and controlled write/recall to the intended Bank. Those checks were not executed in that revision.

3. Six-Stage Evolution and Technical Decisions

The table preserves the source narrative. Missing model tags, successful downloads, temporary-Bank cleanup, and passing checks were reported then, not reverified in that revision.

Stage Core task Violation or failure found Final resolution and output
Stage 1: Requirements and architecture The user required a fully local Hindsight deployment, Gemma-4 12B on the GPU, BGE-M3 on the CPU, and unified OMP/Codex integration. Initially investigated whether a third-party npm package could provide the bridge directly. Established a two-container Docker Compose architecture (Ollama ROCm (GPU) + Hindsight Core (CPU pg0/BGE-M3)).
Stage 2: Separating fact from fiction Investigated the installed hindsight-mcp npm package. It pointed to api.hindsight-ai.com and required a PAT Token; it was a legacy third-party package. Uninstalled it and adopted Vectorize Hindsight’s native Streamable FastMCP /mcp/{bank}/ endpoint.
Stage 3: Container and model preparation Started the Docker containers and pulled the model. 1. The container failed because UID 1000 lacked write permission; 2. the official Ollama registry lacked the gemma4:12b tag; 3. a 6.63GB synchronous download timed out. 1. Granted UID 1000 ownership; 2. wrote a resumable background downloader; 3. built the GGUF import through a Modelfile.
Stage 4: Unifying the communication channels Connected Hindsight to Ollama and built the Codex bridge. 1. Hindsight’s LLM check returned 404; 2. the FastMCP bridge reported Missing session ID (-32600) after the handshake. 1. Added /v1 to BASE_URL; 2. made the bridge capture and forward the mcp-session-id header.
Stage 5: Memory isolation and routing evolution Shared and isolated memory between OMP and Codex. Hard-coded ulticode contaminated projects, while direct process.cwd() naming drifted inside /services/app/ subdirectories. 1. Set OMP scoping: per-project; 2. made the bridge resolve the Git root with git rev-parse --show-toplevel.
Stage 6: Removing temporary spaces and archiving the rules Removed test Banks and ran the full validation. Cleaned temporary codex and omp Banks; added raw snapshots and updated the Index/Log according to the KB constitution. Full knowledge-base validation passed (pnpm kb:lint, pnpm check, and pnpm build).

4. Evidence and Uncertainty

  • Source facts: The hindsight-local-deployment-and-agent-integration raw record contains the seven troubleshooting logs, Docker state, Session ID handshake behavior, and Git toplevel script verification.
  • Synthesis in this page: The troubleshooting logs and experience are distilled into a standard failure-mode matrix and decision-evolution table.
  • Unconfirmed: Streamable HTTP already supports SSE. Whether the historical bridge handles actual long-lived responses and full event framing remains untested. Model provenance, client versions, and nondefault deployments also need evidence.

Research notes

  1. Original document: src/content/note/en/hindsight-troubleshooting.md
  2. Source revision: f49953fb0d4bd52a39bf81167b0c4b7dd7d78a64
  3. Evidence status: active

Original source ↗