cgfixit

OpenClaw + RAG (Without SKYNET vibes ;)

Jan 31st, 2026 (edited)
321
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 26.81 KB | Software | 0 0
  1. ═══════════════════════════════════════════════════════════════════════════════
  2. OPENCLAW OFFLINE-FIRST WITH GATED FALLBACK
  3. (Production Flow - Feb 2026)
  4. v2 (refined)
  5. ═══════════════════════════════════════════════════════════════════════════════
  6.  
  7. ┌──────────────────┐
  8. │ User Query │
  9. │ (e.g., Veeam HA)│
  10. └────────┬─────────┘
  11. │
  12. v
  13. ┌────────────────────────┐
  14. │ OpenClaw Controller │
  15. │ (Clawbot Framework) │
  16. │ Enforce YAML/JSON Rules│
  17. └────────────┬───────────┘
  18. │
  19. ┌────────────v────────────┐
  20. │ Strong Prompt: │
  21. │ ALWAYS RAG FIRST │
  22. │ (Score ≥0.75) │
  23. └────────────┬────────────┘
  24. │
  25. ┌──────────────────┴──────────────────┐
  26. │ │
  27. ┌─────────v──────────┐ ┌─────────────v────────┐
  28. │ MCP TOOL (Priority)│ │ FALLBACK PATH │
  29. │ Offline RAG │ │ (Gated) │
  30. │ rag_query.py │ │ │
  31. └─────────┬──────────┘ │ │
  32. │ │ │
  33. ┌───────────┴─────────┐ │ │
  34. │ │ │ │
  35. ┌───v────┐ ┌────v───┐ ┌───v─────┐ │
  36. │ Ollama │ │ ChromaDB│ │ Low Hit │ │
  37. │Embeddings Vault │ │ Rate │ │
  38. │nomic-embed .md Files (<0.75) │ │
  39. └───┬────┘ └────┬───┘ └──┬──────┘ │
  40. │ │ │ │
  41. └────────┬───────────┘ │ Miss or Low Conf │
  42. │ │ │
  43. v └──────────────────────┘
  44. ┌─────────────────┐ │
  45. │ Hit Rate ≥0.75? │ │
  46. └────────┬────────┘ │
  47. │ YES │
  48. v v
  49. ┌──────────────────┐ ┌───────────────────────┐
  50. │ LM Studio Synth │ │ Vault Miss + General │
  51. │ Response Gen │ │ Reasoning (Offline) │
  52. │ llama.cpp GGUF │ │ │
  53. └────────┬─────────┘ └───────────┬───────────┘
  54. │ │
  55. │ ┌──────────v──────────┐
  56. │ │ Ask "Online? (y/n)" │
  57. │ └──────────┬──────────┘
  58. │ │
  59. │ ┌────────────┴─────────┐
  60. │ │ │
  61. │ ┌─────v──┐ ┌───────v───┐
  62. │ │ YES │ │ NO/Retry │
  63. │ │ (User) │ │ (User) │
  64. │ └─────┬──┘ └───┬───────┘
  65. │ │ │
  66. │ ┌─────────v────────┐ │
  67. │ │ Online Fallback │ │
  68. │ │ Grok/Claude │ │
  69. │ │ OpenRouter/xAI │ │
  70. │ │ **RISK-GATED** │ │
  71. │ └─────────┬────────┘ │
  72. │ │ │
  73. └──────────┬───────┴──────────────┘
  74. │
  75. ┌───────────v────────────┐
  76. │ Response to User │
  77. └───────────┬────────────┘
  78. │
  79. ┌───────────────┴────────────────┐
  80. │ │
  81. ┌────v──┐ ┌───────v────┐
  82. │Logging │ │ Pite Store │
  83. │System │ │ Hit Rate │
  84. │(SQLite)│ │ Query Log │
  85. └────┬───┘ │ Patterns │
  86. │ └────────────┘
  87. v
  88. ┌──────────────────┐
  89. │ Query Patterns │
  90. │ Hit Rate Stats │
  91. │ Performance │
  92. │ Analysis │
  93. └──────────────────┘
  94.  
  95. ═══════════════════════════════════════════════════════════════════════════════
  96. PRIORITY ENFORCEMENT (Strong Prompt Forces this Flow)
  97. ═══════════════════════════════════════════════════════════════════════════════
  98.  
  99. 1️⃣ OFFLINE RAG FIRST (99% of queries)
  100. - Score ≥0.75: Return Chroma hit + LM Studio synthesis
  101. - Score <0.75: Trigger fallback gate (user confirmation required)
  102. - NEVER call online without explicit user approval
  103.  
  104. 2️⃣ USER CONFIRMATION GATE (Security Boundary)
  105. - "Vault miss. Query online for fallback? (y/n)"
  106. - Prevents silent data exfiltration
  107. - Logs all online-flagged queries to Pite
  108.  
  109. 3️⃣ ONLINE FALLBACK (Gated + Logged)
  110. - Only if user confirms "y"
  111. - Grok/Claude via OpenRouter
  112. - LM Studio can proxy (OPENAIAPIBASE=https://openrouter.ai/api/v1)
  113. - Logged to Pite for audit trail
  114.  
  115. 4️⃣ LOGGING & ANALYTICS (Pite Integration)
  116. - Hit Rate: % queries satisfied offline
  117. - Query Patterns: Common misses for vault expansion
  118. - Performance: Latency per phase (embed → search → synth)
  119.  
  120. ═══════════════════════════════════════════════════════════════════════════════
  121. IMPLEMENTATION SKELETON
  122. ═══════════════════════════════════════════════════════════════════════════════
  123.  
  124. config.yaml (OpenClaw):
  125. tools:
  126. - name: rag_query
  127. script: /path/to/rag_query.py
  128. description: "Offline RAG (ALWAYS called first per strong prompt)"
  129.  
  130. system_prompt: |
  131. CRITICAL: For EVERY user query:
  132. 1. Always call rag_query first (MCP tool)
  133. 2. If hit (score ≥0.75) → use vault context in response
  134. 3. If miss → state "not in vault" + general knowledge
  135. 4. NEVER initiate online fallback yourself
  136. 5. Wait for user confirmation on fallback prompt
  137.  
  138. rag_query.py (MCP Tool):
  139. def rag_query(query):
  140. embeddings = OllamaEmbeddings(model="nomic-embed-text")
  141. vectorstore = Chroma(persist_dir="./chromadb")
  142. docs = vectorstore.as_retriever(
  143. search_type="similarity_score_threshold",
  144. search_kwargs={"score_threshold": 0.75, "k": 5}
  145. ).invoke(query)
  146.  
  147. hit = len(docs) > 0
  148. log_to_pite(query, hit, len(docs)) # Analytics
  149.  
  150. return {
  151. "status": "hit" if hit else "miss",
  152. "chunks": [d.page_content for d in docs] if hit else [],
  153. "hit_count": len(docs)
  154. }
  155.  
  156. fallback_controller.py (OpenClaw Hook):
  157. def handle_fallback(query, rag_result):
  158. if rag_result["status"] == "miss":
  159. user_choice = input("⚠️ Vault miss. Query online? (y/n): ").strip()
  160. log_to_pite(query, "fallback_asked", user_choice)
  161.  
  162. if user_choice.lower() == "y":
  163. response = call_grok_or_claude(query) # OpenRouter
  164. log_to_pite(query, "fallback_used", response)
  165. return response
  166. else:
  167. return "Staying offline. Not in vault & user declined online."
  168.  
  169. pite_logging.py (SQLite + Analytics):
  170. def log_to_pite(query, hit_status, details):
  171. conn = sqlite3.connect("./pite_analytics.db")
  172. conn.execute("""
  173. INSERT INTO query_log (timestamp, query, hit_status, details)
  174. VALUES (datetime('now'), ?, ?, ?)
  175. """, (query, hit_status, json.dumps(details)))
  176. conn.commit()
  177.  
  178. # Dashboard query:
  179. SELECT COUNT(*) as total,
  180. SUM(CASE WHEN hit_status='hit' THEN 1 ELSE 0 END) as vault_hits,
  181. 100.0 * SUM(CASE WHEN hit_status='hit' THEN 1 ELSE 0 END) / COUNT(*) as hit_rate
  182. FROM query_log;
  183.  
  184. ═══════════════════════════════════════════════════════════════════════════════
  185. KEY METRICS TO TRACK (Pite Dashboard)
  186. ═══════════════════════════════════════════════════════════════════════════════
  187.  
  188. Hit Rate: % queries answered offline (goal: >85%)
  189. Miss Patterns: Gaps in vault (e.g., "no docs on X")
  190. Latency: Embed (50ms) → Search (10ms) → Synth (500ms) = ~560ms offline
  191. Online Calls: Count & reasons (audit trail for security)
  192. Score Dist: Histogram of similarity scores (refine threshold)
  193.  
  194. ═══════════════════════════════════════════════════════════════════════════════
  195.  
  196.  
  197.  
  198.  
  199.  
  200.  
  201.  
  202.  
  203.  
  204.  
  205.  
  206. ╔═══════════════════════════════════════════════════════════════════╗
  207. ║ CHRIS'S OPENCLAW SETUP ║
  208. ║ (Tier 1: Strong Prompt) ║
  209. ╚═══════════════════════════════════════════════════════════════════╝
  210.  
  211. ┌─────────────────────────┐
  212. │ User Query (HTTP) │ ◄────── Browser/CLI/IDE Extension
  213. │ "Veeam HA best tips?" │
  214. └────────────┬────────────┘
  215. │
  216. v
  217. ┌───────────────────────────────────────────────────────────────────┐
  218. │ OPENCLAW AGENT │
  219. │ ┌─────────────────────────────────────────────────────────┐ │
  220. │ │ System Prompt (config.yaml) │ │
  221. │ │ ┌─────────────────────────────────────────────────────┐ │ │
  222. │ │ │ CRITICAL: For ANY query, call rag_query first. │ │ │
  223. │ │ │ Vault info → answer from vault only. │ │ │
  224. │ │ │ No vault → state "not in vault" + general answer. │ │ │
  225. │ │ └─────────────────────────────────────────────────────┘ │ │
  226. │ └─────────────────────────────────────────────────────────┘ │
  227. │ │
  228. │ Controller Logic: Decides tools needed based on query │
  229. └───────────────────────┬────────────────────────────────────────────┘
  230. │
  231. v (1. Query Analysis)
  232. ┌───────────────────────────────┐
  233. │ Should I use RAG? ──> YES │ (Strong prompt forces this)
  234. └───────────────┬───────────────┘
  235. │
  236. v (2. MCP Tool Call)
  237. ┌───────────────────────────────────────────────────────────────────┐
  238. │ MCP TOOL INTERFACE │
  239. │ ┌─────────────────────────────────────────────────────────┐ │
  240. │ │ rag_script.py (Python Bridge) │ │
  241. │ │ • Receives user query as arg │ │
  242. │ │ • Calls embedding model │ │
  243. │ │ • Returns JSON with retrieved docs │ │
  244. │ └──────────────────────┬──────────────────────────────────┘ │
  245. └─────────────────────────┼─────────────────────────────────────────┘
  246. │
  247. v (3. Generate Query Embedding)
  248. ┌───────────────────────────────────────────────────────────────────┐
  249. │ OLLAMA SERVICE (Localhost:11434) │
  250. │ ┌─────────────────────────────────────────────────────────┐ │
  251. │ │ Embedding Model: nomic-embed-text (768-dim vectors) │ │
  252. │ │ • Converts query → [0.234, -0.891, 0.445, ...] │ │
  253. │ │ • Cached in RAM for fast repeat queries │ │
  254. │ └──────────────────────┬──────────────────────────────────┘ │
  255. └─────────────────────────┼─────────────────────────────────────────┘
  256. │
  257. v (4. Similarity Search)
  258. ┌───────────────────────────────────────────────────────────────────┐
  259. │ VECTOR DATABASE │
  260. │ ┌─────────────────────────────────────────────────────────┐ │
  261. │ │ ChromaDB (Persistent Disk Storage) │ │
  262. │ │ • Pre-indexed .md files from /vault/ │ │
  263. │ │ • Cosine similarity search against query embedding │ │
  264. │ │ • Returns top-k chunks (default k=5) │ │
  265. │ │ Collection: "veeam_docs" (or your collection name) │ │
  266. │ └──────────────────────┬──────────────────────────────────┘ │
  267. └─────────────────────────┼─────────────────────────────────────────┘
  268. │
  269. v (5. Return Context to Agent)
  270. ┌───────────────────────────────────────────────────────────────────┐
  271. │ OPENCLAW AGENT │
  272. │ • Receives: "Top 3 matches from vault + relevance scores" │
  273. │ • Constructs prompt: [CONTEXT] {vault_chunks} [QUERY] {question} │
  274. │ • Sends to LLM for final answer synthesis │
  275. └───────────────────────┬────────────────────────────────────────────┘
  276. │
  277. v (6. LLM Reasoning)
  278. ┌───────────────────────────────────────────────────────────────────┐
  279. │ LM STUDIO (Inference Engine) │
  280. │ ┌─────────────────────────────────────────────────────────┐ │
  281. │ │ Primary Model: Llama 3.1 8B (or Qwen 2.5 14B) │ │
  282. │ │ • Context: 8k-128k tokens (model dependent) │ │
  283. │ │ • Reads vault context + user query │ │
  284. │ │ • Generates final answer │ │
  285. │ │ Endpoint: http://localhost:1234/v1/chat/completions │ │
  286. │ └─────────────────────────────────────────────────────────┘ │
  287. │ │
  288. │ ┌─────────────────────────────────────────────────────────┐ │
  289. │ │ OPTIONAL: API Proxy (for external models) │ │
  290. │ │ • Grok API (xAI) ────> /v1/chat/completions │ │
  291. │ │ • Claude API (Anthropic) ────> /v1/chat/completions │ │
  292. │ │ LM Studio routes to external endpoints when configured │ │
  293. │ └─────────────────────────────────────────────────────────┘ │
  294. └───────────────────────┬────────────────────────────────────────────┘
  295. │
  296. v (7. Response Assembly)
  297. ┌───────────────────────────────────────────────────────────────────┐
  298. │ OPENCLAW AGENT │
  299. │ • Final answer from LLM │
  300. │ • Formats response with sources/citations │
  301. │ • Returns to user interface │
  302. └───────────────────────┬────────────────────────────────────────────┘
  303. │
  304. v (8. Display)
  305. ┌─────────────────────────┐
  306. │ User Interface Output │
  307. │ ┌───────────────────┐ │
  308. │ │ Answer: Based on │ │
  309. │ │ vault doc X... │ │
  310. │ │ │ │
  311. │ │ Sources: │ │
  312. │ │ - veeam_ha.md │ │
  313. │ └───────────────────┘ │
  314. └─────────────────────────┘
  315.  
  316.  
  317. ╔═══════════════════════════════════════════════════════════════════╗
  318. ║ DATA FLOW SUMMARY ║
  319. ╠═══════════════════════════════════════════════════════════════════╣
  320. ║ 1. Query → OpenClaw Agent (strong prompt enforces RAG call) ║
  321. ║ 2. Agent → MCP Tool (rag_script.py) ║
  322. ║ 3. Script → Ollama (embedding generation, cached in RAM) ║
  323. ║ 4. Embedding → ChromaDB (similarity search on .md index) ║
  324. ║ 5. Results → Agent (context chunks returned) ║
  325. ║ 6. Agent → LM Studio (prompt + context for reasoning) ║
  326. ║ 7. LM Studio → Agent (synthesized answer) ║
  327. ║ 8. Agent → User (formatted response with sources) ║
  328. ╚═══════════════════════════════════════════════════════════════════╝
  329.  
  330.  
  331. ╔═══════════════════════════════════════════════════════════════════╗
  332. ║ KEY COMPONENTS ║
  333. ╠═══════════════════════════════════════════════════════════════════╣
  334. ║ OpenClaw Agent │ Orchestration layer (Node/TS + MCP protocol) ║
  335. ║ LM Studio │ LLM inference (local + optional API proxy) ║
  336. ║ Ollama │ Embedding model service (nomic-embed-text) ║
  337. ║ ChromaDB │ Vector store (persistent, indexed .md files) ║
  338. ║ rag_script.py │ Bridge: query → embedding → search → results ║
  339. ║ config.yaml │ System prompt (CRITICAL RAG instruction) ║
  340. ║ /vault/*.md │ Your knowledge base (pre-indexed) ║
  341. ╚═══════════════════════════════════════════════════════════════════╝
  342.  
  343. NOTES:
  344. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  345. • All processing 100% local except optional Grok/Claude API calls
  346. • Ollama embedding model stays resident in RAM (fast repeated queries)
  347. • ChromaDB index built once, queried many times (no re-indexing needed)
  348. • Strong prompt = 85-95% RAG reliability without pre-processor complexity
  349. • LM Studio can hot-swap models (Llama/Qwen/etc.) without code changes
  350. • MCP protocol = language-agnostic tool interface (Python/JS/Rust/etc.)
  351.  
  352. https://www.perplexity.ai/search/whats-the-deal-with-clawbot-mo-6nnbZds4SZmkO8j78nTkMg#16
  353. https://cgfixit.com/livecode
Advertisement
Add Comment
Please, Sign In to add comment