https://sider.ai/share/d11cf2c90f07a03f9514d33af688eb80 (Option D Hybrid Ideally)

# Offline MCP Multi-Model Project
## Primary Source Reference Document
**Version**: 1.0 | **Date**: December 2024 | **Author**: CG

---

## Project Vision

Build an offline-first AI assistant using:
- **RAG database**: Curated `.md` files with vector embeddings
- **Local LLM**: Ollama running open-source models
- **MCP Protocol**: Standardized client-host-server communication
- **Optional Online**: Multi-model routing via Groq/OpenRouter when enabled

---

## Core Architecture

```
┌─────────────────────────────────────────────────────────────────────────┐
│                         HOST (Orchestrator)                              │
│   - LangChain or custom Python                                          │
│   - RAG retrieval from .md corpus                                       │
│   - Routing logic (offline vs online, simple vs complex)                │
│   - Security: sanitize context before sending to cloud                  │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│   ┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐      │
│   │  MCP CLIENT 1    │  │  MCP CLIENT 2    │  │  MCP CLIENT 3    │      │
│   │  (Ollama)        │  │  (Groq)          │  │  (OpenRouter)    │      │
│   │  Transport:stdio │  │  Transport:HTTP  │  │  Transport:HTTP  │      │
│   └────────┬─────────┘  └────────┬─────────┘  └────────┬─────────┘      │
│            │                     │                     │                 │
│   ┌────────▼─────────┐  ┌────────▼─────────┐  ┌────────▼─────────┐      │
│   │  MCP SERVER 1    │  │  MCP SERVER 2    │  │  MCP SERVER 3    │      │
│   │  Ollama Wrapper  │  │  Groq API        │  │  Claude/Grok     │      │
│   │  (Local)         │  │  (Cloud)         │  │  (Cloud)         │      │
│   └──────────────────┘  └──────────────────┘  └──────────────────┘      │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘
```

---

## MCP Protocol Fundamentals

### Source: modelcontextprotocol.io/specification/2025-11-25

**Core Principles**:
1. JSON-RPC 2.0 message format over any transport
2. Stateful sessions with capability negotiation
3. Host controls security; servers are isolated
4. Servers cannot see full conversation or other servers

### Transport Options

| Transport | Use Case | Key Requirements |
|-----------|----------|------------------|
| **stdio** | Local/offline | Newline-delimited JSON, no embedded newlines, stderr for logs |
| **Streamable HTTP** | Cloud/remote | Single endpoint, SSE for streaming, session management via headers |

### stdio Transport Rules (MUST follow)
```
- Messages: Single-line JSON-RPC, terminated by \n
- stdout: ONLY valid MCP messages
- stderr: Logging/debug output
- Client spawns server as subprocess
```

### Capability Negotiation
```json
// Client → Server
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "initialize",
  "params": {
    "protocolVersion": "2025-11-25",
    "capabilities": {"sampling": {}},
    "clientInfo": {"name": "my-host", "version": "1.0"}
  }
}

// Server → Client
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "protocolVersion": "2025-11-25",
    "capabilities": {"tools": {}, "resources": {"subscribe": true}},
    "serverInfo": {"name": "ollama-mcp", "version": "1.0"}
  }
}
```

---

## Ollama MCP Wrapper Implementation

### Why Needed
Ollama exposes REST API at `http://localhost:11434`, not MCP JSON-RPC over stdio.
Wrapper translates between protocols.

### Minimal Implementation
```python
# ollama_mcp_server.py
import sys
import json
import requests

OLLAMA_URL = "http://localhost:11434"
DEFAULT_MODEL = "llama3.2"

def handle_message(msg: dict) -> dict:
    method = msg.get("method")
    msg_id = msg.get("id")

    if method == "initialize":
        return {
            "jsonrpc": "2.0",
            "id": msg_id,
            "result": {
                "protocolVersion": "2025-11-25",
                "capabilities": {"tools": {}, "sampling": {}},
                "serverInfo": {"name": "ollama-mcp", "version": "1.0"}
            }
        }

    elif method == "sampling/createMessage":
        # Extract prompt from MCP params
        messages = msg.get("params", {}).get("messages", [])
        prompt = "\n".join([m.get("content", "") for m in messages])

        # Call Ollama REST API
        response = requests.post(
            f"{OLLAMA_URL}/api/generate",
            json={"model": DEFAULT_MODEL, "prompt": prompt, "stream": False}
        )
        result = response.json()

        return {
            "jsonrpc": "2.0",
            "id": msg_id,
            "result": {
                "content": {"type": "text", "text": result.get("response", "")},
                "model": DEFAULT_MODEL,
                "stopReason": "endTurn"
            }
        }

    else:
        return {
            "jsonrpc": "2.0",
            "id": msg_id,
            "error": {"code": -32601, "message": f"Method not found: {method}"}
        }

# Main stdio loop
for line in sys.stdin:
    try:
        msg = json.loads(line.strip())
        response = handle_message(msg)
        sys.stdout.write(json.dumps(response) + "\n")
        sys.stdout.flush()
    except Exception as e:
        sys.stderr.write(f"Error: {e}\n")
```

### Config for MCP Client (e.g., Claude Desktop, Dive)
```json
{
  "mcpServers": {
    "ollama-local": {
      "command": "python",
      "args": ["/path/to/ollama_mcp_server.py"]
    }
  }
}
```

---

## LangChain Integration (Current API - Dec 2024)

### Source: python.langchain.com

**Note**: API has evolved. Old `RouterChain` deprecated. Current pattern:

```python
# Install: pip install langchain langchain-ollama langchain-openai

from langchain_ollama import ChatOllama
from langchain_openai import ChatOpenAI
from langchain.agents import create_agent

# Local model (offline)
local_llm = ChatOllama(model="llama3.2", base_url="http://localhost:11434")

# Cloud model via OpenRouter (online)
cloud_llm = ChatOpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key="sk-or-...",
    model="x-ai/grok-beta"
)

# Simple routing function
def route_query(query: str, offline_mode: bool = False):
    if offline_mode:
        return local_llm.invoke(query)

    # Complexity-based routing
    if len(query) < 200:
        return local_llm.invoke(query)  # Simple → local
    else:
        return cloud_llm.invoke(query)  # Complex → cloud
```

### Agent Pattern (Current)
```python
from langchain.agents import create_agent

def search_docs(query: str) -> str:
    """Search local .md documentation"""
    # RAG retrieval logic here
    return "Retrieved context..."

agent = create_agent(
    model="llama3.2",  # or cloud model
    tools=[search_docs],
    system_prompt="You are a helpful IT assistant with access to documentation."
)

result = agent.invoke({"messages": [{"role": "user", "content": "How do I configure backups?"}]})
```

---

## RAG with .md Files

### Vector Store Options (Local)

| Store | Pros | Cons |
|-------|------|------|
| **Chroma** | Easy setup, persistent | Moderate performance |
| **FAISS** | Fast, Facebook-backed | More setup |
| **LanceDB** | Serverless, efficient | Newer, less docs |

### Basic RAG Pipeline
```python
from langchain_community.document_loaders import DirectoryLoader, TextLoader
from langchain_text_splitters import MarkdownHeaderTextSplitter
from langchain_community.vectorstores import Chroma
from langchain_ollama import OllamaEmbeddings

# 1. Load .md files
loader = DirectoryLoader("./docs", glob="**/*.md", loader_cls=TextLoader)
docs = loader.load()

# 2. Split by headers (preserves structure)
splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=[("#", "h1"), ("##", "h2"), ("###", "h3")]
)
chunks = []
for doc in docs:
    chunks.extend(splitter.split_text(doc.page_content))

# 3. Embed locally
embeddings = OllamaEmbeddings(model="nomic-embed-text")

# 4. Store in Chroma
vectorstore = Chroma.from_documents(chunks, embeddings, persist_directory="./chroma_db")

# 5. Query
retriever = vectorstore.as_retriever(search_kwargs={"k": 5})
relevant_docs = retriever.invoke("backup configuration")
```

---

## OpenRouter Integration

### Source: openrouter.ai/docs

**Key Features**:
- Unified API for 100+ models
- OpenAI-compatible format
- Automatic prompt caching (Grok, OpenAI, Gemini)
- Token usage tracking

### Prompt Caching (Verified)

| Provider | Cache Write Cost | Cache Read Cost |
|----------|------------------|-----------------|
| Grok | Free | 0.25x input price |
| OpenAI | Free | 0.25-0.50x input price |
| Claude | 1.25x input price | 0.10x input price |
| Gemini | Free + 5min storage | 0.25x input price |

### Usage Example
```python
import openai

client = openai.OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key="sk-or-..."
)

response = client.chat.completions.create(
    model="x-ai/grok-beta",  # or "anthropic/claude-3-haiku"
    messages=[{"role": "user", "content": "Your query"}]
)
```

---

## xAI Grok Pricing & Tiers

### Subscription Tiers (as of Dec 2024)
| Tier | Price | Includes |
|------|-------|----------|
| Free | $0 | Limited Grok 3 access on X |
| SuperGrok | $30/mo | Expanded Grok 3, some Grok 4 |
| SuperGrok Heavy | $300/mo | Full Grok 4 Heavy access |

**⚠️ Subscriptions ≠ API access**. API requires separate xAI developer account.

### API Pricing
| Model | Input | Output |
|-------|-------|--------|
| grok-4-fast | $0.20/M | $0.50/M |
| grok-4 | $3/M | $15/M |
| grok-4-heavy | Higher | Higher |

---

## LoRAX Alternative (Advanced)

### Source: github.com/predibase/lorax

**What it is**: Multi-LoRA inference server. Serves thousands of fine-tuned adapters from single GPU.

**When relevant**: If you fine-tune multiple LoRA adapters on different subsets of your .md corpus (e.g., networking docs, security docs, Veeam docs), LoRAX lets you dynamically load the right adapter per query.

**Trade-off**: Higher complexity than RAG-only approach. Only justified if fine-tuning provides measurably better results than prompt injection.

---

## Learning Path

### Phase 1: Foundations (Current)
1. ✅ MCP Protocol (Architecture, Transports)
2. → **LangChain** (agents, chains, RAG)
3. → Ollama local setup

### Phase 2: Implementation
4. Build Ollama MCP wrapper
5. Implement RAG pipeline with .md files
6. Test offline-only mode

### Phase 3: Hybrid
7. Add OpenRouter/Groq for online fallback
8. Implement routing logic
9. Add caching layer

### Phase 4: Advanced (Optional)
10. Fine-tune LoRA adapters (if RAG insufficient)
11. LoRAX for multi-adapter serving
12. PyTorch fundamentals (for custom model work)

---

## Key Resources

### Official Documentation
| Resource | URL | Notes |
|----------|-----|-------|
| MCP Spec | modelcontextprotocol.io/specification/2025-11-25 | Authoritative protocol reference |
| LangChain | python.langchain.com/docs | Current API docs |
| Ollama | github.com/ollama/ollama | API reference in /docs |
| OpenRouter | openrouter.ai/docs | Pricing, caching, routing |
| LoRAX | github.com/predibase/lorax | Multi-LoRA serving |
| Hugging Face | huggingface.co | Models, PEFT, Transformers |

### Tutorials & Guides
| Topic | Source | Status |
|-------|--------|--------|
| Local MCP Server | stainless.com/mcp/local-mcp-server | ✅ Fetched |
| MCP + Ollama | Medium (kpetropavlov) | ✅ Fetched |
| MCP Tool Use + Ollama | Medium (Renaissance Learning) | ✅ Fetched |
| IBM RAG with .md | developer.ibm.com | ❌ 404 (find alternative) |
| Improved RAG with Markdown | Medium (Data Science) | Not fetched |

### Community
| Topic | Source |
|-------|--------|
| MCP Servers | github.com/modelcontextprotocol (official repos) |
| Ollama Discord | discord.gg/ollama |
| LangChain Discord | discord.gg/langchain |

---

## Grok Voice Agent API (Future Integration)

### Source: x.ai/news/grok-voice-agent-api (blocked during fetch)

**Announced capabilities** (from your quote):
> "With our API, developers can effortlessly integrate their own custom tools or tap into xAI's powerful real-time search capabilities across X and the web."

**Potential project integration**:
- Voice interface for your RAG assistant
- Real-time X/web search as fallback when local docs insufficient
- Custom tool registration (similar to MCP tools concept)

**Action item**: Monitor xAI developer portal for API availability.

---

## Version History

| Version | Date | Changes |
|---------|------|---------|
| 1.0 | 2024-12-20 | Initial reference document |