pop_os-skill

# Claude Code Prompt: Build Pop!_OS Diagnosis Skill

Paste this entire prompt into Claude Code to build your `popos-diagnosing` skill.

---

## PROMPT START

I want to create a Claude Code skill called `popos-diagnosing` for deep system diagnosis on Pop!_OS 24.04 LTS (COSMIC DE, Ubuntu 24.04 base). My main problem right now is unexplained high CPU usage, but I want the skill to cover a comprehensive set of subsystems.

Please build this skill following Anthropic's official skill authoring best practices:
- SKILL.md with proper YAML frontmatter (name, description in third person)
- Progressive disclosure architecture: metadata → SKILL.md body → bundled references
- Concise descriptions that trigger reliably
- Bundled reference files for each diagnostic domain
- Python/bash scripts where deterministic steps are needed

---

### SKILL GOAL

The skill must guide Claude to perform structured, expert-level Linux system diagnosis on Pop!_OS 24.04 LTS, covering:

1. **CPU** — high usage root-cause analysis (the priority problem)
2. **RAM & Swap** — pressure, OOM events, swap usage patterns
3. **I/O Pressure** — disk latency, I/O wait, scheduler analysis
4. **NVIDIA GPU** — utilization, VRAM, temperature, driver health, process attribution
5. **Systemd & Services** — failed units, zombie processes, runaway services
6. **Kernel & Logs** — dmesg anomalies, journald errors, audit events
7. **Pop!_OS–specific** — COSMIC DE process health, system76-scheduler, pop-upgrade status, firmware (fwupd)
8. **eBPF/Advanced** — flame graph generation, off-CPU analysis, scheduler latency (optional deep-dive)

---

### DIAGNOSTIC METHODOLOGY

For each subsystem, the skill should apply Brendan Gregg's **USE Method** (Utilization, Saturation, Errors) as the analytical framework. Tools to use per domain:

**CPU:**
```bash
# Tier 1 – quick triage
top -b -n1 -o %CPU | head -20
ps aux --sort=-%cpu | head -20
mpstat -P ALL 1 5
pidstat -u 1 5

# Tier 2 – scheduler & runqueue
runqlat          # from bcc-tools
runqlen          # from bcc-tools
perf stat -a sleep 5

# Tier 3 – flame graphs (deep-dive)
sudo perf record -F 99 -ag -- sleep 30
sudo perf script | stackcollapse-perf.pl | flamegraph.pl > cpu-flame.svg

# eBPF one-liners
sudo bpftrace -e 'profile:hz:99 { @[kstack] = count(); } interval:s:10 { print(@); exit(); }'
sudo execsnoop   # catch short-lived CPU-spike processes
```

**RAM & Swap:**
```bash
free -h
vmstat -s
cat /proc/meminfo | grep -E 'MemTotal|MemFree|Buffers|Cached|SwapTotal|SwapFree|Dirty|Writeback'
swapon --show
sudo slabtop -o          # kernel slab cache
sudo bpftrace -e 'tracepoint:oom:mark_victim { printf("OOM killed: %s PID %d\n", comm, pid); }'
journalctl -k | grep -i 'oom\|killed process'
```

**I/O Pressure:**
```bash
iostat -xz 1 5
iotop -bod 5 -n 3
cat /proc/pressure/io     # PSI (Pressure Stall Information)
cat /proc/pressure/cpu
cat /proc/pressure/memory
sudo biolatency -m        # from bcc-tools: I/O latency histogram
sudo biosnoop             # per-operation I/O tracing
sudo biotop               # top processes by I/O
lsblk -o NAME,MODEL,ROTA,SCHED,SIZE,MOUNTPOINT
cat /sys/block/*/queue/scheduler
```

**NVIDIA GPU:**
```bash
nvidia-smi
nvidia-smi --query-gpu=timestamp,name,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used,power.draw,pstate --format=csv -l 5
nvidia-smi pmon -s u      # per-process utilization
nvtop                     # interactive GPU monitor (apt install nvtop)
journalctl -k | grep -i nvidia
dmesg | grep -i 'nvidia\|gpu\|xid'   # Xid errors = GPU hardware issues
```

**Systemd & Services:**
```bash
systemctl --failed
systemctl list-units --state=failed
journalctl -p err -b      # errors since last boot
journalctl -p warning -b --no-pager | tail -50
ps aux | awk '$8 == "Z"'  # zombie processes
systemd-analyze blame | head -20
systemd-analyze critical-chain
```

**Kernel & Logs:**
```bash
dmesg -T --level=err,warn | tail -50
journalctl -k -b -p warning --no-pager | tail -50
last reboot
uptime
uname -r
```

**Pop!_OS–Specific:**
```bash
# COSMIC DE process health
ps aux | grep -E 'cosmic|pop-shell|system76'
systemctl --user status cosmic-session 2>/dev/null || true

# system76-scheduler (CPU scheduler tuning daemon)
systemctl status system76-scheduler
journalctl -u system76-scheduler -b --no-pager | tail -20

# Pop!_OS upgrade daemon
systemctl status pop-upgrade
journalctl -u pop-upgrade -b --no-pager | tail -20

# Firmware
fwupdmgr get-updates 2>/dev/null | head -20

# NVIDIA driver on Pop (uses system76 NVIDIA scripts)
ls /etc/X11/ | grep nvidia
prime-select query 2>/dev/null || echo "PRIME not available"
```

---

### SKILL STRUCTURE TO BUILD

```
popos-diagnosing/
├── SKILL.md                        # frontmatter + orchestration logic
└── references/
    ├── cpu-diagnosis.md            # CPU deep-dive procedures
    ├── memory-swap-diagnosis.md    # RAM/swap procedures
    ├── io-pressure-diagnosis.md    # I/O PSI procedures
    ├── nvidia-diagnosis.md         # GPU procedures
    ├── systemd-logs-diagnosis.md   # services/journal procedures
    └── popos-specific.md           # COSMIC/system76 procedures
```

---

### SKILL.md FRONTMATTER REQUIREMENTS

```yaml
---
name: popos-diagnosing
description: >
  Diagnoses Pop!_OS 24.04 LTS (COSMIC DE) system performance issues covering CPU,
  RAM, swap, I/O pressure, NVIDIA GPU, systemd services, kernel logs, and
  Pop!_OS-specific components (system76-scheduler, pop-upgrade, COSMIC DE).
  Use this skill whenever the user mentions high CPU usage, slowdowns, GPU issues,
  system hangs, kernel errors, memory pressure, swap thrashing, I/O wait, failed
  services, dmesg errors, or any Pop!_OS/COSMIC DE problem. Also trigger for
  general Linux performance diagnosis or "my system feels slow" on Ubuntu-based systems.
---
```

---

### SKILL ORCHESTRATION LOGIC (SKILL.md body)

The SKILL.md body should instruct Claude to:

1. **Triage first** — run a fast 60-second snapshot across all subsystems to identify which area needs deep-dive (never run all deep-dive commands at once)
2. **Apply USE Method** — for the identified hotspot, check Utilization → Saturation → Errors in order
3. **Progressive depth** — Tier 1 (built-in tools: top, free, iostat), Tier 2 (bcc-tools: biolatency, runqlat), Tier 3 (eBPF: bpftrace, flame graphs) — only escalate if Tier 1 is inconclusive
4. **Check prerequisites** — verify tool availability before running (`which bpftrace`, `dpkg -l bcc-tools`)
5. **Summarize findings** — produce a structured report: Problem | Evidence | Likely Cause | Recommended Fix
6. **High CPU priority path** — when "high CPU" is the complaint, start with `execsnoop` (catches short-lived processes that vanish before `top` sees them), then `runqlat`, then flame graph

---

### INSTALL PREREQUISITES SCRIPT

The skill should include guidance to install missing tools:

```bash
# Essential triage tools (always available on Pop!_OS)
# top, ps, vmstat, iostat (sysstat), free, dmesg, journalctl, systemctl

# Install extended tools if missing
sudo apt update
sudo apt install -y \
  sysstat \          # iostat, mpstat, pidstat, sar
  iotop \            # I/O top
  htop \             # enhanced top
  nvtop \            # GPU top
  bpfcc-tools \      # biolatency, biosnoop, runqlat, execsnoop, etc.
  bpftrace \         # eBPF one-liners
  linux-tools-common \  # perf
  linux-tools-$(uname -r) \  # kernel-specific perf
  flamegraph         # flame graph generation (or install from GitHub)
```

---

### HIGH CPU DIAGNOSIS DECISION TREE

Include this in the skill as a decision tree Claude should follow:

```
Is CPU high?
├── Is it user-space or kernel? (us% vs sy% in top/mpstat)
│   ├── USER-SPACE HIGH
│   │   ├── Run execsnoop → catch short-lived bursts
│   │   ├── Run ps aux --sort=-%cpu → identify top consumer
│   │   ├── Run pidstat -u 1 10 → confirm sustained usage
│   │   └── Generate CPU flame graph for that PID
│   └── KERNEL HIGH (sy% high)
│       ├── Check runqlat → scheduler saturation?
│       ├── Check biolatency → I/O blocking?
│       ├── Check dmesg for kernel errors
│       └── Run bpftrace kernel stack profiler
├── Is iowait% high? (wa% in top)
│   └── → Switch to I/O Pressure diagnosis path
├── Is softirq/interrupt high? (si/hi% in top)
│   └── → Check /proc/interrupts, network driver issues, NVIDIA IRQ
└── Is it a specific time pattern?
    └── Check cron jobs, systemd timers: systemctl list-timers
```

---

### NVIDIA-SPECIFIC HIGH CPU CORRELATION

Include this because NVIDIA drivers on Linux often cause CPU spikes:

```bash
# Check if nvidia kernel module is causing CPU overhead
cat /proc/interrupts | grep -i nvidia
# Check Xorg GPU usage
nvidia-smi pmon -s u -d 5
# Check for Xid (GPU error) events which can trigger CPU recovery
dmesg | grep -i 'Xid'
# Common Xid codes: Xid 31 = GPU MMU fault, Xid 79 = GPU hung
```

---

### OUTPUT FORMAT

The skill should instruct Claude to produce findings in this structure:

```
## Pop!_OS System Diagnosis Report

### Triage Summary
| Subsystem | Status | Key Metric |
|-----------|--------|------------|
| CPU       | 🔴 HIGH | 94% user, pid 1234 (firefox) |
| RAM       | 🟡 OK   | 12.4/16 GB, 2.1 GB swap |
| I/O       | 🟢 OK   | <5ms latency |
| GPU       | 🟢 OK   | 23% util, 55°C |

### Root Cause Analysis
**Subsystem:** CPU
**Evidence:** [command output]
**Likely Cause:** [explanation]
**Recommended Fix:** [actionable steps]

### Commands Run
[list of commands executed]
```

---

### TEST CASES TO VALIDATE THE SKILL

After building the skill, please test it with these prompts:

1. `"My Pop!_OS system has very high CPU usage and I can't find which process is causing it"`
2. `"System feels sluggish, how do I check if it's RAM or swap pressure?"`
3. `"My NVIDIA GPU temperature is very high on Pop!_OS, how do I investigate?"`
4. `"Several systemd services are failing on my Pop!_OS 24.04, help me diagnose"`
5. `"I'm getting dmesg kernel errors on Pop!_OS, help me understand them"`
6. `"How do I generate a CPU flame graph on Pop!_OS to find what's using CPU?"`

---

### SKILL AUTHORING CONSTRAINTS (follow strictly)

- Keep SKILL.md body under 500 lines; put domain details in `references/` files
- Description must be in **third person** (not "I can help you...")
- Only ~100 tokens loaded at startup (name + description metadata)
- Full SKILL.md loads only when triggered; reference files load on demand
- Include `when to read` annotations on each reference file pointer in SKILL.md
- Never assume tools are installed — always check with `which <tool>` first
- For commands requiring sudo, note it explicitly and explain why

Please build the complete skill directory, write all reference files, and then run the test cases above to validate it works correctly. Start with the SKILL.md and the CPU and GPU reference files first, as those cover my immediate problem.

## PROMPT END