memory fault analysis gfx1151

# GPU Memory Access Fault - Root Cause Analysis

**Date:** 2026-01-23
**System:** Fedora Linux 43, Kernel 6.18.5-200.fc43.x86_64
**Hardware:** AMD Ryzen AI Max+ 395 w/ Radeon 8060S (gfx1151)

## Root Cause: Missing amdkfd Kernel Module

### Evidence from Debug Logs:

**Log Line 36 (H3 - Kernel Modules):**
```json
{"amdgpu": true, "amdkfd": false, ...}
```

**System Verification:**
- `amdgpu` module: ✅ Loaded
- `amdkfd` module: ❌ **NOT FOUND** in kernel
- `modprobe amdkfd`: "Module not found in directory /lib/modules/6.18.5-200.fc43.x86_64"

### What is amdkfd?

`amdkfd` (AMD Kernel Fusion Driver) is a **critical kernel module** required for:
- ROCm compute operations
- GPU memory allocation for compute workloads
- HSA (Heterogeneous System Architecture) runtime GPU memory management

**Without amdkfd:**
- Display/graphics work (handled by amdgpu)
- GPU compute operations **FAIL** with memory access faults
- PyTorch GPU tensor allocation crashes

### Why is amdkfd Missing?

1. **Kernel 6.18.5 is very new** - ROCm kernel modules may not be available yet
2. **No ROCm kernel module packages installed** - Only user-space ROCm packages are present
3. **Kernel's built-in amdgpu** doesn't include amdkfd (it's a separate module)

### Current System State:

- ✅ ROCm user-space: Installed (ROCm 6.4.2 + some 7.1.1 packages)
- ✅ PyTorch ROCm: Installed (ROCm 7.11 nightly from gfx1151)
- ✅ HSA Runtime: Working (GPU detected, memory pools found)
- ✅ Architecture: Correctly detected as gfx1151 (after removing override)
- ❌ **amdkfd kernel module: MISSING**

## Hypothesis Evaluation Summary:

| Hypothesis | Status | Evidence |
|------------|--------|----------|
| H1: HSA Runtime | ✅ CONFIRMED | GPU agent detected, memory pools found |
| H2: Memory Mapping | ⚠️ PARTIAL | Basic mapping exists, some permission issues |
| **H3: amdkfd Module** | **❌ CONFIRMED (ROOT CAUSE)** | **Module not found in kernel** |
| H4: IOMMU | ✅ REJECTED | Not needed for integrated GPUs |
| H5: Version Compatibility | ⚠️ INCONCLUSIVE | Kernel 6.18.5 very new |
| H6: PyTorch Allocation | ✅ CONFIRMED | Fails at allocation step |
| H7: Device Permissions | ✅ CONFIRMED | Device files accessible |

## Possible Solutions:

### Option 1: Install ROCm Kernel Modules (If Available)

Check if ROCm DKMS packages are available for kernel 6.18.5:
```bash
dnf search rocm-dkms
dnf search rocm-kernel
```

If available, install:
```bash
sudo dnf install rocm-dkms
```

### Option 2: Compile ROCm Kernel Modules from Source

If packages aren't available, may need to:
1. Download ROCm kernel module source code
2. Compile against kernel 6.18.5
3. Install the modules

This is complex and may require kernel development tools.

### Option 3: Use Older Kernel (If Compatible)

Downgrade to a kernel version that has ROCm kernel module support:
- Kernel 6.6 LTS
- Kernel 6.1 LTS
- Check ROCm compatibility matrix

### Option 4: Wait for Official Support

Kernel 6.18.5 is very new. ROCm kernel modules may be available in:
- Future Fedora updates
- Future ROCm releases
- Community patches

### Option 5: Use CPU Training (Current Workaround)

Continue using CPU training, which is working well:
```bash
./scripts/start_pretraining.sh --cpu --epochs 450 --include_labeled
```

With 32 cores and 128GB RAM, CPU training is efficient.

## Recommendation:

**For now: Continue with CPU training** until:
1. ROCm kernel modules become available for kernel 6.18.5, OR
2. You can compile/install amdkfd module from source

The architecture detection issue (gfx1100 vs gfx1151) has been fixed by removing `HSA_OVERRIDE_GFX_VERSION`. The remaining memory fault is due to the missing amdkfd kernel module, which requires system-level installation that may not be available for your kernel version yet.