Quantizing AI Models with convert_to_quant (Windows + NVIDIA)

# Quantizing AI Models with convert_to_quant (Windows + NVIDIA)

A practical guide to setting up [silveroxides/convert_to_quant](https://github.com/silveroxides/convert_to_quant) on Windows and quantizing diffusion models to INT8 ConvRot for use in ComfyUI.

---

## What is INT8 ConvRot?

INT8 ConvRot is a quantization method that compresses model weights from 16-bit (BF16/FP16) down to 8-bit integers, roughly halving VRAM usage with minimal quality loss. The "ConvRot" part refers to a rotation-based technique that improves quantization accuracy by redistributing outliers before quantizing — giving you better results than naive INT8 at the same file size.

---

## Requirements

| Feature | Requirement |
|---|---|
| Minimum (FP8/INT8) | Python 3.10+, PyTorch 2.8+, CUDA 12.8+ |
| Full (NVFP4/MXFP8) | Python 3.12+, PyTorch 2.10+, CUDA 13.0+ |
| INT8 Kernels | Triton (Linux native, Windows via `triton-windows`) |

> **Note:** PyTorch must be installed manually with the correct CUDA version. The package intentionally does not install it automatically to avoid environment conflicts.

---

## Installation (Windows)

### 1. Create and activate a virtual environment

```powershell
python -m venv venv
.\venv\Scripts\Activate.ps1
```

### 2. Install PyTorch with CUDA 13.0

CUDA 13.0 is the current recommended version:

```powershell
pip install torch --index-url https://download.pytorch.org/whl/cu130
```

Check which torch version was installed:

```powershell
python -c "import torch; print(torch.__version__)"
```

### 3. Install Triton for Windows (required for INT8)

Match your Triton version to your torch version:

```powershell
# torch 2.10 or 2.11
pip install -U "triton-windows<3.7"

# torch 2.12
pip install -U "triton-windows<3.8"
```

### 4. Install convert_to_quant

```powershell
pip install convert-to-quant
```

Or from source:

```powershell
git clone https://github.com/silveroxides/convert_to_quant.git
cd convert_to_quant
pip install -e .
```

---

## How INT8 ConvRot Quantization Works

The basic command structure for INT8 ConvRot looks like this:

```powershell
ctq -i model_bf16.safetensors -o model_int8_convrot.safetensors \
  --int8 --scaling_mode tensor --simple --low-memory \
  --convrot --convrot-group-size [GROUP_SIZE] \
  --comfy_quant --save-quant-metadata
```

The key flags:

- `--int8` — use INT8 quantization instead of the default FP8
- `--scaling_mode tensor` — one global scale per tensor (most compatible)
- `--convrot` — enable the ConvRot rotation technique
- `--convrot-group-size` — group size for ConvRot; must evenly divide all 2D weight dimensions
- `--simple` — skip SVD optimization (faster, still good quality)
- `--low-memory` — stream tensors from disk instead of loading all at once (essential for large models)
- `--comfy_quant --save-quant-metadata` — ComfyUI compatibility

---

## Determining the Right ConvRot Group Size

Before running quantization, you need to find the largest power-of-4 that divides all 2D weight tensor dimensions in the model. Run this shape check:

```powershell
python -c "
from safetensors import safe_open
f = safe_open(r'path\to\your_model.safetensors', framework='pt')
shapes = set()
[shapes.add(f.get_slice(k).get_shape()[-1]) for k in f.keys() if len(f.get_slice(k).get_shape()) == 2]
print(sorted(shapes))
"
```

Then find the smallest dimension in that list and pick the largest value from `[4, 16, 64, 256]` that divides it evenly:

| Smallest dim | Max group size |
|---|---|
| 12 or 64 | 4 |
| 128+ (divisible by 64) | 64 |
| 256+ (divisible by 256) | 256 |

A smaller group size means slightly less accurate quantization, but it's still far better than no ConvRot at all.

---

## Handling Model-Specific Layer Exclusions

Some layers should be kept in full precision (BF16/F32) rather than quantized — typically input/output projections, embedding layers, and timestep pathways. Quantizing these can cause quality degradation or instability.

`convert_to_quant` includes built-in presets for common architectures (check them with `ctq --help-filters`). For models without a preset, you can identify and exclude the right layers manually.

### Step 1: Dump the layer list

```powershell
python -c "
from safetensors import safe_open
f = safe_open(r'path\to\model.safetensors', framework='pt')
for k in sorted(f.keys()):
    print(k, f.get_slice(k).get_shape())
" | Out-File -FilePath layers.txt

# Filter out repetitive block layers to find top-level structure
Get-Content layers.txt | Where-Object { $_ -notmatch "^blocks\." }
```

### Step 2: Identify what to protect

Look for layers matching these patterns:

- **Input projections** — `img_in`, `vid_in`, `txt_in`, `x_embedder`, `first`
- **Output projections** — `proj_out`, `vid_out`, `last`
- **Timestep/conditioning embeddings** — `time_text_embed`, `tmlp`, `tproj`, `emb_in`
- **Fixed conditioning tensors** — `positive_conditioning`, `negative_conditioning`
- **Tiny matrices** — anything with a dimension under 64 (e.g. `[1, 12]`, `[64, 3072]`)

### Step 3: Exclude them with a regex

```powershell
--exclude-layers "^(vid_in|vid_out|txt_in|emb_in|first|last|tmlp|tproj)"
```

---

## Real Examples

### Krea 2 Turbo (12B DiT, group size 4)

Krea 2 has a custom architecture with an `[6144, 64]` input projection that caps the group size at 4. The top-level layers `first`, `last`, `tmlp`, and `tproj` are excluded.

```powershell
ctq -i "krea2_turbo_bf16.safetensors" -o "krea2_turbo_int8_convrot.safetensors" `
  --int8 --scaling_mode tensor --simple --low-memory `
  --convrot --convrot-group-size 4 `
  --exclude-layers "^(first|last|tmlp|tproj|txtfusion\.projector)\." `
  --comfy_quant --save-quant-metadata
```

### SeedVR2 7B (video upscaler, group size 4)

SeedVR2's `vid_in.proj.weight` has a `[3072, 132]` shape — the 132 dimension caps the group size at 4. Input/output projections and conditioning tensors are excluded.

```powershell
ctq -i "seedvr2_7b_fp16.safetensors" -o "seedvr2_7b_int8_convrot.safetensors" `
  --int8 --scaling_mode tensor --simple --low-memory `
  --convrot --convrot-group-size 4 `
  --exclude-layers "^(vid_in|vid_out|txt_in|emb_in|negative_conditioning|positive_conditioning)" `
  --comfy_quant --save-quant-metadata
```

---

## Verifying the Output

After conversion, inspect which layers stayed in full precision:

```powershell
python -c "
from safetensors import safe_open
f = safe_open(r'path\to\output_model.safetensors', framework='pt')
for k in sorted(f.keys()):
    d = f.get_slice(k).get_dtype()
    if 'I8' not in str(d).upper() and 'U8' not in str(d).upper():
        print(k, f.get_slice(k).get_shape(), d)
"
```

You should see your excluded layers listed as `F32`, and everything else will show as `I8` (the quantized weights) plus `F32` scale tensors and `U8` metadata blobs — that's the expected structure for a ComfyUI-compatible INT8 ConvRot file.

---

## Tips

- **Group size 4 is still worth it.** Even with a small group size forced by a bottleneck dimension, ConvRot outperforms plain INT8.
- **Check built-in presets first.** Run `ctq --help-filters` — if your model architecture has a preset (like `--qwen`, `--wan`, `--ltxv2`), use it instead of manual exclusions.
- **`--low-memory` is almost always recommended** for models over a few GB — it keeps peak RAM manageable.
- **The `--simple` flag** skips learned rounding (SVD optimization). For most use cases the quality difference is negligible and it's dramatically faster.