Then, the results.
RTX 5090
| PP | TG | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s |
|-------|--------|--------|----------|----------|----------|----------|
| 2048 | 512 | 0 | 0.441 | 4641.54 | 6.669 | 76.78 |
| 2048 | 512 | 2048 | 0.464 | 4409.15 | 6.956 | 73.60 |
| 2048 | 512 | 4096 | 0.493 | 4153.09 | 7.323 | 69.92 |
| 2048 | 512 | 6144 | 0.524 | 3910.02 | 7.706 | 66.44 |
This is using about 425W.
RTX 4090
| PP | TG | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s |
|-------|--------|--------|----------|----------|----------|----------|
| 2048 | 512 | 0 | 0.565 | 3625.95 | 9.415 | 54.38 |
| 2048 | 512 | 2048 | 0.599 | 3420.78 | 10.007 | 51.17 |
| 2048 | 512 | 4096 | 0.637 | 3215.54 | 10.602 | 48.29 |
| 2048 | 512 | 6144 | 0.675 | 3034.13 | 11.059 | 46.30 |
This is using about 375W.
RTX 3090
| PP | TG | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s |
|-------|--------|--------|----------|----------|----------|----------|
| 2048 | 512 | 0 | 1.331 | 1538.49 | 11.435 | 44.78 |
| 2048 | 512 | 2048 | 1.374 | 1490.80 | 12.017 | 42.61 |
| 2048 | 512 | 4096 | 1.448 | 1414.76 | 12.700 | 40.32 |
| 2048 | 512 | 6144 | 1.524 | 1343.63 | 13.344 | 38.37 |
This is using about 360W.
RTX A6000
| PP | TG | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s |
|-------|--------|--------|----------|----------|----------|----------|
| 2048 | 512 | 0 | 1.297 | 1578.69 | 13.265 | 38.60 |
| 2048 | 512 | 2048 | 1.366 | 1499.08 | 13.984 | 36.61 |
| 2048 | 512 | 4096 | 1.440 | 1421.99 | 14.754 | 34.70 |
| 2048 | 512 | 6144 | 1.510 | 1356.03 | 15.553 | 32.92 |