Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- Using: https://github.com/NVIDIA/nccl-tests?tab=readme-ov-file
- Command: nccl-tests build/all_reduce_perf -b 8 -e 128M -f 2 -g 2
- Forked driver: https://github.com/tinygrad/open-gpu-kernel-modules
- Before:
- # nThread 1 nGpus 2 minBytes 8 maxBytes 134217728 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
- #
- # Using devices
- # Rank 0 Group 0 Pid 2748 on qualid device 0 [0x01] NVIDIA GeForce RTX 4070 Ti SUPER
- # Rank 1 Group 0 Pid 2748 on qualid device 1 [0x03] NVIDIA GeForce RTX 4070 Ti SUPER
- #
- # out-of-place in-place
- # size count type redop root time algbw busbw #wrong time algbw busbw #wrong
- # (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
- 8 2 float sum -1 6.21 0.00 0.00 0 6.23 0.00 0.00 0
- 16 4 float sum -1 6.16 0.00 0.00 0 6.18 0.00 0.00 0
- 32 8 float sum -1 6.29 0.01 0.01 0 6.34 0.01 0.01 0
- 64 16 float sum -1 6.28 0.01 0.01 0 6.30 0.01 0.01 0
- 128 32 float sum -1 6.32 0.02 0.02 0 6.44 0.02 0.02 0
- 256 64 float sum -1 6.56 0.04 0.04 0 6.44 0.04 0.04 0
- 512 128 float sum -1 6.78 0.08 0.08 0 6.63 0.08 0.08 0
- 1024 256 float sum -1 7.01 0.15 0.15 0 6.78 0.15 0.15 0
- 2048 512 float sum -1 7.07 0.29 0.29 0 7.00 0.29 0.29 0
- 4096 1024 float sum -1 7.62 0.54 0.54 0 7.50 0.55 0.55 0
- 8192 2048 float sum -1 8.69 0.94 0.94 0 8.40 0.98 0.98 0
- 16384 4096 float sum -1 10.78 1.52 1.52 0 10.51 1.56 1.56 0
- 32768 8192 float sum -1 14.23 2.30 2.30 0 13.92 2.35 2.35 0
- 65536 16384 float sum -1 22.09 2.97 2.97 0 21.55 3.04 3.04 0
- 131072 32768 float sum -1 37.58 3.49 3.49 0 37.15 3.53 3.53 0
- 262144 65536 float sum -1 52.97 4.95 4.95 0 52.63 4.98 4.98 0
- 524288 131072 float sum -1 85.45 6.14 6.14 0 86.20 6.08 6.08 0
- 1048576 262144 float sum -1 156.8 6.69 6.69 0 157.6 6.65 6.65 0
- 2097152 524288 float sum -1 292.0 7.18 7.18 0 290.7 7.21 7.21 0
- 4194304 1048576 float sum -1 555.1 7.56 7.56 0 554.1 7.57 7.57 0
- 8388608 2097152 float sum -1 1066.4 7.87 7.87 0 1067.5 7.86 7.86 0
- 16777216 4194304 float sum -1 2126.5 7.89 7.89 0 2124.0 7.90 7.90 0
- 33554432 8388608 float sum -1 4235.0 7.92 7.92 0 4230.0 7.93 7.93 0
- 67108864 16777216 float sum -1 8397.7 7.99 7.99 0 8424.7 7.97 7.97 0
- 134217728 33554432 float sum -1 16732 8.02 8.02 0 16746 8.01 8.01 0
- # Out of bounds values : 0 OK
- # Avg bus bandwidth : 3.38645
- #
- After:
- # nThread 1 nGpus 2 minBytes 8 maxBytes 134217728 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
- #
- # Using devices
- # Rank 0 Group 0 Pid 10959 on qualid device 0 [0x01] NVIDIA GeForce RTX 4070 Ti SUPER
- # Rank 1 Group 0 Pid 10959 on qualid device 1 [0x03] NVIDIA GeForce RTX 4070 Ti SUPER
- #
- # out-of-place in-place
- # size count type redop root time algbw busbw #wrong time algbw busbw #wrong
- # (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
- 8 2 float sum -1 5.26 0.00 0.00 0 5.17 0.00 0.00 0
- 16 4 float sum -1 5.28 0.00 0.00 0 5.17 0.00 0.00 0
- 32 8 float sum -1 5.31 0.01 0.01 0 5.23 0.01 0.01 0
- 64 16 float sum -1 5.34 0.01 0.01 0 5.27 0.01 0.01 0
- 128 32 float sum -1 5.45 0.02 0.02 0 5.31 0.02 0.02 0
- 256 64 float sum -1 5.44 0.05 0.05 0 5.33 0.05 0.05 0
- 512 128 float sum -1 5.75 0.09 0.09 0 5.46 0.09 0.09 0
- 1024 256 float sum -1 5.58 0.18 0.18 0 5.52 0.19 0.19 0
- 2048 512 float sum -1 5.67 0.36 0.36 0 5.55 0.37 0.37 0
- 4096 1024 float sum -1 5.83 0.70 0.70 0 5.77 0.71 0.71 0
- 8192 2048 float sum -1 6.33 1.29 1.29 0 6.21 1.32 1.32 0
- 16384 4096 float sum -1 7.69 2.13 2.13 0 7.71 2.12 2.12 0
- 32768 8192 float sum -1 10.65 3.08 3.08 0 10.33 3.17 3.17 0
- 65536 16384 float sum -1 15.55 4.21 4.21 0 15.42 4.25 4.25 0
- 131072 32768 float sum -1 25.74 5.09 5.09 0 25.75 5.09 5.09 0
- 262144 65536 float sum -1 36.45 7.19 7.19 0 36.16 7.25 7.25 0
- 524288 131072 float sum -1 56.45 9.29 9.29 0 56.54 9.27 9.27 0
- 1048576 262144 float sum -1 94.47 11.10 11.10 0 94.45 11.10 11.10 0
- 2097152 524288 float sum -1 175.1 11.98 11.98 0 175.1 11.97 11.97 0
- 4194304 1048576 float sum -1 336.6 12.46 12.46 0 336.4 12.47 12.47 0
- 8388608 2097152 float sum -1 659.2 12.73 12.73 0 659.0 12.73 12.73 0
- 16777216 4194304 float sum -1 1308.2 12.82 12.82 0 1308.0 12.83 12.83 0
- 33554432 8388608 float sum -1 2608.1 12.87 12.87 0 2605.9 12.88 12.88 0
- 67108864 16777216 float sum -1 5203.7 12.90 12.90 0 5203.8 12.90 12.90 0
- 134217728 33554432 float sum -1 10395 12.91 12.91 0 10394 12.91 12.91 0
- # Out of bounds values : 0 OK
- # Avg bus bandwidth : 5.34378
- #
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
