Source video (excerpt from ffprobe):

Duration: 00:00:53.25, start: 0.000000, bitrate: 100135 kb/s
Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 3840x2160 [SAR 1:1 DAR 16:9], 100113 kb/s, 29.97 fps, 29.97 tbr, 30k tbn, 59.94 tbc (default)

Intel i7-1185G7, Ubuntu 22.04:
Recoding to scaled down video: 7 seconds (7.5x)
Command used: ffmpeg -hide_banner -y -stats -v error -hwaccel vaapi -hwaccel_output_format vaapi -i DJI_0004.MP4 -vf 'scale_vaapi=w=640:h=-2,format=nv12|vaapi,hwupload' -codec:v h264_vaapi -g 1 -bf 0 -qp 26 -acodec ac3 -ab 128k -sn -dn -map 0 output-x264-vaapi-scale.mkv

Nvidia 1060 GTX (with Intel i7-4790K), Ubuntu 22.04, nvidia-driver-550.67
Recoding to scaled down video: 10 seconds (5.1x)
Command used: fmpeg -hide_banner -y -stats -v error -threads 8 -hwaccel cuda -hwaccel_output_format cuda -c:v h264_cuvid -resize 640x360  -i DJI_0004.MP4  -vcodec h264_nvenc  -rc cbr -acodec aac -ab 192k -sn -dn -map 0 -ignore_unknown output-NVENC-h264.mp4

Alternate command: fmpeg -hide_banner -y -stats -v error -hwaccel cuda -hwaccel_output_format cuda  -i DJI_0004.MP4 -vf scale_cuda=640:-2,hwupload -vcodec h264_nvenc -vb 30000k -rc cbr -acodec aac -ab 192k -sn -dn -map 0 -ignore_unknown output-NVENC-h264-scale_cuda.mp4

Both nvidia "encodings" complete within 10-11 secs, during which time GPU usage is around 11-12% (shown by nvtop)