Benchmark encoding against ffmpeg cli

[Dan-Flores]

This PR creates a benchmark to compare VideoEncoder against FFmpeg CLI. These tools aren't one-to-one, so some assumptions are made:

For VideoEncoder, we use this simple workflow:

encoder = VideoEncoder(frames=frames, frame_rate=30)
encoder.to_file(dest=output_path, codec="h264_nvenc", extra_options={"qp": 1})

For FFmpeg CLI, we count the time used to write frames from a tensor to a file if the flag is used: --write_frames

if write_frames:
	raw_frames = frames.permute(0, 2, 3, 1).contiguous()[:num_frames]
    	with open(raw_path, "wb") as f:
        	f.write(raw_frames.cpu().numpy().tobytes())
        
ffmpeg_cmd = [...]
subprocess.run(ffmpeg_cmd, check=True, capture_output=True)

Result Summary:

• VideoEncoder shows better performance on GPU + CPU.
  • When the time required to write frames to bytes is added, FFmpeg CLI is much slower.
• On GPU, VideoEncoder shows a significant speed improvement, up to 3.5x faster than FFmpeg CLI for decoding 30 frames, without adding the time required to write frames to bytes.
  • NVENC utilization is higher for VideoEncoder, while Median GPU memory used values are the same.
• On CPU, FFmpeg CLI has a slight edge without adding the time required to write frames to bytes. Otherwise, VideoEncoder is significantly faster.

Details

All benchmarks are run using a 1280x720 video: Command to generate video: `ffmpeg -f lavfi -i testsrc2=duration=600:size=1280x720:rate=30 -c:v libx264 -pix_fmt yuv420p test/resources/testsrc2_10min.mp4`

Benchmarking nasa_13013.mp4, writing frames in FFmpeg

$ python benchmarks/encoders/benchmark_encoders.py

Benchmarking 390 frames from nasa_13013.mp4 over 30 runs:
Decoded 390 frames of size 270x480

VideoEncoder on GPU   med = 119.26 ms, max = 122.06 ms, fps = 3270.1
GPU memory used:      med = 1231.0 MB, max = 1231.0 MB
NVENC utilization:    med = 30.0%,     max = 38.0%

FFmpeg CLI on GPU     med = 1174.55 ms, max = 1524.59 ms, fps = 332.0
GPU memory used:      med = 1231.0 MB, max = 1231.0 MB
NVENC utilization:    med = 15.0%,     max = 22.0%

VideoEncoder on CPU   med = 408.43 ms, max = 454.66 ms, fps = 954.9

FFmpeg CLI on CPU     med = 1184.47 ms, max = 1219.28 ms, fps = 329.3

Benchmarking nasa_13013.mp4, with --skip-write-frames

$ python benchmarks/encoders/benchmark_encoders.py --skip-write-frames

Benchmarking 390 frames from nasa_13013.mp4 over 30 runs:
Decoded 390 frames of size 270x480

VideoEncoder on GPU   med = 120.21 ms, max = 122.40 ms, fps = 3244.4
GPU memory used:      med = 1231.0 MB, max = 1231.0 MB
NVENC utilization:    med = 26.0%,     max = 39.0%

FFmpeg CLI on GPU     med = 419.66 ms, max = 1189.17 ms, fps = 929.3
GPU memory used:      med = 1231.0 MB, max = 1231.0 MB
NVENC utilization:    med = 18.0%,     max = 23.0%

VideoEncoder on CPU   med = 408.86 ms, max = 449.01 ms, fps = 953.9

FFmpeg CLI on CPU     med = 383.65 ms, max = 410.91 ms, fps = 1016.5

[mollyxu]

Are we currently using qp=1 for torchcodec encoder vs qp=0 for ffmpeg cli? (line 155)

[Dan-Flores]

Yes, and we should not be, thanks for catching this!

[mollyxu]

Great work on the benchmarks @Dan-Flores! I liked the detailed analysis of the results. I left two clarifying questions.

[NicolasHug]

Thanks @Dan-Flores , this looks good!

[NicolasHug]

On NVENCMonitor above, I think we might want to use pynvml instead, as done e.g. in P1984513849.

The main reason is that NVENCMonitor is sampling utilization value every 50ms, which isn't exactly in sync with the number of iterations in the loop. That is, the returned nvenc_tensor doesn't contain the same amount of values the times tensor, and so their reported values aren't averaged over the same amount of experiments either.

[Dan-Flores]

Thanks for the example, I'll update to use pynvml.

I see how arbitrarily selecting 50ms will not get the same amount of values as the times tensor, but I don't completely understand how pynvml.nvmlDeviceGetDecoderUtilization manages it. It seems like it is always sampling the device for usage, then when called returns a single median/max/average over an automatically determined sampling period when called?

[NicolasHug]

Nit, no strong opinion but I find that reporting the mean isn't super useful when the std is small enough. Same for min and max (one is enough). It makes the logs slightly easier to read.

[NicolasHug]

I think we'll want the complete view an not exclude the zero values. That the utilization is sometimes zero is actually relevant information!

[Dan-Flores]

The reason I skipped any zero value was that for shorter benchmarks with the previous NVENC Monitor approach, the NVENC was not active most of the time, so the median was often zero. It seems the pynvml library is smarter about sampling frequency, so this is not a problem anymore!

[NicolasHug]

Do we need this? I think we should be able to remove the corresponding parameter in write_raw_frames and just use len(frames)?

[NicolasHug]

Nit, something similar is already printed above. We only need one

[NicolasHug]

Same here. We should only need a single if cuda_available: block and benchmark both torchcodec and the ffmpeg cli in there.