jpeg - Meaningful thumbnails for a Video using FFmpeg

FFmpeg can capture images from videos that can be used as thumbnails to represent the video. Most common ways of doing that are captured in the FFmpeg Wiki.

But, I don't want to pick random frames at some intervals. I found some options using filters on FFmpeg to capture scene changes:

The filter thumbnail tries to find the most representative frames in the video:

ffmpeg -i input.mp4 -vf  "thumbnail,scale=640:360" -frames:v 1 thumb.png

and the following command selects only frames that have more than 40% of changes compared to previous (and so probably are scene changes) and generates a sequence of 5 PNGs.

ffmpeg -i input.mp4 -vf  "select=gt(scene\,0.4),scale=640:360" -frames:v 5 thumb%03d.png

Info credit for the above commands to Fabio Sonnati. The second one seemed better as I could get n images and pick the best. I tried it and it generated the same image 5 times.

Some more investigation led me to:

ffmpeg -i input.mp4 -vf "select=gt(scene\,0.5)" -frames:v 5 -vsync vfr  out%02d.png

-vsync vfr ensures that you get different images. This still always picks the first frame of the video, in most cases the first frame is credits/logo and not meaningful, so I added a -ss 3 to discard first 3 seconds of the video.

My final command looks like this:

ffmpeg -ss 3 -i input.mp4 -vf "select=gt(scene\,0.5)" -frames:v 5 -vsync vfr out%02d.jpg

This was the best I could do. I have noticed that since I pick only 5 videos , all of them are mostly from beginning of the video and may miss out on important scenes that occur later in the video

I would like to pick your brains for any other better options.

Answer

How about looking for, ideally, the first >40%-change frame within each of 5 time spans, where the time spans are the 1st, 2nd, 3rd, 4th, and 5th 20% of the video.

You could also split it into 6 time spans and disregard the 1st one to avoid credits.

In practice, this would mean setting the fps to a low number while applying your scene change check and your argument to throw out the first bit of the video.

...something like:

ffmpeg -ss 3 -i input.mp4 -vf "select=gt(scene\,0.4)" -frames:v 5 -vsync vfr -vf fps=fps=1/600 out%02d.jpg