VideoDetector vs FrameDetector output


#1

Hi,

How similar the outputs from VideoDetector vs FrameDetector supposed to be?
If we are charting the VideoDetector at 25fps, it produces clean lines, if charting the FrameDetector for the same content at same fps, we get huge fluctuations between two frames. 1/25sec frames following each other could differ 50-70 points which seems unreasonable.

We are evaluating the FrameDetector, as the VideoDetector was keep losing frames. Specified parameters for it to process the content at 25fps, but it was skipping/losing frames, resulting anywhere from 8 to 25 samples per second. We have switched to FrameDetector to be able to aggregate results across videos and getting the above issue now.


#2

In the frames where you are seeing discrepancies, are the frame timestamps the same?

There are some similarities and differences in the way that the VideoDetector and the FrameDetector discard frames.

They are similar in that they both discard frames whose inter-frame timestamps (relative to the prior un-discarded frame) are smaller than the duration implied by the processFPS constructor param (i.e. if this param is 30, then they will discard frames with timestamps < .0333 seconds after the prior un-discarded frame).

In addition, in an effort to produce results in real time, the FrameDetector will discard frames that are submitted at a rate faster than it is capable of processing (after factoring in some buffering, as configured by the bufferSize constructor param).

So, it’s possible that the FrameDetector and the VideoDetector are not processing exactly the same frames, and this would account for some differences in results.


#3

Can we ensure by some configuration that it processes all the 25 frames per second (as per parameter) of a video content? It is running on a quite powerful machine, we don’t mind if it takes long, but we need to aggregate somehow, so need to line up timestamps across multiple videos.

Example of VideoDetector processed timestamps at 25fps parameter:
4
4.0667
4.1333
4.2
4.2667
4.3333
4.4
4.4667
4.5333
4.6
4.6667
4.7333
4.8
4.8667
4.9333
5
5.0667
5.1333
5.2
5.2667
5.3333
5.4
5.4667
5.5333
5.6
5.6667
5.7333
5.8
5.8667
5.9333
6
6.0667
6.1333
6.2
6.2667
6.3333
6.4
6.4667
6.5333
6.6
6.6667
6.7333
6.8
6.8667
6.9333
7

Example of FrameDetector processed timestamps at 25fps parameter:
4
4.12
4.2
4.28
4.4
4.48
4.56
4.68
4.76
4.88
4.96
5.08
5.16
5.28
5.36
5.48
5.56
5.68
5.76
5.84
5.96
6.04
6.16
6.24
6.36
6.44
6.52
6.64
6.72
6.84
6.92
7.04


#4

To process all frames, use the VideoDetector and specify a processFPS value which is as high or higher than the input FPS. From the output above, it looks like your input video is 30 FPS (the timestamps are 0.0333 apart). If you are configuring the VideoDetector to process only 25 FPS, it will discard 5 frames per second (roughly… when you have timestamps like those, you can also get floating point precision issues, so I would suggest specifying 31 FPS to eliminate that possibility).


#5

oops, wait, I misread those timestamps… looks like they are .0666 apart. Is your source video 15 FPS?


#6

Source video is 30fps, was trying to sample it at 25fps.

Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf57.50.100
Duration: 00:01:00.07, start: 0.000000, bitrate: 1315 kb/s
Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 1:1 DAR 4:3], 1238 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc (default)
Metadata:
handler_name : VideoHandler
Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, mono, fltp, 69 kb/s (default)
Metadata:
handler_name : SoundHandler


#7

OK, so if you’re specifying 25 FPS as the requested processing rate, and the input is 30 FPS, only alternating input frames will get processed. This is because the input frame timestamps are .033 seconds apart, but the threshold for processing/discarding is .04 seconds apart (the inverse of 25 FPS).

input timestamp
0 (OK)
0.0333 (not >= .04, so discard)
0.0666 (OK)
0.1 (not >= .1066, so discard)
.1333 (OK)
etc.

So if you set your processing FPS to >30, you should see all frames processed. I recommend 31 to avoid floating point precision issues.


#8

Thany you, testing further and may get back on this.


#9

#10

Closing due to lack of activity.