Affectiva drops every second frame

Hello

I am running Affectiva SDK 4.0 on a GoPro video recording. I’m using a C++ program on Ubuntu 16.04. The GoPro video was recorded with 60 fps. The problem is that Affectiva only provides results for around half of the frames (i.e. 30 fps). If I look at the timestamps provided by Affectiva, the last timestamp matches the video duration, that means Affectiva somehow skips around every second frame.

Before running Affectiva I was running ffmpeg with the following command to make sure that the video has a constant frame rate of 60 fps:

ffmpeg -i in.MP4 -vf -y -vcodec libx264 -preset medium -r 60 -map_metadata 0:g -strict -2 out.MP4 </dev/null 2>&1

When I inspect the presentation timestamp using ffprobe -show_entries frame=pict_type,pkt_size -of csv in.MP4 I’m getting for the raw video the following values:

    Stream #0:0(eng): Video: h264 (Main) (avc1 / 0x31637661), yuvj420p(pc, bt709), 1280x720 [SAR 1:1 DAR 16:9], 14982 kb/s, 59.94 fps, 59.94 tbr, 60k tbn, 119.88 tbc (default)
    frame,341
    frame,345
    frame,338
    frame,341
    frame,341
    frame,342
    frame,341
    frame,354
    frame,72156,I
    frame,3620,P
    frame,349
    frame,7588,P
    frame,357
    frame,1440,P
    frame,311
    frame,7624,P
    frame,8080,P
    frame,336
    frame,3040,P
    frame,341
    frame,7196,P
    frame,341
    frame,125072,I
    frame,342
    frame,27068,P
    frame,26108,P
    frame,467
    frame,30892,P
    frame,289
    frame,29160,P
    frame,304
    frame,23848,P
    frame,24184,P
    frame,306
    frame,22028,P
    ...

When I do the same after applying ffmpeg I’m getting the following values:

    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc), 1280x720 [SAR 1:1 DAR 16:9], 392 kb/s, 60 fps, 60 tbr, 15360 tbn, 120 tbc (default)
    frame,25
    frame,516
    frame,1163
    frame,756
    frame,188
    frame,580
    frame,231
    frame,522
    frame,32131,I
    frame,247
    frame,507,B
    frame,783,B
    frame,501
    frame,306,B
    frame,271
    frame,3929,P
    frame,249
    frame,409,B
    frame,1112,B
    frame,519
    frame,535,B
    frame,213
    frame,5567,P
    frame,499
    frame,565,B
    frame,279
    frame,1205,B
    frame,602,B
    frame,519
    frame,5893,P
    frame,224
    frame,2967,P
    frame,520
    frame,3087,P
    ...

If I run Affectiva on the raw video (not processed by ffmpeg) I face the same problem of dropped frames. I was using Affectiva with affdex::VideoDetector detector(60);

Is there a problem with the ffmpeg command or with Affectiva?

Hi @raphael_fischer can you try to reproduce this with our sample app (https://github.com/Affectiva/cpp-sdk-samples). Check out the emotion-sdk branch and try to pass the video using the video-demo sample application. The application is parameterized to set the processing frame-rate so change that to 60 for your use case.

Also refer to this response: Video Processing frame per second results in half of expected result

Thanks!