Presenting videos while listening to audio

[FelicityF]

I've been running an experiment in which participants are presented with four static scenes, listen to a sentence, and select the scene they think matches the sentence they have just heard. For my next experiment I would like to present animations rather than images (so the scenes are now dynamic), but I don't get it to work and would really appreciate any help!

I created a .js file with all the stimuli. Here is an example of the first two trials:

var block1 = [
    {
        Foil1: "animations/Hep tibu lotop.mp4",
        Target: "animations/Hep simo lotop.mp4",
        correct_response: "animations/Hep simo lotop.mp4",
        audio: "audio/hep simo lotop.mp3",
        Trial: "1",
        button: "1",
        html: ['<video style = "width:350px; visibility: hidden;" src = "%choice%" />',
               '<video style = "width:350px;" src = "%choice%" />']
    },
    {
        Foil1: "animations/Pel yober brobo.mp4",
        Target: "animations/Lub tibu brobo.mp4",
        correct_response: "animations/Lub tibu brobo.mp4",
        audio: "audio/lub tibu brobo.mp3",
        Trial: "2",
        button: "1",
        html: ['<video style = "width:350px; visibility: hidden;" src = "%choice%" />',
               '<video style = "width:350px;" src = "%choice%" />']
    },`

This is the code for the trial itself (the shuffle is there because in the original experiment there were 3 foils):

var trial = {
type: 'audio-button-response',
stimulus: jsPsych.timelineVariable('audio'),  // Get the audio for the trial
        choices: function() {
            // Define foils (choices other than the target video)
            var foils = [jsPsych.timelineVariable('Foil1', true)];
            var foils_in_random_order = jsPsych.randomization.shuffle(foils);
            // Insert target in correct position (button index)
            foils_in_random_order.splice(jsPsych.timelineVariable('button', true), 0, jsPsych.timelineVariable('Target', true));
            return foils_in_random_order;
        },
        button_html: '<video style = "width:350px;" src = "%choice%" />',
        margin_horizontal: "80px",
        margin_vertical: "50px",
        data: {
            correct_response: jsPsych.timelineVariable('correct_response'),
            trial: jsPsych.timelineVariable('Trial'),
            button: jsPsych.timelineVariable('button')
        },
        on_finish: function(data) {
            // Check if the correct button was pressed
            data.button_pressed_content = latest_order[data.button_pressed];
            if (data.button_pressed == data.button) {
                data.correct = true;
            } else {
                data.correct = false;
            }
        }
    };`


    // Procedure Block 1
    var procedure_Block1 = {
        timeline: [fixation, trial],
        timeline_variables: block1, 
        randomize_order: true
    };

@jodeleeuw / @becky-gilbert do you have any idea why the experiment doesn't load? Is there something incorrect in this code? When I try to run it, I don't get any errors in the console. I also tried it with images rather than movies and that works. Thank you!

[jodeleeuw]

Hi @FelicityF !

I don't see anything obviously wrong with the code. I don't think you shared the bit of code that is attempting to play the video. Did I miss that?

Also, are you preloading audio/video? I don't rememeber off the top of my head what happens in version 6 when the media isn't preloaded, but that might cause a hiccup with the experiment running. It is odd that you aren't getting any messages in the console.

If you can send a link to the experiment that might provide more context for debugging.

[FelicityF]

Thanks for the reply, @jodeleeuw!

The buttons are the videos that are being played. So the 'target' and 'foil1' are the videos. When I run the code below with images instead of videos it does work (so 2 images are being shown in the screen, audio is played, and you click on the image that you think matches the interpretation of the audio). But when I change it to video, it's not working. So I'm not sure whether it's the .mp4 format or that perhaps there is something wrong with the html coding itself.

var trial = {
type: 'audio-button-response',
stimulus: jsPsych.timelineVariable('audio'),  // Get the audio for the trial
        choices: function() {
            // Define foils (choices other than the target video)
            var foils = [jsPsych.timelineVariable('Foil1', true)];
            var foils_in_random_order = jsPsych.randomization.shuffle(foils);
            // Insert target in correct position (button index)
            foils_in_random_order.splice(jsPsych.timelineVariable('button', true), 0, jsPsych.timelineVariable('Target', true));
            return foils_in_random_order;
        },
        button_html: '<video style = "width:350px;" src = "%choice%" />',
        margin_horizontal: "80px",
        margin_vertical: "50px",
        data: {
            correct_response: jsPsych.timelineVariable('correct_response'),
            trial: jsPsych.timelineVariable('Trial'),
            button: jsPsych.timelineVariable('button')
        },
        on_finish: function(data) {
            // Check if the correct button was pressed
            data.button_pressed_content = latest_order[data.button_pressed];
            if (data.button_pressed == data.button) {
                data.correct = true;
            } else {
                data.correct = false;
            }
        }
    };

Yes, I do preload the audio and the video.