Extending mouse-tracking to include mouseleave and nudges to move the mouse faster

[paulstillman]

I'm trying to add some functionality for the mouse-tracking extension, but running into a few issues. Briefly, in my experiment I will have participants selecting choices at either the top-left or top-right corners of the screen, and I'll measure the path that their mouse arrived at each choice. I have it mostly working using html-button-response in conjunction with the mouse-tracking extension, but I'd like to add two things.

First, the data get a bit messed up if participants leave the browswer window (since js can't record mouse-movements beyond the browser window, I believe). I'd like to record if a participant has left the window on that trial, and display a message encouraging them not to do that (ideally the message would be displayed as soon as the mouseleave event gets triggered).

From googling, it appears the best way to do this would be to use mouseleave, however when I try the following, it never seems to fire (tested across browsers). The following is the MouseTrackingExtension class with my code added (flagged using /* */ comments). I'v removed references to mouseup and mousedown to make it more concise:

class MouseTrackingExtension {
    constructor(jsPsych) {
        this.jsPsych = jsPsych;
        this.initialize = ({ minimum_sample_time = 0 }) => __awaiter(this, void 0, void 0, function* () {
            this.domObserver = new MutationObserver(this.mutationObserverCallback);
            this.minimumSampleTime = minimum_sample_time;
        });
        this.on_start = (params) => {
            params = params || {};
            this.currentTrialData = [];
            this.currentTrialTargets = new Map();
            this.currentTrialSelectors = params.targets || [];
            this.lastSampleTime = null;
            this.eventsToTrack = params.events || ["mousemove"];
            this.domObserver.observe(this.jsPsych.getDisplayElement(), { childList: true });
        };
        this.on_load = () => {
            // set current trial start time
            this.currentTrialStartTime = performance.now();
            // start data collection
            if (this.eventsToTrack.includes("mousemove")) {
                window.addEventListener("mousemove", this.mouseMoveEventHandler);
            }
            /* add listener for mouseleave */
            }
            if (this.eventsToTrack.includes("mouseleave")) {
                window.addEventListener("mouseleave", this.mouseLeaveEventHandler);
            }
        };
        this.on_finish = () => {
            this.domObserver.disconnect();
            if (this.eventsToTrack.includes("mousemove")) {
                window.removeEventListener("mousemove", this.mouseMoveEventHandler);
            }
            /* remove listener for mouseleave */
            if (this.eventsToTrack.includes("mouseleave")) {
                window.removeEventListener("mouseleave", this.mouseLeaveEventHandler);
            }
            return {
                mouse_tracking_data: this.currentTrialData,
                mouse_tracking_targets: Object.fromEntries(this.currentTrialTargets.entries()),
            };
        };
        this.mouseMoveEventHandler = ({ clientX: x, clientY: y }) => {
            const event_time = performance.now();
            const t = Math.round(event_time - this.currentTrialStartTime);
            if (this.lastSampleTime === null ||
                event_time - this.lastSampleTime >= this.minimumSampleTime) {
                this.lastSampleTime = event_time;
                this.currentTrialData.push({ x, y, t, event: "mousemove" });
            }
        }; 
        /* push the mouseleave event to the data object */
        this.mouseLeaveEventHandler = ({ clientX: x, clientY: y }) => {
            const event_time = performance.now();
            const t = Math.round(event_time - this.currentTrialStartTime);
            this.currentTrialData.push({ x, y, t, event: "mouseleave" });
        };

        this.mutationObserverCallback = (mutationsList, observer) => {
            for (const selector of this.currentTrialSelectors) {
                if (!this.currentTrialTargets.has(selector)) {
                    const target = this.jsPsych.getDisplayElement().querySelector(selector);
                    if (target) {
                        this.currentTrialTargets.set(selector, target.getBoundingClientRect());
                    }
                }
            }
        };
    }
}

It doesn't seem that mouseleave ever fires, however, and I'm not sure why that is.

My second question is related, but I would like to add more to the mouseMoveEventHandler or mouseLeaveEventHandler functions such that it makes a text element appear if some condition is met (e.g., if the first time participants move their mouse isn't for 500 ms, display a text that says "please start moving your mouse as soon as the options appear"; or displaying text that says 'please keep your cursor inside the browser window' if mouseleave is triggered). Is it possible to do that from within the mouse-tracking extension, or do I need to add similar event listeners to the trials? I know it would be easy to do it after each trial (e.g., by using the on_data_update event), but it would be ideal if I could display the message immediately since I think that will be more effective than displaying it after the trial.

Thanks so much!

[paulstillman]

Following up on this -- I wasn't ever able to get window.addEventListener("mouseleave", ...) to fire (same for document.addEventListener), but I did find that the following (somewhat inelegant) solution did work:

var disp_el = document.getElementsByClassName("jspsych-display-element")[0];
disp_el.addEventListener("mouseleave", mouseLeaveEventHandler);

If others have thoughts on better, less hacky ways to do that let me know.

For the second part, I adapted the html-button-response plugin to add in a few html elements corresponding to the messages I wanted to display, and set them to 'hidden'. Then I added some event listeners that would change their property to 'visible' if the conditions were met.

It's not the prettiest thing in the world but I'm pretty far on my way to having a working mouse-tracking plugin. Hoping in the next month or so to get it fully functional and add it to jspsych-contrib. If anyone is interested and/or has features they would like it to include please get in touch!