willxxy / awesome-mmps Public

Notifications You must be signed in to change notification settings
Fork 5
Star 114

Corpus of resources for multimodal machine learning with physiological signals (mmps).

114 stars 5 forks Branches Tags Activity

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 325 Commits
CITATION.cff		CITATION.cff
LICENSE		LICENSE
README.md		README.md

Repository files navigation

awesome-mmps

Corpus of resources for multimodal machine learning with physiological signals.

Any additions, corrections, or concerns please submit an issue. For additions to the list, please provide the relevant information. Thank you! :)

Table of Contents

Publications and Preprints
Datasets
Laboratories
Citation

Publications-and-Preprints

EEG 🧠 + X

Comparing Recognition Performance and Robustness of Multimodal Deep Learning Models for Multimodal Emotion Recognition, IEEE TCDS 2021; [EEG, Eye Movement, Peripheral Physiological Signals, ECG]
Multi-view Emotion Recognition Using Deep Canonical Correlation Analysis, ICONIP 2018; [EEG, Eye Movement]
Correlated Attention Networks for Multimodal Emotion Recognition, IEEE BIBM 2018; [EEG, Eye Movement]
Investigating Sex Differences in Classification of Five Emotions from EEG and Eye Movement Signals, IEEE EMBC 2019; [EEG, Eye Movement]
Can Brain Signals Reveal Inner Alignment with Human Languages?, EMNLP Findings 2023; [EEG, Text]
Open Vocabulary Electroencephalography-to-Text Decoding and Zero-Shot Sentiment Classification, AAAI 2022; [EEG, Text]
Decoding EEG Brain Activity for Multi-Modal Natural Language Processing, Frontiers in Human Neuroscience 2021; [EEG, Eye Movement, Text]
Advancing NLP with Cognitive Language Processing Signals, arxiv 2019; [EEG, Text, Gaze]
Recognition of emotions using multimodal physiological signals and an ensemble deep learning model, Computer Methods and Programs in Biomedicine 2017; [EEG, Peripheral Physiological Signals]
Multi-modal emotion analysis from facial expressions and electroencephalogram, Computer Vision and Image Understanding 2016; [EEG, Facial Expressions]
An Ensemble Learning Method for Emotion Charting Using Multimodal Physiological Signals, Sensors 2022; [EEG, ECG, GSR]
Emotion recognition based on multi-modal physiological signals and transfer learning, Frontiers in Neuroscience 2022; [EEG, RSP, GSR, Photo-Plethysmograph]
Multimodal Affective States Recognition Based on Multiscale CNNs and Biologically Inspired Decision Fusion Model, IEEE Transactions on Affective Computing 2023; [EEG, Peripheral Physiological Signals]
A Multimodal Approach to Estimating Vigilance Using EEG and Forehead EOG, Journal of Neural Engineering 2017; [EEG, EOG]
Multimodal Emotion Recognition Using Deep Neural Networks, ICONIP 2017; [EEG, Eye Movement]
EmotionMeter: A Multimodal Framework for Recognizing Human Emotions, IEEE Transactions on Cybernetics 2019; [EEG, Eye Movement]
A Deep Learning Architecture for Temporal Sleep Stage Classification Using Multivariate and Multimodal Time Series, IEEE Transactions on Neural Systems and Rehabilitation Engineering 2018; [EEG, EOG, EMG]
Multimodal Adaptive Emotion Transformer with Flexible Modality Inputs on A Novel Dataset with Continuous Labels, ACM 2023; [EEG, Eye Movement]
Emotion Transformer Fusion: Complementary Representation Properties of EEG and Eye Movements on Recognizing Anger and Surprise, IEEE BIBM 2023; [EEG, Eye Movement]
Multimodal Physiological Signals Fusion for Online Emotion Recognition, ACM Multimedia 2023; [EEG, ECG, GSR]
Graph to Grid: Learning Deep Representations for Multimodal Emotion Recognition, ACM Multimedia 2023; [EEG, Eye Movement, GSR, Respiration, ECG]
Stress Classification by Multimodal Physiological Signals Using Variational Mode Decomposition and Machine Learning, Hindawi Journal of Healthcare Engineering 2021; [ECG, EEG]
Language Mapping using tEEG and EEG Data with Convolutional Neural Networks, IEEE 2022; [EEG, tEEG]
Understanding language-elicited EEG data by predicting it from a fine-tuned language model, NAACL 2019; [EEG, ERP, Text]
Validation and Interpretation of a Multimodal Drowsiness Detection System Using Explainable Machine Learning, Computer Methods and Programs in Biomedicine 2023; [EEG, ECG, EOG]
A Classification Model for Sensing Human Trust in Machines Using EEG and GSR, ACM Transactions on Interactive Intelligent Systems 2018; [EEG, GSR]
Evaluating the Stressful Commutes Using Physiological Signals and Machine Learning Techniques, IEEE 3ICT 2022; [HR, BP, EEG]
Review of Machine and Deep Learning Techniques in Epileptic Seizure Detection using Physiological Signals and Sentiment Analysis, ACM 2022; [iEEG, EEG]
ECG and EEG based detection and multilevel classification of stress using machine learning for specified genders: A preliminary study, PLoS One 2023; [EEG, ECG]
Modeling Multivariate Biosignals With Graph Neural Networks and Structured State Space Models, CHIL 2023; [EEG, ECG, PSG]
Cognitive Load Prediction from Multimodal Physiological Signals using Multiview Learning., IEEE journal of biomedical and health informatics 2023; [EEG, EDA, ECG, EOG, & Eye Movements]
An Efficient Multimodal Emotion Identification Using FOX Optimized Double Deep Q-Learning, Wireless Personal Communications 2023; [GSR, ECG, and EEG]
EEG2TEXT: Open Vocabulary EEG-to-Text Decoding with EEG Pre-Training and Multi-View Transformer, arxiv 2024; [EEG, Text]
Are EEG-to-Text Models Working?, arxiv 2024; [EEG, Text]
Aligning Semantic in Brain and Language: A Curriculum Contrastive Method for Electroencephalography-to-Text Generation, IEEE Transactions on Neural Systems and Rehabilitation Engineering 2023, [EEG, Text]
A Multitask Framework for Emotion Recognition Using EEG and Eye Movement Signals with Adversarial Training and Attention Mechanism, IEEE BIBM 2023; [EEG, Eye Movement]
Temporal-Spatial Prediction: Pre-Training on Diverse Datasets for EEG Classification, ICASSP 2024; [EEG]
Joint Contrastive Learning with Feature Alignment for Cross-Corpus EEG-based Emotion Recognition, arxiv 2024; [EEG]
EEGFormer: Towards Transferable and Interpretable Large-Scale EEG Foundation Model, arxiv 2024; [EEG]
AC-CfC: An attention-based convolutional closed-form continuous-time neural network for raw multi-channel EEG-based emotion recognition, Biomedical Signal Processing and Control 2024; [EEG]
EEG-GPT: Exploring Capabilities of Large Language Models for EEG Classification and Interpretation, arxiv 2024; [EEG, Text]
Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI, ICLR 2024; [EEG, Text]
Naturalistic Emotion Recognition Using EEG and Eye Movements, Springer Nature Singapore 2023; [EEG, Eye Movement]
EmotionKD: A Cross-Modal Knowledge Distillation Framework for Emotion Recognition Based on Physiological Signals, ACM 2023; [EEG, GSR]
Negative emotion recognition using multimodal physiological signals for advanced driver assistance systems, Artificial Life and Robotics 2023; [ECG, EDA, EEG, and fNIRS]
Physiological Fusion Net: Quantifying Individual VR Sickness with Content Stimulus and Physiological Response, IEEE International Conference on Image Processing (ICIP) 2019; [Vision, EEG, ECG, GSR]
DeWave: Discrete EEG Waves Encoding for Brain Dynamics to Text Translation, Neurips 2023; [EEG, Text]
DreamDiffusion: Generating High-Quality Images from Brain EEG Signals, arxiv 2023; [EEG, Vision]
Multi-Signal Reconstruction Using Masked Autoencoder From EEG During Polysomnography, IEEE BCI 2023; [EEG, EOG, Chin EMG, Event Makers]
Multimodal Physiological Signal Emotion Recognition Based on Convolutional Recurrent Neural Network, IOP 2020; [EOG, EMG, GSR, RSP, BVP, TMP, EEG]
Comparative analysis of physiological signals and electroencephalogram (EEG) for multimodal emotion recognition using generative models, IEEE STSIVA 2014; [EEG, EOG, EMG, GSR, HR]
Large Language Models for Time Series: A Survey, arxiv 2024; [EEG, ECG, Text, Vision]
An Exploratory Study of Multimodal Physiological Data in Jazz Improvisation Using Basic Machine Learning Techniques, arxiv 2024; [EDA, EEG]
Multimodal Multi-View Spectral-Spatial-Temporal Masked Autoencoder for Self-Supervised Emotion Recognition, ICASSP 2024; [EEG, Eye Movement]
Functional Emotion Transformer for EEG-Assisted Cross-Modal Emotion Recognition, ICASSP 2024; [EEG, Eye Movement]
EEG-Transformer: Self-attention from transformer architecture for decoding EEG of imagined speech, IEEE BCI 2022; [EEG, Speech]
Towards Voice Reconstruction from EEG during Imagined Speech, AAAI 2023; [EEG, Speech]
Toward Fully-End-to-End Listened Speech Decoding from EEG Signals, InterSpeech 2024; [EEG, Speech]
Enhancing Affective Representations Of Music-Induced Eeg Through Multimodal Supervision And Latent Domain Adaptation, ICASSP 2022; [EEG, Audio]
Scaling Law in Neural Data: Non-Invasive Speech Decoding with 175 Hours of EEG Data, arxiv 2024; [EEG, Speech]
Hypercomplex Multimodal Emotion Recognition from EEG and Peripheral Physiological Signals, IEEE ICASSP Workshops 2023; [EEG, ECG, GSR]
EEG-Based Multimodal Emotion Recognition: A Machine Learning Perspective, IEEE Transactions on Instrumentation and Measurement 2024; [EEG, ECG, EMG]
SleepFM: Multi-modal Representation Learning for Sleep Across Brain Activity, ECG and Respiratory Signals, ICML 2024; [ECG, EEG, EOG, EMG, Respiratory Sensors]
A generative foundation model for five-class sleep staging with arbitrary sensor input, arxiv 2024; [EEG, EOG, EMG]
NeuroLM: A Universal Multi-task Foundation Model for Bridging the Gap between Language and EEG Signals, arxiv 2024; [EEG, Text]
Explainable Artificial Intelligence on Biosignals for Clinical Decision Support, KDD 2024; [EEG, ECG]
Recent Trends of Multimodal Affective Computing: A Survey from NLP Perspective, arxiv 2024; [EEG, ECG, Audio, Text, Image]
EEG-Language Modeling for Pathology Detection, arxiv 2024; [EEG, Text]
An ASR-based Hybrid Approach for Auditory Attention Decoding, Lund University Publications 2024; [EEG, Audio]
SEE: Semantically Aligned EEG-to-Text Translation, arxiv 2024; [EEG, Text]
Neural Speech Tracking in EEG: Integrating Acoustics and Linguistics for Hearing Aid Users, Lund University Publications 2024; [EEG, Audio]
LLM-enhanced Multi-teacher Knowledge Distillation for Modality-Incomplete Emotion Recognition in Daily Healthcare, IEEE Journal of Biomedical and Health Informatics 2024; [EEG, GSR]
Nested Deep Learning Model Towards A Foundation Model for Brain Signal Data, arxiv 2024; [EEG, MEG]
EEG Emotion Copilot: Pruning LLMs for Emotional EEG Interpretation with Assisted Medical Record Generation, arxiv 2024; [EEG, Text]
Thought2Text: Text Generation from EEG Signal using Large Language Models (LLMs), arxiv 2024; [EEG, Text]
EEGPT: Unleashing the Potential of EEG Generalist Foundation Model by Autoregressive Pre-training, OpenReview 2024; [EEG, Text]
ARIEL: Brain-Computer Interfaces meet Large Language Models for Emotional Support Conversation, Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization 2024; [EEG, Text]
Promoting cross-modal representations to improve multimodal foundation models for physiological signals, Medical Foundation Models (AIM-FM) workshop at NeurIPS 2024; [EEG, EMG, EOG, ECG]
DPD (DePression Detection) Net: a deep neural network for multimodal depression detection, Health Information Science and Systems 2024; [EEG, Speech]
Deep-learning models reveal how context and listener attention shape electrophysiological correlates of speech-to-language transformation, PLOS Computational Biology 2024; [EEG, Speech]
Learning under Label Noise through Few-Shot Human-in-the-Loop Refinement, arxiv 2024; [EEG, IMU, EMG, EOG, ECG]
Position: Addressing Ethical Challenges and Safety Risks in GenAI-Powered Brain-Computer Interfaces, GenAI for Health Workshop @ NeurIPS 2024; [EEG, Text]
A Hybrid Artificial Intelligence System for Automated EEG Background Analysis and Report Generation, IEEE Journal of Biomedical and Health Informatics 2024; [EEG, Text]
PedSleepMAE: Generative Model for Multimodal Pediatric Sleep Signals, IEEE International Conference on Biomedical and Health Informatics 2024; [EEG, EMG, EOG]
Decoding text from electroencephalography signals: A novel Hierarchical Gated Recurrent Unit with Masked Residual Attention Mechanism, Engineering Applications of Artificial Intelligence 2024; [EEG, Text]
Towards Neural Foundation Models for Vision: Aligning EEG, MEG, and fMRI Representations for Decoding, Encoding, and Modality Conversion, arxiv 2024; [EEG, fMRI, MEG, Image]
A Survey of LLMs on Biosignal Applications, arxiv 2024; [ECG, PPG, EEG, IMU, Text]
A novel multimodal EEG-image fusion approach for emotion recognition: introducing a multimodal KMED dataset, Neural Computing and Applications 2024; [EEG, Image, Video]
Representation of Articulatory Features in EEG During Speech Production Tasks, International Symposium on Chinese Spoken Language Processing 2024; [EEG, Speech]
[Decoding individual words from non-invasive brain recordings across 723 participants], arxiv 2024; [EEG, MEG, Text]
From EEG to Eye Movements: Cross-modal Emotion Recognition Using Constrained Adversarial Network with Dual Attention, IEEE Transactions on Affective Computing 2024; [EEG, Eye Movement]
Brain-model neural similarity reveals abstractive summarization performance, Nature Scientific Reports 2025; [EEG, Text]
Develop an emotion recognition system using jointly connectivity between electroencephalogram and electrocardiogram signals, Heliyon CellPress 2025; [ECG, EEG]
On Creating A Brain-To-Text Decoder, arxiv 2025; [EEG, Text]
Cross-modality fusion with EEG and text for enhanced emotion detection in English writing, Frontiers in Neurorobot 2025; [EEG, Text]
Decoding Neural Signatures of Language Comprehension and Production through EEG based Brain-Computer Interfaces, preprint 2025; [EEG, Text]
Neural Spelling: A Spell-Based BCI System for Language Neural Decoding, arxiv 2025; [EEG, Text]
Multi-branch convolutional neural network with cross-attention mechanism for emotion recognition, Nature Scientific Reports 2025; [EEG, Text]
A Multimodal Sleep Foundation Model Developed with 500K Hours of Sleep Recordings for Disease Predictions, medRxiv 2025; [EEG, EOG, EMG, ECG, Respiratory]
Multimodal Physiological Signals Representation Learning via Multiscale Contrasting for Depression Recognition, Proceedings of the 32nd ACM International Conference on Multimedia 2024; [EEG, fNIRS]
Finetuning and Quantization of EEG-Based Foundational BioSignal Models on ECG and PPG Data for Blood Pressure Estimation, arxiv 2025; [ECG, PPG, EEG]
Brain-to-Text Decoding: A Non-invasive Approach via Typing, arxiv 2025; [EEG, MEG, Text]
Bridging Brain Signals and Language: A Deep Learning Approach to EEG-to-Text Decoding, arxiv 2025; [EEG, Text]
Data Foundations for Large Scale Multimodal Clinical Foundation Models, arxiv 2025; [EEG, ECG, Text, Image, Video]
Integrated Word2Vec-Based Speech Annotations for Enhanced EEG Decoding of Speech Intentions, IEEE International Winter Workshop on Brain-Computer Interface (BCI) 2025; [EEG, Text]
Towards Robust Multimodal Physiological Foundation Models: Handling Arbitrary Missing Modalities, arxiv 2025; [EEG, ECG, EMG, EOG]
Robust Sleep Staging over Incomplete Multimodal Physiological Signals via Contrastive Imagination, NeurIPS 2024; [EEG, EMG, EOG]
Pretraining Large Brain Language Model for Active BCI: Silent Speech, arxiv 2025; [EEG, Text]
EEG-to-Text Translation: A Model for Deciphering Human Brain Activity, arxiv 2025; [EEG, Text]
Learning Interpretable Representations Leads to Semantically Faithful EEG-to-Text Generation, arxiv 2025; [EEG, Text]
Recognizing Natural Images From EEG With Language-Guided Contrastive Learning, IEEE Transactions on Neural Networks and Learning Systems 2025; [EEG, Text, Image]
Neural-MCRL: Neural Multimodal Contrastive Representation Learning for EEG-based Visual Decoding, arxiv 2025; [EEG, Image, Text]
BrainOmni: A Brain Foundation Model for Unified EEG and MEG Signals, arxiv 2025; [EEG, MEG]
EEG2TEXT-CN: An Exploratory Study of Open-Vocabulary Chinese Text-EEG Alignment via Large Language Model and Contrastive Learning on ChineseEEG, arxiv 2025; [EEG, Text]
A Pre-trained Framework for Multilingual Brain Decoding Using Non-invasive Recordings, arxiv 2025; [EEG, Text]
Bridging Brain with Foundation Models through Self-Supervised Learning, arxiv 2025; [EEG, MEG, fMRI, fNIRS, ECoG, Intracortical]
Efficient Personalized Adaptation for Physiological Signal Foundation Model, ICML 2025; [EEG, EOG, EMG, ECG]
EEG-Language Pretraining for Highly Label-Efficient Clinical Phenotyping, ICML 2025; [EEG, Text]
UniMind: Unleashing the Power of LLMs for Unified Multi-Task Brain Decoding, arxiv 2025; [EEG, Text]
Exploring the Potential of Electroencephalography Signal–Based Image Generation Using Diffusion Models: Integrative Framework Combining Mixed Methods and Multimodal Analysis, JMIR 2025; [EEG, Image]
Interpretable EEG-to-Image Generation with Semantic Prompts, Actionable Interpretability Workshop (non-archival) at the 42nd ICML; [EEG, Image, Text]
BrainFLORA: Uncovering Brain Concept Representation via Multimodal Neural Embeddings, ACM Multimedia 2025; [Image, EEG, fMRI, MEG]

ECG 🫀 + X

Comparing Recognition Performance and Robustness of Multimodal Deep Learning Models for Multimodal Emotion Recognition, IEEE TCDS 2021; [EEG, Eye Movement, Peripheral Physiological Signals, ECG]
Transfer Knowledge from Natural Language to Electrocardiography: Can We Detect Cardiovascular Disease Through Language Models?, EACL Findings 2023; [ECG, Text]
Converting ECG Signals to Images for Efficient Image-text Retrieval via Encoding, Machine Learning for Health 2023; [ECG, Image, Text]
An Ensemble Learning Method for Emotion Charting Using Multimodal Physiological Signals, Sensors 2022; [EEG, ECG, GSR]
Automated detection of panic disorder based on multimodal physiological signals using machine learning, ETRI Journal 2022; [ECG, EDA, Respiration, Peripheral Temperature]
ECGBERT: Understanding Hidden Language of ECGs with Self-Supervised Representation Learning, arxiv 2023; [ECG, Text]
ECG Language Processing (ELP): A new technique to analyze ECG signals, Computer Methods and Programs in Biomedicine; [ECG, Text]
Robust Patient Information Embedding and Retrieval Mechanism for ECG Signals, IEEE Access 2020; [ECG, Text]
Multimodal Physiological Signals and Machine Learning for Stress Detection by Wearable Devices, IEEE MEMEA 2022; [EDA, ECG, PPG]
Multimodal Physiological Signals Fusion for Online Emotion Recognition, ACM Multimedia 2023; [EEG, ECG, GSR]
Graph to Grid: Learning Deep Representations for Multimodal Emotion Recognition, ACM Multimedia 2023; [EEG, Eye Movement, GSR, Respiration, ECG]
Automated detection of panic disorder based on multimodal physiological signals using machine learning, ETRI Journal 2023; [ECG, EDA, RESP, PT]
Stress Classification by Multimodal Physiological Signals Using Variational Mode Decomposition and Machine Learning, Hindawi Journal of Healthcare Engineering 2021; [ECG, EEG]
Validation and Interpretation of a Multimodal Drowsiness Detection System Using Explainable Machine Learning, Computer Methods and Programs in Biomedicine 2023; [EEG, ECG, EOG]
A Novel Algorithm for Movement Artifact Removal in ECG Signals Acquired from Wearable Systems Applied to Horses, PLoS One 2015; [ECG, Accelerometer]
Automatic stress detection in car drivers based on non-invasive physiological signals using machine learning techniques, Neural Computing and Applications 2023; [ECG, EMG, GSR, Respiration Rate]
Hierarchical extreme puzzle learning machine-based emotion recognition using multimodal physiological signals, Biomedical Signal Processing and Control 2023; [EDA, ECG, TEMP, RESP, EMR]
Machine Learning–Enabled Multimodal Fusion of Intra-Atrial and Body Surface Signals in Prediction of Atrial Fibrillation Ablation Outcomes, Journal of American Heart Association 2022; [ECG, iECG]
ECG and EEG based detection and multilevel classification of stress using machine learning for specified genders: A preliminary study, PLoS One 2023; [EEG, ECG]
Modeling Multivariate Biosignals With Graph Neural Networks and Structured State Space Models, CHIL 2023; [EEG, ECG, PSG]
Cognitive Load Prediction from Multimodal Physiological Signals using Multiview Learning., IEEE journal of biomedical and health informatics 2023; [EEG, EDA, ECG, EOG, & Eye Movements]
ECG Semantic Integrator (ESI): A Foundation ECG Model Pretrained with LLM-Enhanced Cardiological Text, arxiv 2024; [ECG, Text]
MEIT: Multi-Modal Electrocardiogram Instruction Tuning on Large Language Models for Report Generation, arxiv 2024; [ECG, Text]
An Efficient Multimodal Emotion Identification Using FOX Optimized Double Deep Q-Learning, Wireless Personal Communications 2023; [GSR, ECG, and EEG]
A Framework for Cognitive Load Recognition Based on Machine Learning and Multimodal Physiological Signals by Wearable Sensors, IEEE PRML 2023; [ECG, EDA, and PPG]
A Household Multimodal Physiological Signals Monitoring System, IEEE 2023; [ECG, lLEMG, rLEMG, pBCG]
ECG-QA: A Comprehensive Question Answering Dataset Combined With Electrocardiogram, NeurIPS 2023; [ECG, Text]
Multimodal 12-lead ECG data classification using multi-label DenseNet for heart disease detection, IEEE DSIT 2022; [HRV, ECG, EHR]
Time Synchronization of Multimodal Physiological Signals through Alignment of Common Signal Types and Its Technical Considerations in Digital Health, J Imaging 2022; [ECG]
Single-modal and multi-modal false arrhythmia alarm reduction using attention-based convolutional and recurrent neural networks, PLoS One 2024; [PPG, ABP, ECG]
Negative emotion recognition using multimodal physiological signals for advanced driver assistance systems, Artificial Life and Robotics 2023; [ECG, EDA, EEG, and fNIRS]
Position Paper: What Can Large Language Models Tell Us about Time Series Analysis, arxiv 2024, [ECG, Text]
Large Language Model-informed ECG Dual Attention Network for Heart Failure Risk Prediction, arxiv 2024; [ECG, Text]
Frozen Language Model Helps ECG Zero-Shot Learning, MIDL 2023; [ECG, Text]
Zero-Shot ECG Diagnosis with Large Language Models and Retrieval-Augmented Generation, ML4H 2023; [ECG, Text]
Physiological Fusion Net: Quantifying Individual VR Sickness with Content Stimulus and Physiological Response, IEEE International Conference on Image Processing (ICIP) 2019; [Vision, EEG, ECG, GSR]
Text-to-ECG: 12-Lead Electrocardiogram Synthesis conditioned on Clinical Text Reports, ICASSP 2023; [ECG, Text]
Large Language Models for Time Series: A Survey, arxiv 2024; [EEG, ECG, Text, Vision]
Integrating multimodal information in machine learning for classifying acute myocardial infarction, Physiological Measurement 2023; [ECG, Text]
Multimodal explainable artificial intelligence identifies patients with non-ischaemic cardiomyopathy at risk of lethal ventricular arrhythmias, Scientific reports 2024; [MRI, ECG]
Multimodal Estimation Of Change Points Of Physiological Arousal During Driving, ICASSP 2023; [ECG, EDA, EMG, HR, BVP]
Hypercomplex Multimodal Emotion Recognition from EEG and Peripheral Physiological Signals, IEEE ICASSP Workshops 2023; [EEG, ECG, GSR]
Deep learning for ECG classification: A comparative study of 1D and 2D representations and multimodal fusion approaches, Biomedical Signal Processing and Control 2024; [ECG, Image]
Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge Enhancement, ICML 2024; [ECG ,Text]
SleepFM: Multi-modal Representation Learning for Sleep Across Brain Activity, ECG and Respiratory Signals, ICML 2024; [ECG, EEG, EOG, EMG, Respiratory Sensors]
ECG Heartbeat Classification Using Multimodal Fusion, IEEE Access 2021; [ECG, Image]
Multimodal ChatGPT-4V for Electrocardiogram Interpretation: Promise and Limitations, JMIR 2024; [ECG, Image]
Multimodal ECG heartbeat classification method based on a convolutional neural network embedded with FCA, Nature Scientific Reports 2024; [ECG, Image]
Large-scale Training of Foundation Models for Wearable Biosignals, ICLR 2024; [PPG, ECG]
ECG-Chat: A Large ECG-Language Model for Cardiac Disease Diagnosis, arxiv 2024; [ECG, Text]
CardioGPT: An ECG Interpretation Generation Model, IEEE Access 2024; [ECG, Text]
Harnessing the Power of ChatGPT in Cardiovascular Medicine: Innovations, Challenges, and Future Directions, preprint 2024; [ECG, Text]
The potential for large language models to transform cardiovascular medicine, The Lancet Digital Health 2024; [ECG, Text]
Explainable Artificial Intelligence on Biosignals for Clinical Decision Support, KDD 2024; [EEG, ECG]
Recent Trends of Multimodal Affective Computing: A Survey from NLP Perspective, arxiv 2024; [EEG, ECG, Audio, Text, Image]
Electrocardiogram Report Generation and Question Answering via Retrieval-Augmented Self-Supervised Modeling, arxiv 2024; [Text, ECG]
Foundation Models for Cardiovascular Disease Detection via BioSignals from Digital Stethoscopes, Research Square 2024; [PCG, ECG]
From Hospital to Portables: A Universal ECG Foundation Model Built on 10+ Million Diverse Recordings, arxiv 2024; [Text, ECG]
Electrocardiogram–Language Model for Few-Shot Question Answering with Meta Learning, arxiv 2024; [Text, ECG]
Promoting cross-modal representations to improve multimodal foundation models for physiological signals, Medical Foundation Models (AIM-FM) workshop at NeurIPS 2024; [EEG, EMG, EOG, ECG]
Foundation Models in Electrocardiogram: A Review, arxiv 2024; [ECG, Text]
C-MELT: Contrastive Enhanced Masked Auto-Encoders for ECG-Language Pre-Training, arxiv 2024; [ECG, Text]
Let Your Heart Speak in its Mother Tongue: Multilingual Captioning of Cardiac Signals, arxiv 2021, [ECG, Text]
Learning under Label Noise through Few-Shot Human-in-the-Loop Refinement, arxiv 2024; [EEG, IMU, EMG, EOG, ECG]
Precision of artificial intelligence in paediatric cardiology multimodal image interpretation, Cambridge University Press 2024; [ECG, Text]
A Survey of LLMs on Biosignal Applications, arxiv 2024; [ECG, PPG, EEG, IMU, Text]
HeartBERT: A Self-Supervised ECG Embedding Model for Efficient and Effective Medical Signal Analysis, arxiv 2024; [ECG, Text]
De-biased Multimodal Electrocardiogram Analysis, arxiv 2024; [ECG, Text]
Cross-Modality Cardiac Insight Transfer: A Contrastive Learning Approach to Enrich ECG with CMR Features, MICCAI 2024; [ECG, CMR]
Large-scale cross-modality pretrained model enhances cardiovascular state estimation and cardiomyopathy detection from electrocardiograms: An AI system development and multi-center validation study, arxiv 2024; [ECG, CMR]
Automated Medical Report Generation for ECG Data: Bridging Medical Text and Signal Processing with Deep Learning, arxiv 2024; [ECG, Text]
ECHOPulse: ECG controlled echocardio-grams video generation, arxiv 2024; [ECG, ECHO, Video]
Application of Contrastive Learning on ECG Data: Evaluating Performance in Japanese and Classification with Around 100 Labels, arxiv 2024; [ECG, Text]
Foundation model of ECG diagnosis: Diagnostics and explanations of any form and rhythm on ECG, Cell Reports Medicine 2024; [ECG, Text]
ECG-LM: Understanding Electrocardiogram with Large Language Model, SPJ 2024; [ECG, Text]
MedTsLLM: Leveraging LLMs for Multimodal Medical Time Series Analysis, Machine Learning for Healthcare 2024; [ECG, Text]
Develop an emotion recognition system using jointly connectivity between electroencephalogram and electrocardiogram signals, Heliyon CellPress 2025; [ECG, EEG]
Teach Multimodal LLMs to Comprehend Electrocardiographic Images, arxiv 2024; [ECG, Text, Image]
DiffuSETS: 12-lead ECG Generation Conditioned on Clinical Text Reports and Patient-Specific Information, arxiv 2025; [ECG, Text]
Automated Diagnostic Reports from Images of Electrocardiograms at the Point-of-Care, medRxiv 2024; [ECG, Image, Text]
Multi-Modal Contrastive Learning across Cardiac Diagnostics, preprint 2024; [Video, ECG, Image, Text]
Enhancing Visual Inspection Capability of Multi-Modal Large Language Models on Medical Time Series with Supportive Conformalized and Interpretable Small Specialized Models, arxiv 2025; [ECG, Image, Text]
Improving Atrial Fibrillation Detection Using a Shared Latent Space for ECG and PPG Signals, Harvard Data Science Review 2025; [ECG, PPG]
High-Accuracy ECG Image Interpretation using Parameter-Efficient LoRA Fine-Tuning with Multimodal LLaMA 3.2, arxiv 2025; [ECG, Text]
A Multimodal Sleep Foundation Model Developed with 500K Hours of Sleep Recordings for Disease Predictions, medRxiv 2025; [EEG, EOG, EMG, ECG, Respiratory]
Interpretable Pre-Trained Transformers for Heart Time-Series Data, arxiv 2024; [ECG, PPG]
CardioGAN: Attentive Generative Adversarial Network with Dual Discriminators for Synthesis of ECG from PPG, AAAI 2021; [ECG, PPG]
Signal-quality-aware multisensor fusion for atrial fibrillation detection, Healthcare Technology Letters 2025; [ECG, PPG]
Finetuning and Quantization of EEG-Based Foundational BioSignal Models on ECG and PPG Data for Blood Pressure Estimation, arxiv 2025; [ECG, PPG, EEG]
IHILLM-RAG: a safe and private medical large language model based on intelligent hardware interaction and retrieval-augmented generation (RAG), SPIE Digital Library 2025; [ECG, PPG, Text]
SuPreME: A Supervised Pre-training Framework for Multimodal ECG Representation Learning, arxiv 2025; [ECG, Text]
TELL ME: Tackle Electrocardiogram with Large Language Model Effectively, ICASSP 2025; [ECG, Text]
GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images, arxiv 2025; [ECG, Image, Text]
ECG Representation Learning with Multi-Modal EHR Data, TMLR 2023; [ECG, Text, sEHR]
BioSignal Copilot: Leveraging the power of LLMs in drafting reports for biomedical signals, medRxiv 2023; [ECG, Text]
Data Foundations for Large Scale Multimodal Clinical Foundation Models, arxiv 2025; [EEG, ECG, Text, Image, Video]
Towards Robust Multimodal Physiological Foundation Models: Handling Arbitrary Missing Modalities, arxiv 2025; [EEG, ECG, EMG, EOG]
PainFormer: a Vision Foundation Model for Automatic Pain Assessment, arxiv 2025; [ECG, Image, EMG, GSR, fNIRS]
Q-HEART: ECG Question Answering via Knowledge-Informed Multimodal LLMs, arxxiv 2025; [ECG, Text]
Fine-Grained ECG-Text Contrastive Learning via Waveform Understanding Enhancement, arxiv 2025; [ECG, Text]
Signal, Image, or Symbolic: Exploring the Best Input Representation for Electrocardiogram-Language Models Through a Unified Framework, arxiv 2025; [ECG, Text, Image]
anyECG-chat: A Generalist ECG-MLLM for Flexible ECG Input and Multi-Task Understanding, arxiv 2025; [ECG, Text]
Heartcare Suite: Multi-dimensional Understanding of ECG with Raw Multi-lead Signal Modeling, arxiv 2025; [ECG, Text]
TimeMaster: Training Time-Series Multimodal LLMs to Reason via Reinforcement Learning, arxiv 2025; [ECG, EMG, Text]
ECGFM: A foundation model for ECG analysis trained on a multi-center million-ECG dataset, Informatic Fusion 2025; [ECG, Text]
Efficient Personalized Adaptation for Physiological Signal Foundation Model, ICML 2025; [EEG, EOG, EMG, ECG]
ECG-Byte: A Tokenizer for End-to-End Generative Electrocardiogram Language Modeling, MLHC 2025; [ECG, Text]
Sensing Cardiac Health Across Scenarios and Devices: A Multi-Modal Foundation Model Pretrained on Heterogeneous Data from 1.7 Million Individuals, arxiv 2025; [ECG, PPG, Text]
From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining, ICML 2025; [ECG, Text]
TAC-ECG: A task-adaptive classification method for electrocardiogram based on cross-modal contrastive learning and low-rank convolutional adapter, Computer Methods and Programs in Biomedicine 2025; [ECG, Text]
ECGTwin: Personalized ECG Generation Using Controllable Diffusion Model, arxiv 2025; [ECG, Text]
Deep ECG-Report Interaction Framework for Cross-Modal Representation Learning, arxiv 2025; [ECG, Text]
DiagECG: An LLM-Driven Framework for Diagnostic Reasoning via Discretized ECG Tokenization, arxiv 2025; [ECG, Text]
MedTVT-R1: A Multimodal LLM Empowering Medical Reasoning and Diagnosis, arxiv 2025; [ECG, Image, Text]
TolerantECG: A Foundation Model for Imperfect Electrocardiogram, ACM Multimedia 2025; [ECG, Text]

EDA 💦 + X

A Multimodal Music Recommendation System with Listeners' Personality and Physiological Signals, ACM/IEEE JCDL 2020; [Heart Rate, EDA, IBI, Skin Temperature, BVP]
Stress Detection with Deep Learning Approaches Using Physiological Signals, IoT Technologies for Healthcare 2020; [EDA, BVP]
Multimodal Physiological Signals and Machine Learning for Stress Detection by Wearable Devices, IEEE MEMEA 2022; [EDA, ECG, PPG]
Automated detection of panic disorder based on multimodal physiological signals using machine learning, ETRI Journal 2023; [ECG, EDA, RESP, PT]
Transformer-Based Physiological Feature Learning for Multimodal Analysis of Self-Reported Sentiment, ICMI 2022; [EDA, BVP, HR, TEMP]
Hierarchical extreme puzzle learning machine-based emotion recognition using multimodal physiological signals, Biomedical Signal Processing and Control 2023; [EDA, ECG, TEMP, RESP, EMR]
A Hybrid Deep Learning Emotion Classification System Using Multimodal Data, Sensors 2023; [EDA, IBI, TEMP, Text, Audio]
Exploring the Potential of Multimodal Emotion Recognition for Hearing-Impaired Children Using Physiological Signals and Facial Expressions, ICMI Companion 2023; [ST, EDA, BVP, ACC, HR, Vision]
Performance Exploration of RNN Variants for Recognizing Daily Life Stress Levels by Using Multimodal Physiological Signals, ICMI 2023; [EDA, ST, ACC, HR]
Transformer-based Self-supervised Multimodal Representation Learning for Wearable Emotion Recognition, IEEE Transactions On Affective Computing 2023; [EDA, BVP, TEMP]
Cognitive Load Prediction from Multimodal Physiological Signals using Multiview Learning., IEEE journal of biomedical and health informatics 2023; [EEG, EDA, ECG, EOG, & Eye Movements]
A Framework for Cognitive Load Recognition Based on Machine Learning and Multimodal Physiological Signals by Wearable Sensors, IEEE PRML 2023; [ECG, EDA, and PPG]
Stress appraisal in the workplace and its associations with productivity and mood: Insights from a multimodal machine learning analysis, PLoS One 2024; [EDA, ST, BVP, Accelerometer]
Effects of Physiological Signals in Different Types of Multimodal Sentiment Estimation, IEEE AC 2023; [Language, EDA, Audio, Vision]
Negative emotion recognition using multimodal physiological signals for advanced driver assistance systems, Artificial Life and Robotics 2023; [ECG, EDA, EEG, and fNIRS]
An Exploratory Study of Multimodal Physiological Data in Jazz Improvisation Using Basic Machine Learning Techniques, arxiv 2024; [EDA, EEG]
Multimodal Estimation Of Change Points Of Physiological Arousal During Driving, ICASSP 2023; [ECG, EDA, EMG, HR, BVP]

Eye Movement 👁️ + X

Investigating Sex Differences in Classification of Five Emotions from EEG and Eye Movement Signals, IEEE EMBC 2019; [EEG, Eye Movement]
Decoding EEG Brain Activity for Multi-Modal Natural Language Processing, Frontiers in Human Neuroscience 2021; [EEG, Eye Movement, Text]
Multi-view Emotion Recognition Using Deep Canonical Correlation Analysis, ICONIP 2018; [EEG, Eye Movement]
Correlated Attention Networks for Multimodal Emotion Recognition, IEEE BIBM 2018; [EEG, Eye Movement]
Multimodal Emotion Recognition Using Deep Neural Networks, ICONIP 2017; [EEG, Eye Movement]
EmotionMeter: A Multimodal Framework for Recognizing Human Emotions, IEEE Transactions on Cybernetics 2019; [EEG, Eye Movement]
Multimodal Adaptive Emotion Transformer with Flexible Modality Inputs on A Novel Dataset with Continuous Labels, ACM 2023; [EEG, Eye Movement]
Emotion Transformer Fusion: Complementary Representation Properties of EEG and Eye Movements on Recognizing Anger and Surprise, IEEE BIBM 2023; [EEG, Eye Movement]
Graph to Grid: Learning Deep Representations for Multimodal Emotion Recognition, ACM Multimedia 2023; [EEG, Eye Movement, GSR, Respiration, ECG]
Cognitive Load Prediction from Multimodal Physiological Signals using Multiview Learning., IEEE journal of biomedical and health informatics 2023; [EEG, EDA, ECG, EOG, & Eye Movements]
Navigating Brain Language Representations: A Comparative Analysis of Neural Language Models and Psychologically Plausible Models, arxiv 2024; [fMRI, Eye Movement]
A Multitask Framework for Emotion Recognition Using EEG and Eye Movement Signals with Adversarial Training and Attention Mechanism, IEEE BIBM 2023; [EEG, Eye Movement]
Naturalistic Emotion Recognition Using EEG and Eye Movements, Springer Nature Singapore 2023; [EEG, Eye Movement]
Multimodal Multi-View Spectral-Spatial-Temporal Masked Autoencoder for Self-Supervised Emotion Recognition, ICASSP 2024; [EEG, Eye Movement]
Functional Emotion Transformer for EEG-Assisted Cross-Modal Emotion Recognition, ICASSP 2024; [EEG, Eye Movement]
EEG2Video: Towards Decoding Dynamic Visual Perception from EEG Signals, NeurIPS 2024; [EEG, fMRI, Video, Text]
From EEG to Eye Movements: Cross-modal Emotion Recognition Using Constrained Adversarial Network with Dual Attention, IEEE Transactions on Affective Computing 2024; [EEG, Eye Movement]

EOG 👁️ + X

Machine-Learning-Based Detection of Craving for Gaming Using Multimodal Physiological Signals: Validation of Test-Retest Reliability for Practical Use, Sensors 2019; [PPG, GSR, EOG]
A Multimodal Approach to Estimating Vigilance Using EEG and Forehead EOG, Journal of Neural Engineering 2017; [EEG, EOG]
A Deep Learning Architecture for Temporal Sleep Stage Classification Using Multivariate and Multimodal Time Series, IEEE Transactions on Neural Systems and Rehabilitation Engineering 2018; [EEG, EOG, EMG]
Validation and Interpretation of a Multimodal Drowsiness Detection System Using Explainable Machine Learning, Computer Methods and Programs in Biomedicine 2023; [EEG, ECG, EOG]
Cognitive Load Prediction from Multimodal Physiological Signals using Multiview Learning., IEEE journal of biomedical and health informatics 2023; [EEG, EDA, ECG, EOG, & Eye Movements]
Multi-Signal Reconstruction Using Masked Autoencoder From EEG During Polysomnography, IEEE BCI 2023; [EEG, EOG, Chin EMG, Event Makers]
Multimodal Physiological Signal Emotion Recognition Based on Convolutional Recurrent Neural Network, IOP 2020; [EOG, EMG, GSR, RSP, BVP, TMP, EEG]
Comparative analysis of physiological signals and electroencephalogram (EEG) for multimodal emotion recognition using generative models, IEEE STSIVA 2014; [EEG, EOG, EMG, GSR, HR]
SleepFM: Multi-modal Representation Learning for Sleep Across Brain Activity, ECG and Respiratory Signals, ICML 2024; [ECG, EEG, EOG, EMG, Respiratory Sensors]
A generative foundation model for five-class sleep staging with arbitrary sensor input, arxiv 2024; [EEG, EOG, EMG]
Promoting cross-modal representations to improve multimodal foundation models for physiological signals, Medical Foundation Models (AIM-FM) workshop at NeurIPS 2024; [EEG, EMG, EOG, ECG]
Learning under Label Noise through Few-Shot Human-in-the-Loop Refinement, arxiv 2024; [EEG, IMU, EMG, EOG, ECG]
PedSleepMAE: Generative Model for Multimodal Pediatric Sleep Signals, IEEE International Conference on Biomedical and Health Informatics 2024 2024; [EEG, EMG, EOG]
A Multimodal Sleep Foundation Model Developed with 500K Hours of Sleep Recordings for Disease Predictions, medRxiv 2025; [EEG, EOG, EMG, ECG, Respiratory]
Towards Robust Multimodal Physiological Foundation Models: Handling Arbitrary Missing Modalities, arxiv 2025; [EEG, ECG, EMG, EOG]
Robust Sleep Staging over Incomplete Multimodal Physiological Signals via Contrastive Imagination, NeurIPS 2024; [EEG, EMG, EOG]
Efficient Personalized Adaptation for Physiological Signal Foundation Model, ICML 2025; [EEG, EOG, EMG, ECG]

EMG 💪 + X

A Deep Learning Architecture for Temporal Sleep Stage Classification Using Multivariate and Multimodal Time Series, IEEE Transactions on Neural Systems and Rehabilitation Engineering 2018; [EEG, EOG, EMG]
Automatic stress detection in car drivers based on non-invasive physiological signals using machine learning techniques, Neural Computing and Applications 2023; [ECG, EMG, GSR, Respiration Rate]
Multi-Signal Reconstruction Using Masked Autoencoder From EEG During Polysomnography, IEEE BCI 2023; [EEG, EOG, Chin EMG, Event Makers]
Multimodal Physiological Signal Emotion Recognition Based on Convolutional Recurrent Neural Network, IOP 2020; [EOG, EMG, GSR, RSP, BVP, TMP, EEG]
Comparative analysis of physiological signals and electroencephalogram (EEG) for multimodal emotion recognition using generative models, IEEE STSIVA 2014; [EEG, EOG, EMG, GSR, HR]
Multimodal Estimation Of Change Points Of Physiological Arousal During Driving, ICASSP 2023; [ECG, EDA, EMG, HR, BVP]
SleepFM: Multi-modal Representation Learning for Sleep Across Brain Activity, ECG and Respiratory Signals, ICML 2024; [ECG, EEG, EOG, EMG, Respiratory Sensors]
A generative foundation model for five-class sleep staging with arbitrary sensor input, arxiv 2024; [EEG, EOG, EMG]
Embracing Large Language and Multimodal Models for Prosthetic Technologies, arxiv 2024; [EMG, Image, Video, Text, Audio]
EMG-to-Speech: Direct Generation of Speech From Facial Electromyographic Signals, IEEE/ACM Transactions on Audio, Speech, and Language Processing 2024; [EMG, Speech]
Promoting cross-modal representations to improve multimodal foundation models for physiological signals, Medical Foundation Models (AIM-FM) workshop at NeurIPS 2024; [EEG, EMG, EOG, ECG]
Learning under Label Noise through Few-Shot Human-in-the-Loop Refinement, arxiv 2024; [EEG, IMU, EMG, EOG, ECG]
PedSleepMAE: Generative Model for Multimodal Pediatric Sleep Signals, IEEE International Conference on Biomedical and Health Informatics 2024 2024; [EEG, EMG, EOG]
A Multimodal Sleep Foundation Model Developed with 500K Hours of Sleep Recordings for Disease Predictions, medRxiv 2025; [EEG, EOG, EMG, ECG, Respiratory]
Towards Robust Multimodal Physiological Foundation Models: Handling Arbitrary Missing Modalities, arxiv 2025; [EEG, ECG, EMG, EOG]
Robust Sleep Staging over Incomplete Multimodal Physiological Signals via Contrastive Imagination, NeurIPS 2024; [EEG, EMG, EOG]
PainFormer: a Vision Foundation Model for Automatic Pain Assessment, arxiv 2025; [ECG, Image, EMG, GSR, fNIRS]
TimeMaster: Training Time-Series Multimodal LLMs to Reason via Reinforcement Learning, arxiv 2025; [ECG, EMG, Text]
Efficient Personalized Adaptation for Physiological Signal Foundation Model, ICML 2025; [EEG, EOG, EMG, ECG]

Other 🪧 + X

Multimodal Representation Learning of Cardiovascular Magnetic Resonance Imaging, ICML Workshop on Machine Learning for Multimodal Healthcare Data 2023; [CMR, Text]
Biometric Recognition Using Multimodal Physiological Signals, IEEE 2019; [Heart Rate, Breathing Rate, Palm Electrodermal Activity, Perinasal Perspitation]
Video-based multimodal spontaneous emotion recognition using facial expressions and physiological signals, CVPR Workshop 2022; [Image, iPPG, Heart Rate]
Emotion Recognition from Multimodal Physiological Signals for Emotion Aware Healthcare Systems, Journal of Medical and Biological Engineering 2020; [Respiratory Belt, Photoplethysmography, Fingertip Temperature]
Every word counts: A multilingual analysis of individual human alignment with model attention, AACl 2022; [Eye-Tracking, Text]
Early Life Stress Detection Using Physiological Signals and Machine Learning Pipelines, Biology 2023; [Heart Rate, GSR]
Comparative Analysis of Emotion Classification Based on Facial Expression and Physiological Signals Using Deep Learning, Applied Sciences 2022; [Facial Expressions, HRV]
Towards Autonomous Physiological Signal Extraction From Thermal Videos Using Deep Learning, ICMI 2023; [HR, Respiration Rate, Temperature]
MAD: Multi-Alignment MEG-to-Text Decoding, arxiv 2024; [MEG, Text]
BTS: Bridging Text and Sound Modalities for Metadata-Aided Respiratory Sound Classification, InterSpeech 2024; [Respiratory Sound, Text]
Brant: Foundation Model for Intracranial Neural Signal, NeurIPS 2023; [iEEG]
BrainBERT: Self-supervised representation learning for intracranial recordings, ICLR 2023; [iEEG]
Evaluating Large Language Models on Time Series Feature Understanding: A Comprehensive Taxonomy and Benchmark, arxiv 2024; [Time Series Agnostic]
TOTEM: TOkenized T ime Series EMbeddings for General Time Series Analysis, arxiv 2024; [Time Series Agnostic]
SIM-CNN: Self-Supervised Individualized Multimodal Learning for Stress Prediction on Nurses Using Biosignals, medRxiv 2023; [EDA, HR, ST, Interbeat Interval, BVP, Accelerometer]
Interpretation of Intracardiac Electrograms Through Textual Representations, CHIL 2024; [EGM, Text]
Multimodal Risk Prediction with Physiological Signals, Medical Images and Clinical Notes, medRxiv 2023; [Vision, Text, EHR]
Do Self-Supervised Speech and Language Models Extract Similar Representations as Human Brain?, ICASSP 2024; [ECoG, Speech, Text]
Emotion Recognition From Multimodal Physiological Signals via Discriminative Correlation Fusion With a Temporal Alignment Mechanism, IEEE Transactions on Cybernetics 2023; [CNS, ANS]
Health-LLM: Large Language Models for Health Prediction via Wearable Sensor Data, CHIL 2024; [Time Series Agnostic]
BIOT: Biosignal Transformer for Cross-data Learning in the Wild, Neurips 2023; [Time Series Agnostic]
PhysioMTL: Personalizing Physiological Patterns using Optimal Transport Multi-Task Regression, CHIL 2022; [Time Series Agnostic]
Transformers in biosignal analysis: A review, Elsevier Information Fusion 2024; [Time Series Agnostic]
Towards a Personal Health Large Language Model, arxiv 2024; [Time Series Agnostic]
Event-Based Contrastive Learning for Medical Time Series, MLHC 2024; [Time Series Agnostic]
PaPaGei: Open Foundation Models for Optical Physiological Signals, arxiv 2024; [PPG]
Multi-Modal Forecaster: Jointly Predicting Time Series and Textual Data, arxiv 2024; [Time Series Agnostic]
COCOA: Cross Modality Contrastive Learning for Sensor Data, Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2022; [Time Series Agnostic]
Language Models Still Struggle to Zero-shot Reason about Time Series, arxiv 2024; [Time Series Agnostic]
TimeSeriesExam: A time series understanding exam, NeurIPS 2024 Time Series in the Age of Large Models Workshop; [Time Series Agnostic]
LLM-ABBA: Understand time series via symbolic approximation, arxiv 2024; [Time Series Agnostic]
ChatTS: Aligning Time Series with LLMs via Synthetic Data for Enhanced Understanding and Reasoning, arxiv 2024; [Time Series Agnostic]
LETS-C: Leveraging Text Embedding for Time Series Classification, arxiv 2024; [Time Series Agnostic]
Wearable Accelerometer Foundation Models for Health via Knowledge Distillation, arxiv 2024; [PPG, Accelerometer]
A Multimodal Sleep Foundation Model Developed with 500K Hours of Sleep Recordings for Disease Predictions, medRxiv 2025; [EEG, EOG, EMG, ECG, Respiratory]
ADAPTive Input Training for Many-to-One Pre-Training on Time-Series Classification, arxiv 2025; [Time Series Agnostic]
AAD-LLM: Neural Attention-Driven Auditory Scene Understanding, arxiv 2025; [iEEG, Text, Audio]
By My Eyes: Grounding Multimodal Large Language Models with Sensor Data via Visual Prompting, EMNLP 2025; [Time Series Agnostic]
SACM: SEEG-Audio Contrastive Matching for Chinese Speech Decoding, arxiv 2025; [SEEG, Audio]
SensorLM: Learning the Language of Wearable Sensors, arxiv 2025; [PPG, ACC, EDA, TEMP, ALT]

Datasets

SEED; [EEG, Eye Movement, Video]
SEED-IV; [EEG, Eye Movement, Video]
SEED-VIG; [EEG, EOG]
SEED-V; [EEG, Eye Movement, Video]
SEED-GER; [EEG, Eye Movement, Video]
SEED-FRA; [EEG, Eye Movement, Video]
ZuCo 2.0; [EEG, Text, Eye Movement]
DEAP; [EEG, Peripheral Physiological Signals, Video]
K-EmoCon; [EEG, Peripheral Physiological Signals, Video, Audio]
AMIGOS; [EEG, ECG, GSR, Video]
Multimodal Adaptive Emotion Transformer with Flexible Modality Inputs on A Novel Dataset with Continuous Labels; [EEG, Eye Movement]
Dataset of Speech Production in intracranial Electroencephalography; [Intracranial EEG, Text]
Temple University Hospital EEG Seizure Corpus; [EEG, Artifacts]
ECG & EEG Stress Features; [EEG, ECG]
A multi-modal open dataset for mental-disorder analysis; [EEG, Text]
Sleep-EDF; [EEG, EOG, Chin EMG, Event Makers]
Neurosity EEG Dataset; [EEG]
ECG-QA; [ECG, Text]
A Large and Rich EEG Dataset for Modeling Human Visual Object Recognition; [EEG, Image]
MIMIC-IV-ECG: Diagnostic Electrocardiogram Matched Subset; [ECG, EHR, Text]
EEG dataset and OpenBMI toolbox for three BCI paradigms: an investigation into BCI illiteracy; [EEG, EMG]
EIT-1M: One Million EEG-Image-Text Pairs for Human Visual-textual Recognition and More; [EEG, Image, Text]
Alljoined1 - A dataset for EEG-to-Image decoding; [EEG, Image]
MindBigData; [EEG, Image]
Human EEG recordings for 1,854 concepts presented in rapid serial visual presentation streams; [EEG, Image]
EEG-ImageNet: An Electroencephalogram Dataset and Benchmarks with Image Visual Stimuli of Multi-Granularity Labels; [EEG, Image]
The Dutch EEG Speech Register Corpus; [EEG, Audio]
EAV: EEG-Audio-Video Dataset for Emotion Recognition in Conversational Contexts; [EEG, Audio, Video]
IMPRESSION; [ECG, BVP, GSR, Video, Audio, Eye Gaze]
EEVR; [PPG, EDA, Video]
KMED dataset; [EEG, Image, Video]
ECG-Chat Pretrain Dataset; [ECG, Text]
ECG-Chat Instruction Tuning Dataset; [ECG, Text]
PULSE-ECG ECGInstruct Dataset; [ECG, Image, Text]
PULSE-ECG ECGBench Dataset; [ECG, Image, Text]
BIDMC PPG and Respiration Dataset; [ECG, PPG]
PPG-DaLiA; [ECG, PPG]
WESAD (Wearable Stress and Affect Detection); [ECG, EDA, EMG]
ECG-Expert-QA: A Benchmark for Evaluating Medical Large Language Models in Heart Disease Diagnosis; [ECG, Text]
ECG-Grounding; [ECG, Image, Text]

Laboratories

Citation

If you found this repository helpful in your research, please cite the following:

@software{han_2025_15151285,
  author       = {Han, William},
  title        = {A corpus of resources for Multimodal Learning with
                   Physiological Signals
                  },
  month        = apr,
  year         = 2025,
  publisher    = {Zenodo},
  version      = {second},
  doi          = {10.5281/zenodo.15151285},
  url          = {https://doi.org/10.5281/zenodo.15151285},
  swhid        = {swh:1:dir:d44f23c5c3ed58e8f53d29c22dab067c30f0769b
                   ;origin=https://doi.org/10.5281/zenodo.10903466;vi
                   sit=swh:1:snp:571aa8cdb6e22b70e7833f00c447e7841903
                   938b;anchor=swh:1:rel:9fc43ffed2239a01f7dc6a4a96c3
                   ccac93e3dea1;path=willxxy-awesome-mmps-68d74c0
                  },
}