Multimodal Knowledge Distillation for Egocentric Action Recognition Robust to Missing Modalities

Maria Santos-Villafranca^{* 1}    Dustin Carrión-Ojeda^{* 2,3}    Alejandro Perez-Yus¹    Jesus Bermudez-Cameo¹    Jose J. Guerrero¹    Simone Schaub-Meyer^2,3

¹I3A - University of Zaragoza    ²Technical University of Darmstadt    ³hessian.AI
^*Equal contribution

Accepted at ICRA 2026

TL;DR 🚀: KARMMA is a multimodal-to-multimodal distillation framework for egocentric action recognition that does not require modality-aligned data and supports any subset of modalities at inference. It produces a fast and lightweight student that remains robust under missing modalities without retraining.

Installation

This project was developed on Linux with Python 3.9, PyTorch 2.1.1, and CUDA 12.1. To reproduce the environment:

# 1) Clone the repository
git clone https://github.com/visinf/KARMMA

# 2) Move into the repository
cd KARMMA

# 3) Create and activate the environment
conda create -n karmma python=3.9
conda activate karmma

# 4) Install the dependencies
pip install -r requirements.txt

The versions in requirements.txt match the environment used for the released experiments. For a different CUDA version or a CPU-only environment, install the appropriate PyTorch, TorchVision, and TorchAudio builds from the PyTorch installation guide before installing the remaining dependencies.

Dataset Preparation

KARMMA uses EPIC-KITCHENS-100 and Something-Something-v2. The data-loading pipeline expects RGB frames, optical flow, and audio resources in HDF5 files. The Something-Something-v2 metadata files also contain the object detections used by the layout encoder.

The required datasets and preprocessing format can be obtained by following the instructions from Multimodal Distillation for Egocentric Action Recognition.

The EPIC-KITCHENS-100 annotation files used in our experiments are included in:

data/ek100_splits/
    train.csv
    validation.csv
    custom_train.csv
    custom_validation.csv

Before training or evaluation, replace the path placeholders in the corresponding configuration file:

DATA:
  VIDEOS_PATH: <PATH_TO_DATASET>/dataset.hdf5
  FLOW_PATH: <PATH_TO_DATASET>/flow_dataset.hdf5
  AUDIO_PATH: <PATH_TO_DATASET>/audio_dataset.hdf5

For Something-Something-v2, also update TRAIN_DATASET_PATH, VAL_DATASET_PATH, TEST_DATASET_PATH, and LABELS_PATH. Audio is only used for EPIC-KITCHENS-100.

Pre-trained Encoders

The teacher uses Swin-B encoders for RGB, optical flow, and audio, while the student uses Swin-T for RGB and optical flow. Download the corresponding checkpoints:

mkdir -p models/pre-trained

wget https://github.com/SwinTransformer/storage/releases/download/v1.0.4/swin_base_patch244_window877_kinetics400_22k.pth \
    -P models/pre-trained/

wget https://github.com/SwinTransformer/storage/releases/download/v1.0.4/swin_tiny_patch244_window877_kinetics400_1k.pth \
    -P models/pre-trained/

For the Something-Something-v2 teacher, download the STLT model pre-trained on Action Genome detections from the STLT model zoo.

The EPIC-KITCHENS-100 student uses AST-T for audio, and the Something-Something-v2 student uses STLT-9 for object detections. These encoders do not require a checkpoint path in the provided training configs. The AST implementation initializes its transformer through timm, which downloads the DeiT-T weights on first use unless they are already cached.

Configurations

Training configurations are organized by dataset:

Dataset	KARMMA_T	KARMMA_S
EPIC-KITCHENS-100	configs/ek100/karmma_t.yaml	configs/ek100/karmma_s.yaml
EPIC-KITCHENS-100 custom split	configs/ek100_custom/karmma_t.yaml	configs/ek100_custom/karmma_s.yaml
Something-Something-v2	configs/ssv2/karmma_t.yaml	configs/ssv2/karmma_s.yaml

Each config contains placeholders for the local dataset. The remaining hyperparameters are inherited from utils/defaults.py.

Training

KARMMA is trained in two stages. First, the multimodal teacher is trained with frozen feature extractors. The resulting teacher is then frozen and used to train the student through knowledge distillation.

KARMMA_T

For example, train the EPIC-KITCHENS-100 teacher with:

python karmma_t.py \
    --config_path configs/ek100/karmma_t.yaml \
    --devices 0

KARMMA_S

Before training the student, set STUDENT_CONFIG.TEACHER_PATH to the experiment directory containing the trained teacher. Then, run:

python karmma_s.py \
    --config_path configs/ek100/karmma_s.yaml \
    --devices 0

Multiple GPUs can be selected with a comma-separated list such as --devices 0,1,2. The per-device batch size and data-loading settings can be adjusted under TRAIN.DATA_LOADER.

To resume an interrupted experiment from last.ckpt, use the same config and add --resume.

Checkpoints

We provide the KARMMA_T and KARMMA_S checkpoints used to reproduce the paper results. Each archive contains both models and their evaluation configurations for one dataset.

Dataset	Checkpoints
EPIC-KITCHENS-100	download
EPIC-KITCHENS-100 custom split	download
Something-Something-v2	download

Download the required archives and extract them into experiments/:

mkdir -p experiments

wget https://github.com/visinf/KARMMA/releases/download/v1.0.0/karmma_ek100.zip
unzip karmma_ek100.zip -d experiments

wget https://github.com/visinf/KARMMA/releases/download/v1.0.0/karmma_ek100_custom.zip
unzip karmma_ek100_custom.zip -d experiments

wget https://github.com/visinf/KARMMA/releases/download/v1.0.0/karmma_ssv2.zip
unzip karmma_ssv2.zip -d experiments

The resulting structure should be:

experiments/
    ek100/
        karmma_t/
            config.yaml
            logs/version_0/checkpoints/best_model.ckpt
        karmma_s/
            config.yaml
            logs/version_0/checkpoints/best_model.ckpt
    ek100_custom/
        karmma_t/
        karmma_s/
    ssv2/
        karmma_t/
        karmma_s/

Update the dataset path placeholders in each downloaded config.yaml before evaluation.

Evaluation

Evaluate a teacher checkpoint with:

python karmma_t.py \
    --experiment_path experiments/ek100/karmma_t \
    --devices 0 \
    --eval \
    --eval_modalities video,flow,audio \
    --eval_mod_drop_rate 0.0

Evaluate a student checkpoint with:

python karmma_s.py \
    --experiment_path experiments/ek100/karmma_s \
    --devices 0 \
    --eval \
    --eval_modalities video,flow,audio \
    --eval_mod_drop_rate 0.0

--eval_modalities selects the modalities initialized by the dataloader and model. The order is normalized according to the checkpoint configuration, so video,audio,flow and video,flow,audio are equivalent.

--eval_mod_drop_rate controls stochastic modality dropout during evaluation. It accepts values in [0, 1). Pass it explicitly for reproducible evaluation; when omitted, the dropout rate stored in the checkpoint configuration is used. When evaluating a single modality, this value must be 0.0.

By default, metrics are printed to the terminal. They can also be saved as JSON:

python karmma_s.py \
    --experiment_path experiments/ek100/karmma_s \
    --devices 0 \
    --eval \
    --eval_modalities video,flow,audio \
    --eval_mod_drop_rate 0.0 \
    --results_path results \
    --results_file ek100-karmma_s-vfa

Evaluation Scripts

The scripts in eval_scripts/ evaluate all modality combinations and modality-dropout settings required to reproduce the paper results. Each script receives the GPU ID as its only argument and saves the metrics under results/.

For EPIC-KITCHENS-100:

(cd eval_scripts/ek100 && bash karmma_t.sh 0 && bash karmma_s.sh 0)

For the custom EPIC-KITCHENS-100 split:

(cd eval_scripts/ek100_custom && bash karmma_t.sh 0 && bash karmma_s.sh 0)

For Something-Something-v2:

(cd eval_scripts/ssv2 && bash karmma_t.sh 0 && bash karmma_s.sh 0)

Repository Structure

KARMMA/
    configs/             # Training configurations
    data/
        datasets/        # Dataset implementations
        ek100_splits/    # EPIC-KITCHENS-100 annotation files
        utils/           # Sampling, augmentation, and preprocessing
    eval_scripts/        # Paper reproduction scripts
    karmma/
        encoders/        # Swin, AST, and STLT encoders
        lightning/       # PyTorch Lightning modules
        losses/          # Teacher and student objectives
        models/          # KARMMA teacher, student, fusion, and classification modules
    utils/               # Configuration, logging, and evaluation utilities
    karmma_t.py          # KARMMA_T training and evaluation
    karmma_s.py          # KARMMA_S training and evaluation

Citation

If you find our work helpful, please consider citing the paper and ⭐ the repository.

@inproceedings{carrion2026karmma,
    title={Multimodal Knowledge Distillation for Egocentric Action Recognition Robust to Missing Modalities},
    author={Dustin Carrión-Ojeda* and Maria Santos-Villafranca* and Alejandro Perez-Yus and Jesus Bermudez-Cameo and Jose J. Guerrero and Simone Schaub-Meyer},
    booktitle={Proceedings of the IEEE International Conference on Robotics & Automation (ICRA)},
    year={2026},
    note={to appear}
}

Acknowledgements

Parts of the data-loading pipeline and multimodal processing build on Multimodal Distillation for Egocentric Action Recognition. The encoder implementations build on Video Swin Transformer, Audio Spectrogram Transformer, and Revisiting Spatio-Temporal Layouts. We thank the authors for making their code and models available.

This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy (EXC-3057/1 "Reasonable Artificial Intelligence", Project No. 533677015). We gratefully acknowledge support from the hessian.AI Service Center (funded by the Federal Ministry of Research, Technology and Space, BMFTR, grant No. 16IS22091) and the hessian.AI Innovation Lab (funded by the Hessian Ministry for Digital Strategy and Innovation, grant No. S-DIW04/0013/003). SSM has been funded by the DFG – 529680848. MSV, APY, JBC, and JJG further acknowledge support by projects PID2024-158322OB-I00 and PID2021-125209OB-I00, (MCIN/AEI/10.13039/501100011033/ FEDER, UE), project JIUZ2024-IyA-07, DGA 2022-2026 scholarship, and Grant SMT Erasmus+, project 2022-1-ES01-KA131-HED-000065592 funded by Campus Iberus.