Noise-guided Masking for Image-based Joint-Embedding Predictive Architectures

Abstract

In masked image modeling methods, both data-adaptive and data-independent masking strategies are employed. While noise-guided masking strategies that are data-independent have been observed to improve generative models such as Masked Autoencoders (MAE), their impact on non-generative frameworks remains unexplored. This work fills this gap by adapting noise-guided masking to the Image-based Joint Embedding Predictive Architecture (I-JEPA). Our method, which guides the selection of regions to mask using spectrally filtered noise improves training efficiency by 25–30% while matching or exceeding multi-block masking performance in linear probing on ImageNet-1K.