CrossJEPA vs. JEPA-WM: Advancing Efficient 3D Learning via Cross-Modal Latent View Prediction. JEPA.

The Masking vs. Cross-Modal Shift. In a standard JEPA World Model, the architecture learns by seeing a partial "context" and predicting the latent embedding of a "target" block that has been masked out. This forces the model to understand the internal structure of the data (like the "world rules" of how an object looks or moves). CrossJEPA replaces the masking mechanism with a cross-modal challenge. It doesn't mask the 3D point cloud; instead, it asks the model: "Given this 3D shape, what would its 2D embedding look like from this specific camera angle? Teacher-Student Dynamics Standard JEPA models typically use a Self-Distillation approach where the teacher is a momentum-updated version of the student. CrossJEPA opts for a Frozen Teacher approach. By using a pre-trained 2D foundation model as the teacher, the 3D student "inherits" the massive visual vocabulary of the 2D model without the 2D model ever needing to be updated during 3D training. Computational Footprint The "World Model" variants of JEPA are efficient compared to Contrastive Learning (like CLIP), but CrossJEPA takes efficiency a step further. Because the 2D teacher is frozen, its embeddings for various views can be precomputed and cached. This removes the need to run the massive teacher model during every training epoch, allowing the 3D encoder to train at speeds that traditional JEPA models—which must process masks and teacher updates in real-time—rarely match.