Can Cross-Layer Transcoders Replace Vision Transformer Activations? An Interpretable Perspective on Vision

arXiv:2604.13304v1 Announce Type: cross Abstract: Understanding the internal activations of Vision Transformers (ViTs) is critical for building interpretable and trustworthy models. While Sparse Autoencoders (SAEs) have been used to extract human-interpretable features, they operate on individual layers and fail to capture the cross-layer computational structure of Transformers, as well as the relative significance of each layer in forming the last-layer representation. Alternatively, we introdu