On the Geometric Structure of Layer Updates in Deep Language Models

1. 1Decompose layerwise updates into tokenwise and residual components
2. 2Analyze the geometric structure of these components across multiple architectures
3. 3Apply insights to improve model interpretability and performance

ML researchers and AI engineers on a team benefit from understanding the geometric structure of layer updates to improve model performance and interpretability, while software engineers can apply these insights to optimize model architecture

💡 Layer updates can be decomposed into a dominant tokenwise component and a residual pattern

💡 Layer updates in deep language models have a geometric structure with dominant tokenwise and residual components