Discovering Symbolic Models from Deep Learning with Inductive Biases (Paper Explained)

Neural networks are very good at predicting systems' numerical outputs, but not very good at deriving the discrete symbolic equations that govern many physical systems. This paper combines Graph Networks with symbolic regression and shows that the strong inductive biases of these models can be used to derive accurate symbolic equations from observation data. OUTLINE: 0:00 - Intro & Outline 1:10 - Problem Statement 4:25 - Symbolic Regression 6:40 - Graph Neural Networks 12:05 - Inductive Biases for Physics 15:15 - How Graph Networks compute outputs 23:10 - Loss Backpropagation 24:30 - Graph Network Recap 26:10 - Analogies of GN to Newtonian Mechanics 28:40 - From Graph Network to Equation 33:50 - L1 Regularization of Edge Messages 40:10 - Newtonian Dynamics Example 43:10 - Cosmology Example 44:45 - Conclusions & Appendix Paper: https://arxiv.org/abs/2006.11287 Code: https://github.com/MilesCranmer/symbolic_deep_learning Abstract: We develop a general approach to distill symbolic representations of a learned deep model by introducing strong inductive biases. We focus on Graph Neural Networks (GNNs). The technique works as follows: we first encourage sparse latent representations when we train a GNN in a supervised setting, then we apply symbolic regression to components of the learned model to extract explicit physical relations. We find the correct known equations, including force laws and Hamiltonians, can be extracted from the neural network. We then apply our method to a non-trivial cosmology example-a detailed dark matter simulation-and discover a new analytic formula which can predict the concentration of dark matter from the mass distribution of nearby cosmic structures. The symbolic expressions extracted from the GNN using our technique also generalized to out-of-distribution data better than the GNN itself. Our approach offers alternative directions for interpreting neural networks and discovering novel physical principles from the representations they lea