1D convolution for neural networks, part 1: Sliding dot product

Key Takeaways

the power of convolutional neural networks comes from the convolution operator which is a way that features can be pulled out of a signal if it's a two-dimensional signal an image these can be little pieces of the thing you want to find if you're looking for a cat it can be an eye or an ear or a tail what makes convolutional neural networks so robust is they don't care exactly where these are placed in relationship to each other they can be rearranged a little bit like Picasso and the network would still be able to identify them there's two parts to this one is the convolution operator that matches the feature and the other is a pooling operator that gives a little bit of wiggle room to the exact position of where that feature is located so we're going to focus on the convolution piece the pulling of these features out of the signal we can also do convolution on one-dimensional signals for instance audio stock prices anything that can be ordered along a single line often any data that's organized by time fits on a nice one-dimensional line it could also be applied for instance to three dimensional signals say video where you have both the X and y position of a pixel and its position in time because our paper because our images are two-dimensional starting with a low dimensional signal to illustrate the process will be helpful convolution is the process of taking a kernel and doing a sliding dot product with a signal here the signal is the thing that you're trying to classify or to pull the feature out of it might be the image or a snippet of audio or a portion of an electrocardiogram we can visualize this as a lollipop plot where each value is represented sequentially on a line and the height of the lollipop stem is the value of that point points falling right on the axis have a value zero points above the axis have a positive value and points below the axis have a negative value and the length of that lollipop stem shows the magnitude of the value to do convolution we have a signal and a kernel by convention the kernel is much smaller than the signal it's the portion it's the feature that you'd like to pull out of some part of the signal wherever it occurs it's the fingerprint of the feature that we're trying to find the first step of convolution is you take the kernel and you flip it around left to right then you take your flipped kernel and slide it along the signal at each location where your points in your kernel line up with your signal you multiply together the value of any points that line up and then you add all of those together to get the final result so for instance in this location the kernel lines up with these three values of the signal but they're all equal to zero they all sit exactly on the line so even though the kernel has positive values for all those they all get multiplied by zero and added together giving a result of zero so the convolution at that point will be zero and then you slide it one step to the right and do it again this process of multiplying each aligned pair of points together and then adding all of those products together is called taking the dot product and moving our kernel one position at a time and repeating this is a sliding dot product so a convolution is a sliding dot product of a flipped kernel with this signal we can see how this plays out as if we go point by point it's only where both the signal and the kernel are not zero that the result of the convolution is not zero