1D convolution for neural networks, part 8: Padding

Key Takeaways

so far we've been assuming that the number of outputs is the same as the number of inputs so that means that we take our kernel we flip it it's odd-numbered and we put that central value of the kernel line it up with the first value of our inputs and then do our sliding dot product across and stop do our last evaluation when the center value of the kernel lines up with the last element in our signal what that does then is it gives us an output a result that's exactly the same length as the input there's actually no reason why this has to be the case so we can do something called padding which is adding reasonable values to the end of our signal so that we can convey a love that kernel first touches the first non-zero value of the signal it can still give us an output result when we pad with zeros we can get a reasonable set of outputs to go along with that when we the process that we were talking about before is essentially padding with zeros assuming that everything that is outside of that is zero and treating it as such but there are other things that we can do there are lots of ways to pad we could actually pad with a constant value this then might be more representative of the data that we're using the goal of padding is to extend the data set in a neutral way so it doesn't introduce new information or new artifacts any more than necessary so it could be that for your particular data set let's say you're estimating temperatures and you'd like to pad that out a little bit an average temperature wouldn't be zero an average temperature on neutral temperature you know might be 50 degrees Fahrenheit 20 degrees Celsius and you can use that as a fairly neutral way to extend your data set notice though that it does give you a different result those padded values matter they get folded into the convolution result so you want to think about what you use there another thing you can do that might be more representative of your data is to do a mirror padding so if you want to pad would say five values you actually go and take the last five values of your data set flip them around and tack them on to the end similarly you can take the first five flip them around and tack them on to the beginning this mirror padding gets you a nice neutral extension of your data set for the most part but you have to be careful of weird cases for instance if you're tracking weekly phenomena and say your weekends are very different from your weekdays if you do this you might end up with ten weekdays in a row or four or weekend days in a row so it's not entirely neutral all the time you have to be aware of what your data means notice again when you look at the results the results are different depending on how you Pat it another thing that you can do that occasionally make sense is to do circular padding so now instead of taking the elements off the end and flipping it and tacking it on the end as we do and mirror padding we take our five elements off the end reach back around and tack them on to the beginning of the data set similarly we take our five elements or in this case our four elements off the beginning and reach around and tack them on to the end of the data set this can be particularly useful if you're working with data that you know to be cyclical you know to change on a regular basis and if you sampled it over a round number of those cycles it can be a nice neutral way to extend the data set again though notice that it gives you a very different answer depending on how you pad so the takeaway here is to pad mindfully because when you're padding you're actually adding data to your signal if you don't do it well you can end up corrupting your signal