Can a neural network tell if an image is mirrored? – Visual Chirality

Key Takeaways

hello and welcome to this ai coffee break this time we will discuss a very important phenomenon in our world reflection but most importantly we will see how this paper entitled visual chirality analyzes mirroring images when training deep neural networks and how this can affect how these algorithms are trained to perceive our visual reality but first a little introduction of the notions of chirality and data augmentation if you are an expert on these topics already feel free to skip to the next chapter in the video so let's start with chirality every human has experienced what mirroring is perhaps the first time as a baby looking into a mirror physicists and mathematicians understand this phenomenon pretty well and can describe it formally to reflect this image of a face everything one needs to do is to multiply all coordinates by almost an identity matrix almost because the reflection matrix has a catch a minus for the x dimension does making the image go the other way with neural networks processing images the way they handle reflections is very different especially for classical image processing architectures like the convolutional neural network or short cnn but first we need to understand convolutions the convolution operation does something similar to what happens in our eyes it bundles information from image regions together learning parameters that determine how exactly to sum this information up in this architecture convolutions ensure the so called shift invariance meaning that if we have the same pattern or object at a different position in the image like this blue shape here the response will not change because the aggregating parameters stay the same by construction this makes sense as we want to detect a dog independently of its exact position in the image so in this case of a convolutional architecture no other parameters are learnt when shifting but something very different happens when the image is mirrored convolutional neural networks are not invariant to reflections this is because convolutions are very sensitive to ordering and the parameter learned for field number four is not a sensible parameter when applied to the mirrored version anymore this means that the network has to learn one set of parameters when presented with an image and a different set of parameters if presented with the same image but mirrored so while convolutions are specifically built to handle translation of the objects through the image very well there is nothing intrinsic to the network telling it the connection between the mirrored and the non-mirrored version of an image but this however has the following implication if the cnn has to learn different parameters for the image and for the mirrored version it means that the network can do useful parameter updates from both versions and because more training data is always a good bet for deep learning researchers thought if we mirror images we can double our training set if we mirror on the other axis we have now four times as many images and of course this does not make sense for some kind of images that have a preference for the sky to stay upwards but anyway here it is reflection one way to do data augmentation and it is cool or this is the state where this paper here came in to scold us all a little bit because we were so busy augmenting the data sets with reflected images that we did not think about how this might impact the statistics in the data and how we know machine learning is all about fitting the data in very many different ways from linear regression to deep neural models so surprise surprise if we change the data distribution by mirroring our super duper fitting machine might notice that so how can we detect mirroring with a neural network if mirroring happened we as humans can detect it thanks to special objects with the chiral property take for example a left hand if we mirror it we have a right hand but if we try to overlap it with the original we cannot unless we mirror this reflection again these special objects that help us humans detect mirroring are called chiral in the geometrical sense but can computer vision models detect this chirality too well it seems they can to prove this the authors define the concept of visual chirality because what holds for geometric chirality does not necessarily hold for images take the example of a left hand if mirrored it might be as well a right hand in an image it would not necessarily mean that the image has been flipped so instead the authors start with defining visual i chirality if we want to estimate the distribution d of x where x is our data it might be for example the images we usually perform data augmentation by applying a transformation t to the data x this transformation could be the reflection for example for i chiral data this equation here would hold meaning that there would be no difference of the distribution of the data before and after the transformation then the degree in which an image x would violate this equation is a measure for visual chirality so to recapitulate a human hand would be geometrically chiral but not visually chiral but what would be visual chiral then well clocks text but also let's not spoil everything and let the neural network show us what it looks at to tell mirrored images apart from non-mirrored ones for this the authors train a widely used cnn a resnet to classify reflected from non-reflected images and the network gets pretty good at it reaching accuracies up to 90 when testing on the test subset of the instagram data set when being applied to other images a little out of domain it still performs better than random chance at telling mirrored images from non-mirrored images apart they also visualize the discriminative regions of the image giving us a hint at what the network is focusing on when giving the answer it looks at smartphones it watches shirt colors and pockets but also they show that chirality is present in face images too on the applied ffhq data set here seems to be a hint for chirality because the pictures imply that the majority of people part their hair from left to right also gaze is a hint for chirality suggesting a bias in gaze direction possibly because of ocular dominance these are high level features that give away the visual chirality of the image even more interestingly the authors also detect low-level chiral cues they are artifacts from image processing steps like compression i look at you jpeg and make the images chiral these effects are imperceptible to humans but this is something that the neural network could capture so all this time we were using image reflection for data augmentation in computer vision and not many of us were looking at the subtleties like this paper does this is my favorite kind of paper it has a lot of insight gained with a very simple idea and it was always lurking in the mirror when we were not looking great job if you like this paper at least as much as the coffee bean does check out the link in the description below if you like this video do not forget to hit the like and subscribe at hands and let me know in the comments what topic you want miss coffee bean to explain next [Music] you