Estimating Sheep Pain with Facial Recognition

Key Takeaways

all right before we get started Minneapolis Minnesota anybody who lives there or anybody with the wherewithal to get themselves to the Twin Cities you guys listen up anybody else just hit skip 30 seconds twice I'm going to be at farcon 2017 that's August 23rd and 24th I'm telling you right now cuz you got to go get tickets the conference is free of charge thanks to the sponsors you got to get over to farcon 2017. eventbrite.com to pick yours up that's going to be in the show notes as well tickets just became available so if you're listening to this on Friday the day of release you got a good shot at getting a ticket we'll follow up down the line with a few announcements about what's going to be going on there trust me it's going to be worth your while you can read all about it on their website if you're in Minneapolis or you can get yourself there go pick up a ticket like I said these are going to move fast cuz it's free and it's a great event good lineup if you're hearing this late and you want to go and it's sold out maybe come talk to me I'll see if we can sneak somebody in the back or something maybe there's some extra tickets I can squeeze out of them get on slack DM me we'll talk about it but try and go over there and get registered if you can farcon 2017. eventbrite.com data skeptic is the official podcast of datas skeptic.com bringing you stories interviews and mini episodes on topics in data science machine learning statistics and artificial [Music] intelligence Mara Mahmud received her PhD in computer science at the University of Cambridge England in 2015 she is currently a post-doctoral researcher in the graphics and interaction group at the University of Cambridge computer laboratory where she studies emotional inference from gestures and expressions Mara's research interests lie in the field of Effective computing social signal processing and automating machine understanding of emotional body language aspects of her research draw on computer vision machine learning human computer interaction and psychology marel holds a BS and an MS in computer science from the American University in Cairo Egypt marwel welcome to data skeptic welcome hi k so I've uh been enjoying reading through a lot of your research paper you do some interesting work that uh I was not previously deeply familiar with and I came across one particular paper I wanted to talk to you about uh estimating sheep pain with facial recognition so the title sort of tells it all but I'd love to hear your background on what got you interested in uh conducting a study like this so this work started by when my group um actually my supervisor Peter Robinson he got approached by the vet school here in bridge where they were wondering if we can apply the face analysis techniques we use uh on humans on sheep the main contact was actually Christa mclenon who developed a scoring system that Maps specific changes in the facial features of the sheep and map these changes to a score a pain uh level score and the main idea or why this was used mainly is for early detection of some painful diseases in sheep so this pain level scale is used to detect if the Sheep is in pain and since it's based on specific changes in a facial feature and this this is summed up and then threshold and and then a threshold is applied to decide if the Sheep is on pain or not so because there is this scoring system when we talked we thought it's a good idea to really apply same techniques used for human facial expressions and detection of action units on sheep this is how the project started what does a sheep look like when it's in pain or how does it look different from its normal face so the ear flipped is a sign of pain the v-shaped nostril is a sign of of pain also the eyes they also have these sort of partially closed eyes rather than wide open eyes so they have spec so it's really specific features on the face it seems to me that's similar to sort of human just being like have this tense face but it's not it's not very similar I mean we don't change we don't move our our our ears when when we are in P sure but but yeah well there are there are actually clear signs of pain that appear on the sheep face can you tell me a little bit about how you get your ground truth measurement of whether or not they're experiencing pain well we get all the ground truth from vets so we work with Christa mclennen and she's in the vet school she used to work here in Cambridge now it's she's a leurer at University of Chester she works with us on labeling the photos of sheep both labeling of the action unit so labeling on different facial features and label of the final score and this is how we get the ground truth currently the data set as well was collected in the same way so we get all our data from vets that's one of the challenges as well because we need more data but this needs lots of work from vets and from us for uh to do the labeling must be a very expensive and time consuming process to build up a corpus is there you know in the human facial recognition world there's some stuff you can get access to online are there any resources available that can help you in a project like this or do you have to create your entire Corpus no publicly available uh stuff but actually in creating our Corpus we used faces from that were collected from the farm and we augmented our data set as well by just like Googling sheep faces so there are plenty of photos on Google of sheep but then still labeling is needed so so that still doesn't help much challenges also arise because you can't well basically you can't ask a sheep to pose so all the photos will include lots of like background sometimes many sheep at once sometimes lots of other animals pose of the face different viewpoints to start research on that you need to start with like a clean database to start applying your techniques so that was a little bit difficult as well to start with so we excluded lots of images that had maybe different breeds because if you don't have enough data for for example for example sheep with like black faces or half half and half or something then it's better to exclude the whole breed otherwise you won't get enough um results so we hope to extend this work to to be applied for other breeds but again this needs data yeah makes sense which is very similar to human actually I mean the human facial expression analysis the more data more variation this this means better systems better classification system yeah definitely more more data or better data Maybe not so much better algorithms are the key exactly exactly exactly that's corre so you've got these uh veterinarians who fortunately know how to recognize the pain and they can contribute your labels um of course you know that's a specialized skill and not anyone can do it so it makes getting those labels hard but I was also curious is there a threshold to their accuracy you know I I have a a pat parrot and I don't I feel like I understand her too a bit but not even perfectly you know cuz I'm not attuned to perceiving her her pain per se you know how accurate do the human scorers think they get in their labels this is a good point because when we look at the labels for the specific action units that we get from the Vets sometimes it's not very clear for example different for let's take the ear shape for example so it's it's very difficult to uh differentiate between like different rotation so maybe clear ear flat would be very uh different than ear flip but the middle stages might be a little bit tricky and we use these labels in our training but that's why when we when we looked at like the false positives or why we can get where is the error for example the air percentage happen we noticed that this is because some of the labels might really I mean some two shapes might look very similar from like a human eye but we you're still not sure how these were actually labeled by the Vets so maybe the Vets as well don't use own only these different they know I mean somehow maybe they know how a combined features would mean so they would know if the if the Sheep is on pain or not so I'm not sure exactly um how reliable the labeling of the action units that we get from Human labelers would be but I guess if we can have ground Truth at some point if we have more data and also maybe get ground truth on if the Sheep is actually thick or not so maybe this can be um a better ground truth in my opinion instead of having label for every action unit what I call Action Unit which is the difference in different space regions maybe if we can if we have just one score that's a ground truth an actual ground truth that says how sick is this animal then we can maybe train better systems that can jump uh directly from the face to the pain score or the diagnosis would you mind uh defining that term you use an action unit uh action units comes basically from the face facial action coding system the fact that was actually developed by emman it's a systematic way of defining different changes in the facial expressions of human based on the muscle Activation so different action units means different changes in the face so for example there's an action unit for a smile there's an action unit for the eyebrow raise action unit of eye open eye closed and so on it's a coding system for the facial movement this is developed for humans and here we just use the same terminology there isn't really an action unit system that's developed for sheep but I guess from seeing how vets actually Mark every facial feature and have a specific score that's that's actually correlated with every facial feature so it looks it looks very similar to The Human Action units getting back again to the Corpus can you tell me roughly speaking how many labeled images you had to work with so we had around 500 images MH so these were the uh lab we've got so for different stages in our work because we start for example by face detection and for that we did some augmentation for the data so we tried we like cro the faces and try to like put it on different backgrounds just to augment the data a little bit and add more so we reached about 5,000 for the face detection bits but yeah the amount of data working with are only 500 labeled images and we know we've already discussed the cost of of growing that so don't have the luxury that some researchers have with you know imet and mnist and really large databases like that so I imagine you have to do uh some of the pre-processing you discussed but also a fair amount of feature engineering as well the first step was the face detection but later on in detection of the taxonomy so what the features we used are appearance features we use uh histograms of oriented gradients and we use this around the detected facial points so we Define a set a set of box for example for the ear for the eyes and then we uh detect hog out of these uh features and these are then classified using an svm we also explored because of the data and the limited amount of data we did some Exploration with doing binary classification rather than a three-way classifier this improved a little bit the performance also this help in excluding the middle stages for example as I mentioned the ear different so for example intermediate stages are not that clear as in the extreme cases so we did some of that as well another thing as well because of the limited data we just we excluded the profile faces so we just uh used front sheep with frontal faces again because we don't have enough if we don't have enough of the specific uh category so it's better toe exclude it for now and that meant as well excluding some features such as changes in the cheek so I I did mention actually what are the features that are important are mainly ears the nostrils and the eyes and there's also the cheeks but we excluded this bit for the work so far because we don't have profile faces of sheep but actually there are even more action units or more features that can be added if we have more data and that's actually the plans so this summer we're still going to extend the work on that and we're hoping to get some more data that would include some profile faces not necessarily for the same sheep for the same animal but maybe for just different animals and this might actually mean extending the taxonomy as well to include more features so when you've got all your features and then you go train your svm can you tell me a little bit more about those labels exactly I know they're a measurement of pain but is it a regression problem like a score or do you treat it like a categorical problem where there's a few categories of pain what are you trying to learn so there are two scenarios and they are both based on the data that we've got the data that we've got had a label for every feature so for example if the ear is flat or not for every action unit we get a label for its presence so for example you shape nose yes or no so it's a categorization problem it's a classification and that's for every action unit so for the three action units of the ear either uh it's flat slightly rotated or totally flipped these are three action units so we just get a tick which which one of these is present in this photo nostrils for example if it's u-shaped or slightly v-shaped or very extreme V shaped and so on because the data we've got is based on the action units so this is the way we done it in this work we treated the problem as a classification problem for every action unit so we trained an svm for every action unit just to detect the presence so it's a binary problem for every action unit we tried as well another technique which used an svm for every feature so this means it's a three-way classification so for every feature so for example the ear we decide which action unit is present is it ear flat ear slightly rotated or ear flipped and this would be a three-way classification problem which is slightly harder because then you would include lots of the intermediate stages that are not very not not very well labeled it's ambiguous so the intermediate steps can be a little bit ambiguous we experimented with this and then at the end we reduce the pain score from the result that we get from these separate classification these five classification svms why was support Factor machine the right choice to use here I think mainly because of the amount of data so I think for the amount of data we couldn't be using like um deep learning or something that's more sophisticated because we would need more data for that so sbms are simple in terms of the number of of parameters that you need to tune so for this amount of data this would be the best choice we have and it worked I think quite good given that we don't have enough training samples yeah what sort of accuracy did you guys see the most challenging experiment which means like doing this three-way classification for every feature we reached a 67% classification rate which is like fair enough given the amount of data we've got and the amount of ambiguity of the intermediate stages but this actually was boosted to like about 80% or so if we exclude the intermediated stages and treating the problem as binary treating every feature as a binary classification than so this means really excluding the a bit of the ambiguous intermediate stages what do you think is keeping it from being higher is it just more data or there techniques or could it even be that the you know certain animals are are trying to hide their pain does that play into the limited degree to which you can get accuracy on this I guess animals don't hide their pain that much no oh well maybe I don't know we need to maybe I'm anthropomorphizing I guess yeah yeah yeah well I guess there there are many reasons one reason is I mean we've tried to clear to clean the data and have as sort of clear images as possible but again given the examples I don't know if if I mean if you've seen the sample images from theat in the paper but there's still lots of like different sort of faces different colors a little bit of difference in the color we did clean the data so we excluded the ones that look totally different but still there are lots of variation there is also lots of variation in the Viewpoint as well in these images so I think better data will will definitely improve accuracy also as well there is a possibility of actually excluding the whole step of detection of the action unit so maybe we just map directly from the face from the appearance features to the pain level we haven't tried that yet but I think this can improve also the accuracy that's actually the the the next experiment that we're planning to go ahead with doing because then you would actually depend on the classification system or the svm to build the model like a holistic model it could be that actually detection of the action units means accumulating some of the errors in the labeling of the of a specific action units that's how I'm I'm thinking about it so I think if we exclude the intermediate step and maybe map directly from the Sheep phase to the pain level we can get better result so when I read the paper by the way I'll have the a link to the paper in the show notes and I encourage everyone to check it out and see the images and you know read into the more details I kind of presume that I guess some of those intermediaries the you know uh extracting facial action units and the histogram of oriented gradients that we haven't gone into yet that those were essentially useful techniques you developed in a sense to compensate for the limited Corpus of data you couldn't go to deep learning with just 500 images so rather than hoping deep learning would do your feature extraction for you you brought your knowledge of the field and the species and that sort of thing to the table to extract those features and then give it the ability to learn on a more limited data set so are you going to be able to go more directly from the image just by better technique or will that demand a lot more images well definitely if we have more images we'll have better techniques as we mentioned at the very beginning but I mean we need we need to experim to experiment with that the the reason why we used also the these sort of action units because the scoring system that we got from the our vet collaborators was still based on specific scoring for every feature that was one of the reasons we start with that the experiment I'm thinking about is actually if we still use appearance features again still use appearance around specific features so not really feeding completely row data now again still apply some feature extraction that can be Hogs or maybe some geometric features as well on the specific landmarks so around specific landmarks and again around specific features and then map this directly to the pain level what I'm hoping to achieve by doing this is to really just uh compensate for any bias in the labeling of the specific action units let me I think that's a that's especially for the intermediate stages because as I told you I mean when we for example excluded the intermediate action units or the intermediate changes of the shapes we we got better results because basically maybe just like a totally clear ear flat is totally different than ear flipped and that's that was very clear in the appearance it was picked in the appearance features that we extracted that's what I was hoping to do but definitely I mean it's still very difficult to go to something like deep learning with this amount of data today's episode is sponsored by Periscope data Periscope data is a web-based tool for database exploration rapid visualization and dashboard development I like it because of how accessible it makes my data to me no longer do I have to write a query then write some Matt plot lib code or some ggplot one-step Discovery makes things happen a lot faster it connects to all the major database types and even joins across them it's perfect for teams you can get the ad hoc query monkey off your back if you find yourself bombarded with constant requests from your colleagues do that by building the perfect chart do it once make a dashboard out of it and even set up that dashboard to automatically email it to the right people every day never make that same report twice again if you believe in reproducible analysis head over to periscoped dat.com Skeptics to check it out once again that's periscoped dat.com SL Skeptics so how well will these techniques transfer to other animals I think all of them are transferable to other animals given that we have the data and do the proper facial Landmark detection because the landmarks will definitely so retraining all of the um facial Landmark detection systems and doing the a feature normalization based on the animal that we're handling but it should be it should be easy to map the facial feature analysis to other animals as well and given as well that we have the scoring system that we need definitely the work of vets on that because we don't have the knowledge of how like different species express pain in certain cases transfer learning has been celebrated as a a useful technique in image recognition I think a lot of the use cases I've seen are maybe very close to the original so maybe a someone has a neural network that was trained on imet to recognize uh let's say you know certain types of um tools and then someone could extend that to recognize the brand of the tool and this is sort of a uh they benefit from the early training and the way the network was essentially ini IED do you think that there's an opportunity there for you to benefit from transfer learning or is what we've learned about human faces a little bit too foreign to transfer cleanly to your domain there's some work done actually uh I mean I've seen uh recent paper at cvpr that at um the work done by group in UC Davis they did use a little bit something similar by actually warping the face of the animal to look a little bit similar to how the human face look like and then use transfer learning for the facient landmark detection the paper reports very good results but my sort of concern with that is that we have different features on the face of the animal that are important so which parts of the face are the most important parts so for example the ear when we look at the facial normal detection of human we do not detect the ear the ears are totally excluded yeah so while actually the ears are one of I mean I mean the ears one of the very important and clear like feature that actually where it shows different sort of emotions or pain in in many animals not only sheep so that's that would be the tricky part so yeah I guess the regions of interest in the sheep or in different animals might be different than in human as well something like also how to do the normalization what are the static points that so for example human nose usually is used in some of the tips of the eye because these don't move so how this maps to other animals we need to be a little bit cautious with this but uh yeah nothing in my mind so far I mean I'm not sure how we can use transfer learning to do but I mean definitely worth exploring maybe so it seems to me that the animal rights Community ought to be big fans of your research it's not at all invasive and uh the more successful you get that provides them a tool for maybe you know helping those animals to communicate in a way they weren't able to before have you seen that sort of response well there's lots of interest from Animal Welfare definitely I mean having an automatic way to assess the pain or the difference will definitely help early detection as well of specific diseases and it is noninvasive yes there's plenty of Interest mostly from the an animal welfare of people but not sure I think the media is the one that pick this up makes sense can you talk about some obviously it's a research project but uh do you aspire to any industrial applications or where do you see the work ultimately contributing we're trying to move this a little bit again under the sort of the research umbrella not sure about industrial applications but yeah know we haven't hear from any sort of industrial collaborators it's mostly universities here in UK so yeah so what's next in h this line of work what kind of things are you continuing to pursue beyond what we've had a chance to read about in the paper we're working still on sheep and that's mainly uh because of our vet collaborators and the data that we can get access to so next step is to work on profile faces and extend the taxonomy to include maybe other features such as the cheek and improving the technique as well so that we can get better analysis of the pain that's what I mentioned is that if we can just go directly from the face to the pain score this might a little bit increase the accuracy given that as well that also that the pain the the labels that we get is just from one person as well so I think also there are lots of probability of having some bias in the leing where can people follow uh your research and uh future things like this online and maybe keep in touch if they want to yeah sure so we have actually uh a web page for the project under the our group The graphics and interaction group here in the lab if you just go to the website of the University of Cambridge under the graphics and interaction group you can reach this by just www.l.com ac.uk emotions and here we have all the list of projects that we're working on and this uh include oine aect we have like a sort of a summary of what we're doing so far and list of applications that hopefully we're working on very cool well I'll have a link to that in the show notes so people can follow up Mar I want to thank you so much for coming on the show and sharing your research I think it's a really interesting thing uh not only for its you know value uh in in the sort of biological sense but a nice example of machine learning in an applied way so I was very grateful to get a chance to speak with you thank you so much guy it was nice speaking with you as well data skeptic is a listener supported program to support the show visit datas skeptic.com and click on the membership [Music] tab