Hand Pose Detection with ml5.js

hello and welcome back to the coding train today I'm going to look at the hand pose model in ml5 [Music] JS this video is part of my beginner's guide to machine learning with ml5 JS Series in the previous video I looked at the body pose model there are actually two of them there's Blaze pose and movet I went through all the details about them but I'm going to take one of the examples from the previous video it's this one right here loads movet applies it to a real-time video image and I am going to swap in the hand pose model and look at what we can do when instead of tracking a series of key points on the body you are getting detailed key points on your hand for some inspiration about the kinds of creative applications that you might try to make with body pose tracking hand pose tracking and hopefully in the next video when I look at face mesh tracking I encourage you to check out the Instagram page of the wonderful F teacher creative coder and artist Jack BD he's made a lot of demos for these models that are in ml5 JS and a lot of my teaching materials are based on the work that he's been doing in the last year after looking at the basics of how the hand pose model works I'm going to attempt to recreate this particular demo where you draw with your fingers only when your thumb and index finger are closed together once again before I use any particular machine learning model in a project I always like to investigate who who trained that model how that model was trained and with what data set for this model the starting point is the 2019 Google research blog post on device real-time hand tracking with media pipe one thing I first noticed when reading this article is there actually two models one first just detects where the hand is a bounding box for the hand that way that image can be cropped and sent into another model which then estimates the XY positions of all of the points in the hand here we can see documentation of the training data they took 30,000 realworld images of hands and manually annotated 21 key points over all of them in addition to the real world images of hand they used synthetic data or 3D rendered models of a hand here we can see that by mixing both the real world images and the synthetic data the model was able to achieve the lowest error score after being trained the model that is in ml5 is an updated version from 2021 improved accuracy for 2D and can actually track the key points on both hands in the previous video we looked at body pose which receives as an input an image the output is an x a set of XY key points uh you can estimate a z if you want with blaze pose and you also get a confidence score for each one of those points now we're going to look Instead at hand pose there aren't multiple options for hand pose in ml5 and the output is the same a series of XY points with a confidence score and you get 21 of them the hand documentation includes a really nice diagram shows you all of the points their index value into an array as well as their name that you can use to access them instantly and so here I am with the code from before I've got a variable for video I've got a variable for the pose model and then an array to keep track of all of the poses that the model detects the draw Loop checks to see if there is at least one pose detected it only ever looks at the first pose and then Loops through and draws all of the key points as long as the confidence score is greater than 0.1 so I'm going to now adapt this to work with the hand pose model I'm going to change the variable name from body pose to hand pose I'm going to change the variable poses to be hands I'll change this from pose to hand let's change got poses to got hands why not all right I've changed all of the names of everything but I'm still loading the body pose model if you didn't believe that you could name variables whatever you want now you really see it doesn't matter what we call stuff it matters what we do and what I want to do now is switch from calling ml5 body pose to ml5 hand pose I'm also going to remove the movet string there's only one model for detecting hands in ml5 as of now uh oh I don't see anything so what's happening in the previous video I did all of my debugging by console logging the results from the model anytime I click the mouse so let's do that again okay a hand was detected index z. keypoints that looks the same each key Point has an X and A Y and a name aha the key points themselves don't have individual confidence scores there is only an overall confidence score for detecting the hand itself so let's adapt the code according to that so I'm going to take out the if statement with confidence altogether now you can see it's tracking my hand if I wanted to have a confidence threshold I could do that on the hand level now I'm only drawing the hand if the confidence for it is greater than 0.1 so which hand is it drawing unlike with body pose where I only wanted to work with the first pose that was detected here I probably want to look at all of the hands that it's detecting so right here instead of just looking at hands index zero I could add a for Loop and let's let's use a four of loop for every hand in the hands array or let hand of hands if the confidence for that hand is greater than 0.1 draw all of its key points it's also useful to note that there is a handedness property which is either left or right so maybe I could change the color based on whether it's guessing a left hand or a right hand I can add an if statement to check if the handedness value is left draw pink otherwise I could check if it's right but there's only one other option this should be yellow I think pink yellow okay so you see how to draw all the key points hopefully your creative juices are flowing and have all sorts of ideas of what you want to make most of your ideas I'm guessing involve picking specific points on your hand and doing different things with them so let's go back to that idea of drawing with your fingers when the index finger and the thumb are connected I'm going to go back to looking at one hand only again referring back to the diagram I want the index fingertip and the thumb tip of course I could just go back to that array and get index 8 and index 4 but I'm going to use the name instead so now now I have two separate variables holding on to two points on the hand let's just draw them index and thumb so we can see I have just those two points again I'm just looking at whatever the first hand it detects is what I want to do next is determine if the index finger and the thumb are touching I can use the distance between those two points to make a guess as to whether or not they are now of course I could do things like turn my hand like this and they're right on top of each other but they aren't really touching possibly a way around that would be to use the 3D keypoint estimation but I'm going to keep it simple right now I'll leave that as an exercise to you in P5 I can get the distance between any two points with the dist function and let's make a threshold say if that distance is less than five pixels change the color of the dot wait a second that's if greater than that's fine oh I can't get five pixels is not enough so five pixels is a bad threshold also I said greater than than less than let's try 20 pixels they're pink they're yellow now if I want to draw a single dot as I move my fingers around it would probably make sense for me to take the average position of both of those points I can do that by adding the x's and y's and dividing by two oh and look at that there's that average position and it turns yellow when my fingers get close interestingly enough if I make the size of that Circle the distance what happens now I have a circle that is always in between those fingers okay but what I want to do is draw or paint with my fingers here is a simple Mouse painting sketch in p5js this sketch works because I draw the background only once in setup but in this sketch the background is not a plain solid color but the video itself and I want to keep it running continuously which means I can't draw the video only once in setup I have two options that I could pursue here one is I could use an array and accumulate all of the tracking points into an array and always draw a continuous line between them another way I could approach it is by thinking of my P5 sketch output as having layers and the way I can create a separate layer for the painting which is different from the video is by using Create Graphics I'm going to have a global variable called painting when I create the canvas I'm going to make another canvas one that you don't C but that's just stored in a variable called painting by calling create Graphics now instead of drawing this circle directly on the canvas I'm going to call it on the painting all of the P5 drawing functions that you call globally stroke fill circle square all of those functions can be called on either the main canvas that you see or any create Graphics object that you create behind the scenes however now I don't see any of that stuff it's being drawn on a canvas That You Don't See but if I want you to see it I can place that layer that canvas on top of the other one with the image function and I'm going to do that at the end image painting right there at 0 as well and look at this it's essentially working already I kind of almost didn't expect it to do that one thing I didn't bother to think about is whether I should draw a background on the painting at all if I had done that in setup like let me just draw a white background for example because that painting layer now has a white background and it's drawn on top of the video I don't see the video anymore if I were to draw the background every frame now we only see the single dot but no video in P5 in addition to the background function there's also a function called clear which basically draws a fully transparent background on a canvas it clears all the pixels setting them totally transparent so clear is actually what I want to do I just didn't actually have to because when you first call create Graphics it makes a clear canvas okay but the interaction isn't really right here I only want to draw the circle if the distance between these fingers is less than 20 all right drawing nothing put my fingers together now I am drawing my cat [Music] there's a few things I can do to improve this one is it would be nice for me to actually connect the current position of the hand with the previous one just like in this painting demo that's done with mouse X Mouse y p Mouse X and P Mouse y but there's no P hand X and P hand y I'm gonna have to make that myself I'll just call it PX and py and I'll give them an initial value of zero then at the end of draw I can save the XY from this current frame so that when I get the new one next frame I can connect those two with a line and I have to do it inside this if statement here where I'm calculating that XY position now that I think about it I should always get that XY position but only paint if the distance is less than 20 so let's move a few things around so I'm always going to get the XY position I'm always going to store it as previous X and previous y but I'm only going to paint if the distance threshold is less than 20 still working and now instead of a circle I can change this to stroke I can change this to line from PX py to X comma Y and I can make the stroke weight something like eight hm ah look what I forgot to do if I want to change the stroke weight on that layer the painting layer I've got to say painting. stroke weight there's so many more things I could do in this video but I'm going to stop here and I'm really excited to see what you the viewer might make with this ability to track hands here's a few prompts one is how might you use your other hand to change maybe the thickness of the line as well as perhaps the color could you design some sort of system by which you use other fingers to do that and finally you might be thinking what about recognizing particular gestures like thumbs up versus thumbs down or rock paper scissors well this is an interesting question could you use if statements to recognize if somebody is making a thumbs up or thumbs down based on the relative position of the thumb tip and other fingers other parts of the hand that's one option however what I'm going to be doing in a later video which is hopefully coming up soon is looking at how we can take the output of the hand pose model and feed that into another machine Learning System to do hand pose classification okay if you like this video the next one I'm going to do is on the face mesh model in ml5 the hand post model has 21 key points that face model it has a lot of points here's the example I'm going to check how many points are in there 468 see you then [Music]