Ultimate Guide to HANDS with Stable Diffusion! (Any pose you imagine)

Key Takeaways

complex hands like this or this are pretty hard to get with a ey alone to get them we will need to follow a process composed by two main steps and we will need both of them but let's look at some things first you might have seen negative embeddings that claim to improve hands or luras for some very specific hand gestures and most of the times they are like Sakura from League of Legends they try to help at best I've tested multiple embeddings and none of them have given me good enough results without cherry picking you can use them but they are not magic while luras have the problem of just being able to create a very specific pose and with strong bleedings at that so they are extremely limited even if the Laura itself is okay then there's the control net option and this can work in some simple cases like these where the hand pose is very shape oriented but we will need to learn how to use it for more complex cases and add a little something to the mix that said the first step of the process will be creating this is where we think about how the hands are going to be posed either having a full image in mind or just planning a specific hand pose on an existing image in any case we will plan our approach to it usually meaning what control and models we will need which will depend on the complexity of the hand pose I divide them in three groups silhouette poses when all the fingers are nicely separated and visible without overlapping each other like a waving sign for this a prompt and a good reference will do the trick preferably an edge detection model like cany soft Edge or linard even though both open pose and death can do the trick two simple gesture poses these are poses that have some overlapping and maybe even interaction between the fingers or with an object but each finger is easily identified like a rock and roll sign a high field of view image like this one or someone holding something simple you will need a good prompt and more than one controlate reference usually a mix of open pose line art and DEP line art models being almost necessary and DEP making it easier for AI to know which finger is in front of which three bra poses yeah this will need fixing all the time because I don't even know what's going on here and this is where our secret weapon will help later on in the video as you can see they are a nightmare for AI to understand and good luck generating them in text to image without fixing anything there's no after detailer that's saving that pie oh and by the way I don't use after detailer for hands never got any good results to make it worth the extra time generating as the first example let's try to make this POS okay I tried to sketch it a little better but as you can see I can't draw so this hand is supposed to be holding a ball and this one is pointing at the Target this will definitely be a challenge one has overlapping with the rest of the hand and also in a hard perspective and the other is not only holding an object but I'd like it to feel like the hand is applying pressure to it why do I make stuff so hard on myself now you have a lot of options when it comes to approaches and creating control net references but if I listed them right now this video would be looking like a Greek's God family tree so instead I'll do some examples and go one by one for this I chose my favorite approach to making concerned references 3D posing tools more specifically pose my art this is great for when you're planning the full composition of the image since you have full control over the camera angle field of view pose and even the dep map contrast you can also export the open pose and DEP maps for the full image instead of trying to make them match the hand position later on I usually export both open pose with and without hands just in case the reference with hands messes up the rest of the body also by activating the preview death option and playing with this slider you can adjust the contrast of the death map in relation to distance to camera this is very useful to avoid having death Maps where the hand is fully white you want to have at least some variation on there so that the ey knows what fingers are in front of each other the only controlled reference we are missing now is The Edge detection one you'll notice that I haven't exported it from Post my art that's because the lines are always very messy and annoying to deal with you can either hit export image and try to get some controled pre-processor to do the work correctly or just draw the outline manually your yourself which is what I'd recommend and what I'm doing right now good thing about this method is that you can draw stuff as you imagine it like the ball being compressed from the strength of the hold just remember to export on white over black with everything done we'll finally move into stable diffusion import our control net references and prompt accordingly to what we want it's important to say that you need to be careful when using multiple control net models try to keep them on the low weights as for now we are not looking for the perfect hands we are looking for a good base image to work from some tips for that are when using multiple control Nets play with the weights of each model lowering them a little for the edge detection model I used liner found it is the one that works best for me but you can experiment for yourself it might be checkpoint dependent I lower the weight a bit and make it stop affecting the image before the last rendering steps edge control net models tend to make the image very flat for this I used the weight of 0.8 and a 0.9 control net ending step for dep this is even more true I used 0.35 as the control net weight and 0.3 ending step that's because death works too well for its own good so it will try to make a character that fits the outline it also trashes the colors if you put it too high your character might end up looking like pink guys cousin and that's why I end it as soon as possible if you're using normal map this is even more true for open pose you can leave one and one if you want I tend to lower the weight a little bit mainly because I don't mind the posst changing slightly and because I'm combining it with other control net models which tends to overblow the image if just one hand is holding something don't prompt it in it will most likely not know where to put it in the image and end up messing up for example here I prompted holding a red ball to see if there was any lucky generation and it just invented a new prothesis taking it out of the prompt we start to get what we were looking for I generated with a bunch of different checkpoints and Styles as you can see Hands ain't looking that bad but they aren't perfect at all mainly the hand that's further away for people with this many fingers they aren't great at holding balls it's time to see the second step for good hands fixing this step comes in when once the image is already done and you know what you want the hands to look like for this example I will choose to go forward with this image right here I quickly imp painted some background and now we will fix the hands I like to fix them before upscaling because that way you can create variations faster of course some hands are easier to fix than others for example the foreshortened hand is not bad in terms of shape and it has the right number of fingers this tells us that the controlled models that we are using can be enough in this case we will just use those and in paint until we get something that looks looks good important to keep in mind that when in painting we can focus on the hand and don't really need to worry about the rest of the image this allows us to use higher control n values to ensure that for example the cany and DEP models are being properly followed go as high as your pose needs without going overboard I would love if we could use only masked for this since sometimes the image resolution is just not high enough to inpaint an automatically correct hand but we need to use whole picture because only masked messes up the control models be aware of that when in painting hands we are looking for the right anatomy And if possible a somewhat correct lighting the rest is secondary as long as the big shapes are right we can fix the rest you could even inpaint over an already inpainted result until you get what you wanted after a few attempts I got the hand I was looking for the skin tone doesn't really match the rest of the image though but we will fix this in Photoshop I will leave a tutorial in the description on how to match skin tones you could spend more time rrolling in the in pain tab to see if you get lucky with a better hand but I'd much rather do it manually and get them right you can also see that I lower the contrast on the outline a little bit as the AI made one that was harsher than what the image already had and now onto the actual challenge for this image the hand squeezing a ball I already have the controll and model and this time I don't have to worry about the prompt ball messing up the full image I will prompt for the hand squeezing a red ball after some tries I noticed that stable diffusion refused to put some balls on the situation I increased the weight of the work and Tred again even with that only one generation actually understood what I was going for so I froze the seat that had the good ball and used the extras option to create variations from there unfortunately I wasn't able to get that squeeze that I was looking for forcing me to do it myself I went into photoshop with the controled reference that we used and painted over it focusing on getting the hand correctly posed and then I used the liquefy tool trying to mimic the shape of a ball being held strongly of course that wasn't a super great result so I manually went over it and fixed some of the issues it had left and taam there we have our clean little hands might not be perfect but they will be if you're willing to spend the time on them since I can't draw realistic yet if I was to redo this with the realistic image I'd probably take the next approach I'm going to show you it's a very nice way to fix hands whenever the pose is pretty easy or if you need the hand to be holding something specific using in paint sketch in this case I wasn't getting any ball at all with with the regular in painting but here we just have to paint the red ball in and then with a simple prompt we are way more likely to get a good result yes the squeeze isn't really there but I drew this in seconds if you spend a little more time on it you can get much better results for the fingers to be okay to you still need to use some control net even though what's great about inpaint sketch is that sometimes it makes it so you don't need control net at all for images like this where the hands are bad but fairly easy to correct there is no need to create a full open pose or line art reference in images where we are missing a finger we just paint the base with a similar color as the skin then hit generate using something like 0.6 the noising strength and this creates the new finger really easily of course it isn't perfect but a very nice starting point with almost zero effort you can also use it to erase extra fingers like here where we can paint the black background back in or both like I'm doing for the generation where I will add the missing finger while erasing this weird strawberry at the same same time painting it out with background colors no need to be super precise that's the fun of it and now it's at least looking like a hand even if the anatomy is not perfect given the low effort it requires it's definitely worth it I would recommend doing this in the last version of automatic 1111 or using the canvas Zoom extension those have great in painting tools that make this process much easier and faster unfortunately canvas Zoom doesn't work for me anymore so I have to guess the Colors by eye you should be able to use the color peer though all right but but what about if I don't already have the control n references for the image and the Hand isn't easy enough to fix within paint sketch what do I do then one option is looking online for images of hands that match the one you are trying to fix and then merge them in Photoshop or whatever software you prefer from here you can either use image to image to fix anything that feels out of place or just compose them indirectly in Photoshop if you already find a good image and have the skills to do so if even with this image to image is messing up the hand you can do what I did in the B beginning going over the composited hand and making an edge detection reference manually now in painting over the new image with low the noising strength something like 0.4 and the control net reference if you don't mess up the prompt it should give pretty good results most of the time the main problem with this method is that looking for an image online that matches what you need can take a lot of time mainly if you're are going for a weird hand posst or it's seen from a very uncommon angle and that's why I made this little tool with I made I mean I screamed that J GPT to make it for about one day this let you import your image so you can see it over here now we adjust the brush size and in paint an area of our choosing in this case I will take out the hand with that done activate your webcam doing this you can pose the hand yourself and know if it looks good or not which is way faster than looking online or trying to match the pose on your own without actually knowing how it will look like you can adjust the position of the camera as well as the size and you can also flip it just in case the POS is really uncomfortable what I do is find the correct Poe first moving my hands around until I know where and how to Poe them then set a timer of about 10 seconds and hit the take picture button the timer will start and freeze the camera when it hits zero if everything's right now you have the post hand if you messed up just start the camera again now we download the image and clean it up in Photoshop I usually mask out the hand a little bit maybe even adjust it to fit better and then make a line art for control net export everything to stable diffusion and with this we should have everything we need if you want to try hard a bit more you can import the image into open post choose the DW open post pre-processor and after hitting the explosion button to check the preview edit the pre-processed reference using the open Bose editor extension this lets you add the hand joints and even Poe them having this map an image to use when imp painting and a wood prompt you should be able to make pretty much any hand you want the idea is to use original mode with a loow the noising strength how low will depend on the image you made if it's very close to how you want it to look like then 0.2 or 0.4 should do it if it captures just the shape but nothing else try to go for a higher the noising strength like 0.5 to 0.75 combine this with high control net weights you can go as high as you need as long as it doesn't break the image if it doesn't work and the Hand pose is extremely hard you might want to stay for some tricks at the end I use this tool to easily correct some hands including this one here with the glass of wine for this I actually post with both hands at the same time using the timer option and thear I use for my weights I will leave this tool for free in the description I hope it works for everyone but I'm no programmer so expect some bugs obviously you don't need that tool you can also get some preate hand death Maps or line arts from libraries like death library and just paste them over your image before imp painting take the background out before exporting though so control net doesn't get confused you can also just create an open post reference from imagination and I mean if you know how to draw you could even draw the line art for the hand yourself or just draw the hand directly now that we're talking about knowing how to draw that will help a lot for this hidden step of the process clean up and retouching here we just go over the little details of the image that we don't quite like fixing the lighting the anatomy the color whatever it is fixing hands manually can be tough so I will leave some awesome hand drawing tutorials in the description that way you can start to see when hands don't look great and why all that's left is for you to actually care about fixing them which we don't always do as promised here are some final tips and tricks if your character is using gloves or has a different different colored skin than you try to match the tone as best as you can when composing so AI has an easier time when in painting that way you can go with a lower the noising strength you could match the skin tone with the tutorial I talked about earlier or just by masking over and painting over the hand with the desired color sometimes you can have trouble with AI putting a finger over another in the incorrect order which happened to me while making Kila from my dress up darling in these cases don't be afraid to create your own death map manually just use the base you made for the line art and fill one f finger with a darker gray than the other even going as far as painting the one that's closest to camera fully White and the other one a dark gray import that into control net dep and use it to in paint if your image has an extreme perspective that you can't match with your webcam you can try to fix it with 3D models for this image I used Maya but you can also use po my art for this give the model a material that's close to the desired skin tone and here on settings you have a field of view option try and match that perspective if you have any questions on this topic please the Discord and I'll be happy to respond the community is full of very knowledgeable people that are willing to help see you there and while you go watch this video right here let me say sorry for not posting in so long I messed up my leg and it still hurts a little it is still healing but it's better now and I also messed up my hand now it's almost fully healed though thank you so much for watching and have a great day