How to use CONTROLNET in Stable Diffusion ( All 17 Tools )

Key Takeaways

Control net in stable diffusion allows you to get predictable results using some source image. It works in the same way no matter where you use it in online generator or locally with comfy UI. I'll use controlNet tools in Cart AI because it allows you to create free images with decent amount of daily credits. After creating account go to image generation tab legacy mode should be enabled. There are two options control net itself and preview. I'll use preview tool a little bit later. In the control net tab you'll have to add your source image. Here you can see standard image generation adjustments and here are control net settings. First four tools I'm going to show you are edge detection algorithms. Cany, line art, line art, anime, HED allow you to create new images with the same objects but in different styles, lighting, etc. This is my source image. Image generation settings will be the same for each of these four tools. After selecting control net type, you'll see additional settings. Pre-processor should be enabled. Honestly, I don't know in which cases this invert option can be useful. There can be different versions of pre-processing models. The higher pre-processor resolution, the more details of the image will be recognized. I'll set it to maximum in this case. You can decide what should affect your generated images more. Control net or your prompt or keep it balanced. Control weight defines how much influence control net will have on the generated pictures. Here are advanced image generation settings standard for stable diffusion. I got two images created with Cany controlNet. So here's what happened. The edges of the objects on the original image were recognized as you can see on this preview picture and then two images were created based on my prompt generation settings and outlines from the original image. And another example everything is the same but image is different. So again here is original picture recognized outlines and generated image. The next edge detection tool is line art. There are three pre-processor versions standard realistic and coarse. We'll compare all three versions at once. Realistic probably is the best line art prep-processor for detecting small details. Only on the image generated with realistic pre-processor, you can see the smile like on the original picture. Line art anime works better with anime or some cartoonish pictures. There are line art anime and line art anime denoise pre-processors. Here you can see that second pre-processor recognized just some basic lines. But as soon as I use realistic source image and realistic model for image generation, results are spoiled. Even images generated with anime model don't look better. And now I'll use anime picture as source and anime model for image generation. All the other settings are the same. Generated images are much better this time. You can see that the noise pre-processor recognized less small details and the result is little bit less detailed. HD also detects edges of the objects. It's similar to line art anime and cany at the same time. I'll use anime and realistic images as source images. Realistic picture is really similar to what we saw when I used cany. anime image looks even better than when I used line art anime control net. Scribble HD can't recognize the edges accurately like previous tools. As you can see, it creates something like rough handdrawn sketch. So, the generated images aren't as similar to the original as in previous cases. MLSD recognizes only straight lines on your image. So, this tool can be useful when the main object on the picture is architecture or some interior. For example, I have image of the city. Of course, not all of these recognized lines will be used on the generated images if the strength value of control net is not very high. Open pose detects the pose of the character on the image. There are few different open pose pre-processors, but open pose full and DW open pose full pre-processors recognize full body position including face and fingers. Here is detected characters pose. Now this pose can be used to generate any other character in any style. But what if I have such image where are no face or hands clearly visible? Fingers and face features weren't detected, but turn of the head is guessed correctly. The next tool is depth. It will try to detect which objects are on the foreground and which are far from the viewer. Black areas are background, white are foreground, and gray ones are objects in between. Both generated images have same composition. The next tool is little bit similar to depth, but it creates some kind of normal map of the image. This means that it specifies the orientation of the surface of the objects. Here is normal map and image generated with it. The next option I'm going to show you is segmentation. Segmentation labels what kind of objects are in the source image. The buildings, sky, trees, river, sidewalks, etc. are labeled with different predefined colors. So the same type of objects will appear in the same areas on the generated pictures. Another example is image with woman. The shapes of the objects can be different on the reference and generated images. Segmentation just defines the borders of the areas where some type of objects can appear. Tile resample option creates more detailed versions of uploaded image with selected model and prompt. You can use detailed description of your picture as prompt. In this case, I used intelligent analysis to generate image description. I selected realistic model to create resampled version of this image and corresponding model to resample anime picture. If you want to extract the main colors of your image, you should use color grid. After selecting this pre-processor, the colors of the generate images will be close to the source image. So, this color grid was created from my reference image. And then I got this landscape I briefly described in my prompt. But the colors of the generated image aren't really similar to those extracted from reference picture. So I increased control strength. So now the result is closer to what I expected. Shuffle works like style reference. It allows you to create some kind of remix of the uploaded image in the same style. It's better to generate images of the same type as reference. I tried to create landscapes from this anime picture, but as you can see, right image is too abstract. Left one still has the face of the girl. So I'll replace anime picture with this cartoonish landscape. All the other settings are the same in this case. Result looks better. Reference only option is little bit similar to shuffle. It allows you to create images with objects similar to the reference. Here is unique settings for this tool. Style fidelity controls how strongly original image will influence the generated ones. I used anime model and minimum style fidelity value. Style is right, but there are not so many elements from the original image. So I increased it to 0.8. I didn't mention the city or bridges in the prompt. But these elements appeared on the image after I increase style fidelity value. Nice. I'll replace this image with realistic portrait. And I enabled intelligent analysis. So prompt is generated automatically. Model will be realistic. Number of images will be set to maximum. The person on the images won't be 100% the same. So this tool is not very good for creating realistic consistent characters. Plus all of the images are overexposed. I have detailed video about different methods how to create your AI character. Too high fidelity value spoils realistic images. So decreased it to 0.2. Now image quality is better as you can see. Okay. Now I have this black and white picture but I can add colors to it with recolor option. I'll just describe the colors of the objects in the prompt. Model can be whatever you want to use. This time I used realistic model and got this result. IP adapter which means image prompt adapter works a little bit similar to reference only and shuffle. You can create variations of your uploaded image. You don't need complex prompts in this case. I just specified what I want to change on my image. And as soon as my image is realistic, the selected model is realistic, too. If you want to make the generated pictures more similar to the original one, you can add one more control net tool. You can use open pose, depth, some edge detection tool, or whatever you want. I selected normal bay. Here is what I got. Probably results can be even better if I'll find perfect image generation settings. Sampling method, steps, cfg scale, etc. can significantly change output. In the beginning of the video, I mentioned preview tool. It allows you to apply control net without generating new images. For example, I'll select Cany. Resolution will be set to 512 pixels. This time you can see this intermediate preview picture which can be saved, edited and used as input instead of regular reference image. Pre-processor resolution can be increased up to 1,024 pixels. Here are both variations. Right? One has less details, but the recognized details are more accurate. And the last thing I want to show you is how to combine any pre-processors for better results. I want to create image of an astronaut sitting on the ground in the field. I'll use this photo as reference for open pose. Model will be realistic. I'll select open pose full. Pre-processor resolution will be set to maximum. Clip skip will be set to one to prevent unnecessary artifacts on the generated pictures. I'll add second control net tool here to use landscape photo with depth map due define environment and background. For this goal also can be used canny line art etc. Strength will be equal to 0.7. Here is what I got step by step. The mountains from the reference picture became clouds on the generated image because I didn't mention them in the prompt, but the shapes are the same.