Mastering AI Animation: Use Auto-Mask, ControlNet and AnimateDiff Evolved! - Video To Video

Key Takeaways

hello everyone and welcome back to Dreaming AI my name is n and today we are finally going to learn how to create decent results starting from videos I say decent because I'm sure that in the future I'll find other methods to do even better and as soon as it happens you'll be the first to know I promise I also take advantage of this tutorial to explain a bit about how an make def Works a node that is famous in the field of animation with stable diffusion uh personally I don't use the basic version of the node even though it works very well uh instead I use the version called animate diff devolved which supports a technique called sliding context window in simple terms it allows us to use only the latest images from a larger set for generating an animation because normally reaching a certain number of in ized images would force us to stop resulting in very short videos unless you have an excessive amount of vram in conjunction with animate diff evolved we need to use comfy UI Advanced control Knack created by the same author for the same reason since the official control met node uh does not support the sliding context window does this mean we can create infinite animations without thatching smaller sizes well it depends from all the tests I've done if we use multiple control net nodes in our workflow even with the version I just mentioned you will almost certainly end up reaching the limit of your vram however if you use the animate diff node and only one control net node to my surprise I've seen that I can reach huge batch sizes without any difficulty except for the OB is processing slowness of various frames in the K sampler uh the vram saturates but it never goes beyond the maximum limit indicating that the sliding context window works well I will leave for links to the nodes I just mentioned and all the ones needed for today's workflow in the description all right let's proceed with our workflow since the workflow is moderately complex and creating it from scratch would probably just make you fall asleep this time I'll explain everything starting from a project I created earlier firstly as you can see I used the video from last time downloaded for free from vidivo reducing the frame rate to a quarter and adapting the resolution so that the height is at most 512 pixels this time I won't use any batch size and I'll leave images limit at zero so that the video is processed in its entirety and in a single large batch so here as you can see I've set up an LCM flow with animate diff and control net and the model I'm using you already know is minimix to which I've attached the clip set last layer node set to -2 as suggested by the author right after we have bolora of LCM which I've already explained in the previous video and finally our beloved anime def as I mentioned the technology behind this node is quite complex and involves modifying the base model so that it can undergo other models trained to allow a very high coherence between frames in the final video but not only that animate diff allows the creation of real animations from scratch without starting from an actual video but by following what is told in the pr I'll make another video about this soon animate diff just like control mat has its own models divided into two categories the base and mandatory models that are divided into those working with sdxl and those for previous models and that handle the coherence between frames and the interpretation of the prod the motion Laura models which are optional they handle the creation of basic animations like pan s Etc as you can see from this example you can download them from the usual hugging face and you'll find all the links in the how to use section of animate diff evolved GitHub after downloading them you need to put them in the respective models and motion Laura folders inside the node folder returning to to the workflow the main node of animate diff is definitely the loader where you'll choose your main model for this workflow I decided to use the temporal net model and then we have the section to choose the beta schedule or we'll practically always choose this for models 1.5 and linear animated if sdxl or hot shot XL when we use models based on sdxl technology in our load checkpoint uh next is the motion scale which is simply the impact of animations on the final result finally since the animated if models also exist in version two we have the option to enable support for them through this toggle you'll notice that you're using a version 2 model usually from its description great the second and last note that we'll delve into for animate diff evolved is uniform context options this node consists of the following entries context length which indicat how large the window in which animated if can compose the animation is the larger the window the more RAM it will require and the longer the animation will be coherent another peculiarity of evolved is that unlike the classic node that can go up to 16 frames K the author has managed to push the context length to 32 context stride which identifies how often a sampling operation will be performed I usually leave it at one context overlap which indicates how many frames from the previous context block will be used at the beginning of the next context block to ensure that the animation remains coherent context scheduler which for now can only be set to uniform closed loop which indicates if we want to make the video a closed loop for the control net section I opted to use a linear model as it helps consist consistently preserved the image in the original video but I applied it only to half of the output now let's move on to the bottom part uh where I applied a technique seen in a video on YouTube I'll leave the link in the description uh this is one of the many possible ways to create an automatic mask for a character in an image or in this case in a video isolating it from the external environment to preserve the ladder from changes uh everything is combined by the vi and code for in painting and then pass to the usual um case sampler which has the necessary values to allow LCM to do its job finally we have a face swap with reactor and a face that I created earlier with sdxl and a frame interpolator to achieve the decent frame rate in the end everything will be saved and compiled by save video per perfect let's start our workflow and see the results as you can see the video has maintained excellent coherence and the original background despite changing many details of the subject remains consistent and that's all for today I hope as always that this video has been useful to you as I mentioned in the next video we will also discuss the generation of animations using animated if only in the meantime you can find this workflow in the links in the description please consider liking and subscribing if found this tutorial useful also if you have any questions please let me know in the comments below I'll be happy to help you out as much as I can and until next time keep [Music] dreaming