[Community Challenge #11] “Masks”

Sorry for the low quality, i´ve found no other way to upload animations directly to lemmy, yet.

Edit: Here is a link to a sharing Portal with a higher Resolution GIF :

Gifyu

The comfyui Workflow is also embedded in this picture, you will have to install several custom extensions to make this work:

• Generator: Comfyui
• Positive Prompt: large format photo of (Masterpiece, best quality:1.2) A dancing ninja girl (wearing a wooden demon mask on the head:1.1) intricate ornamental carvings, medieval background, full body, cinematic lighting, Kodak Funsaver, Kodak Vision3, 50mm
• Negative Prompt: bad anatomy, bad proportions, blurry, cloned face, deformed, disfigured, duplicate, extra arms, extra fingers, extra limbs, extra legs, fused fingers, gross proportions, long neck, malformed limbs, missing arms, missing legs, mutated hands, mutation, mutilated, morbid, out of frame, poorly drawn hands, poorly drawn face, too many fingers, ugly, ugly, duplicate, morbid, mutilated, out of frame, extra fingers, mutated hands, poorly drawn hands, poorly drawn face, mutation, deformed, ugly, blurry, bad anatomy, bad proportions, extra limbs, cloned face, disfigured, out of frame, ugly, extra limbs, bad anatomy, gross proportions, malformed limbs, missing arms, missing legs, extra arms, extra legs, mutated hands, fused fingers, too many fingers, long neck
• Negative Embedding: FastNegativeV2
• Sampler: Euler
• Scheduler: normal
• CFG: 6
• Model: lyriel_v14

This is a quite interesting workflow as you can generate relatively long animations.

You draw the Motions of your character from a video. For this one i googled "dancing girl" and took one of the first i´ve found:

Link to youtube Video

You can draw single images from the Video into comfyui. for this one i´ve skipped the first 500 Frames and took 150 Frames to generate the animation. The single images are scaled to the Resolution 512x512. This gives me a initial set of pictures to work with:

Via the openpose prepocessor you can get the Poses for every single image:

This can be fed to the openpose controlnet to get the correct pose for every single animation. Now we have following problem. We are all set with the Poses, but we also need a set of latent images which have to go trough the ksampler. The solution is to generate a single 512x512 latent image and blend it with every single VAE encoded Picture of the Video to get an empty latent image for every Frame:

We get a nice set of empty latents for the sampler:

then we let the ksampler together with the animate diff nodes and controlnet do its magic and we get a set of images for our animation ( The number of possible images seems to be limited by your system memory. i had no problem with 100, 150, 200, 250 images and have not tested higher numbers yet. I could not load the full video):

Last step is to put everthing together with the video combine node. You can set the frame rate here. 30 FPS seems to produce acceptable results.: